Trying to run the GAIL Example Code

[JankyOo]

First of all thanks a lot for this awesome project. Stable-Baselines helps me a lot!

I try to get a GAIL Agent going to get experience with Inverse Reinforcement Learning in combination of Gym and/or Retro. Therefore i want to get a Super Mario Agent running.

Unfortunatly i cant even run the example code.

The code runs twice. From the beginning till model.learn(total_timesteps=1000). After the learning it doesn't reach print("PrintDebug_2") but starts over at print("PrintDebug_1"). Then it runs till the learning step again but crashes after that with given error message.

As far as i found out with debugging something with 'popen' multiprocessing doesn't work correctly. It seems to somehow mix up the child processes or something similar.

Code

import gym
from stable_baselines import GAIL, SAC
from stable_baselines.gail import ExpertDataset, generate_expert_traj

print("PrintDebug_1")
model = SAC('MlpPolicy', 'Pendulum-v0', verbose=1)

generate_expert_traj(model, 'expert_pendulum', n_timesteps=100, n_episodes=10)
dataset = ExpertDataset(expert_path='expert_pendulum.npz', traj_limitation=10, verbose=1)

model = GAIL("MlpPolicy", 'Pendulum-v0', dataset, verbose=1)
model.learn(total_timesteps=1000)
print("PrintDebug_2")
model.save("gail_pendulum")

Error Message

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
    ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <bound method DataLoader.__del__ of <stable_baselines.gail.dataset.dataset.DataLoader object at 0x000001C21E0617F0>>
Traceback (most recent call last):
  File "C:\Users\_____\AppData\Local\Continuum\anaconda3\envs\Baseline36\lib\site-packages\stable_baselines\gail\dataset\dataset.py", line 339, in __del__
  File "C:\Users\_____\AppData\Local\Continuum\anaconda3\envs\Baseline36\lib\multiprocessing\process.py", line 116, in terminate
AttributeError: 'NoneType' object has no attribute 'terminate'
Exception ignored in: <bound method DataLoader.__del__ of <stable_baselines.gail.dataset.dataset.DataLoader object at 0x000001D2FC76EF98>>
Traceback (most recent call last):
  File "C:\Users\_____\AppData\Local\Continuum\anaconda3\envs\Baseline36\lib\site-packages\stable_baselines\gail\dataset\dataset.py", line 339, in __del__
  File "C:\Users\_____\AppData\Local\Continuum\anaconda3\envs\Baseline36\lib\multiprocessing\process.py", line 116, in terminate
AttributeError: 'NoneType' object has no attribute 'terminate'

System Info

• Another Agent with A2C runs totally fine in this environment
• stable-baselines 2.5.0 installed with pip
• Python Version 3.6.8
• Tensorflow Version 1.13.1
• Gym Version 0.12.0 / Gym-Retro 0.6.0
• Environment was setup with Conda and pip

[araffin]

Hello,

are you working on windows?

See warning in the doc to solve your problem ;) (you should wrap your code in a if __main__:)

Note: as also stated in the doc, the current implementation of GAIL does not properly supports with images (but we welcome PR in that direction ;) )

[JankyOo]

Thanks a lot for the quick response, it helped me a lot. The Demo is working now

And yes i am using Windows to test, but will run the code on a Linux-Server for training.

As far as GAIL is not properly working with Images, i wonder why for example A2C is recording images while generate_expert_traj. Anyway, i want to produce my "expert data" by playing the game myself while recording it in a .npz-file. Is there any documentation about the structure of the .npz-file GAIL expects?

[araffin]

The recording of images is made for the pretraining with behavior cloning (which is different from gail). And yes, there is documentation for that, please look at the pretraining and gail sections. You can also directly look at the source code in that case. Closing this issue then.