Example - DQN Baseline not working

[seboz123]

Describe the bug I am trying to run the example of the DQN Baseline Algorithm. The algorithm does not seem to work properly. It just uses interference mode. When I remove the neural network from the brain, the window opens, but the agent just does nothing.

To Reproduce Steps to reproduce the behavior:

1. Build Gridworld and remove the brain NN.
2. run the example

Console logs / stack traces

Process Process-2:
Traceback (most recent call last):
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\site-packages\mlagents_envs\rpc_communicator.py", line 78, in check_port
    s.bind(("localhost", port))
OSError: [WinError 10048] Normalerweise darf jede Socketadresse (Protokoll, Netzwerkadresse oder Anschluss) nur jeweils einmal verwendet werden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\site-packages\baselines\common\vec_env\subproc_vec_env.py", line 8, in worker
    env = env_fn_wrapper.x()
  File "C:/Users/Sebastian/Desktop/RLUnity/python/test2.py", line 22, in _thunk
    unity_env = UnityEnvironment(env_directory)
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\site-packages\mlagents_envs\environment.py", line 176, in __init__
    self.communicator = self.get_communicator(worker_id, base_port, timeout_wait)
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\site-packages\mlagents_envs\environment.py", line 241, in get_communicator
    return RpcCommunicator(worker_id, base_port, timeout_wait)
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\site-packages\mlagents_envs\rpc_communicator.py", line 49, in __init__
    self.create_server()
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\site-packages\mlagents_envs\rpc_communicator.py", line 55, in create_server
    self.check_port(self.port)
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\site-packages\mlagents_envs\rpc_communicator.py", line 80, in check_port
    raise UnityWorkerInUseException(self.worker_id)
mlagents_envs.exception.UnityWorkerInUseException: Couldn't start socket communication because worker number 0 is still in use. You may need to manually close a previously opened environment or use a different worker number.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\site-packages\mlagents_envs\environment.py", line 490, in _close
    self.communicator.close()
AttributeError: 'UnityEnvironment' object has no attribute 'communicator'
2020-05-17 16:43:57 INFO [environment.py:111] Connected to Unity environment with package version 1.0.0-preview and communication version 1.0.0
Process Process-1:
Traceback (most recent call last):
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\site-packages\baselines\common\vec_env\subproc_vec_env.py", line 8, in worker
    env = env_fn_wrapper.x()
  File "C:/Users/Sebastian/Desktop/RLUnity/python/test2.py", line 23, in _thunk
    env = UnityToGymWrapper(unity_env, rank, use_visual=use_visual, uint8_visual=True)
TypeError: __init__() got multiple values for argument 'use_visual'
2020-05-17 16:44:57 INFO [environment.py:500] Environment timed out shutting down. Killing...
Traceback (most recent call last):
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\multiprocessing\connection.py", line 312, in _recv_bytes
    nread, err = ov.GetOverlappedResult(True)
BrokenPipeError: [WinError 109] Die Pipe wurde beendet

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/Sebastian/Desktop/RLUnity/python/test2.py", line 43, in <module>
    main()
  File "C:/Users/Sebastian/Desktop/RLUnity/python/test2.py", line 34, in main
    env = make_unity_env('C:/Users/Sebastian/Desktop/RLUnity/ml-agents/envs/UnityEnvironment', 2, True)
  File "C:/Users/Sebastian/Desktop/RLUnity/python/test2.py", line 28, in make_unity_env
    return SubprocVecEnv([make_env(i + start_index) for i in range(num_env)])
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\site-packages\baselines\common\vec_env\subproc_vec_env.py", line 49, in __init__
    observation_space, action_space = self.remotes[0].recv()
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\multiprocessing\connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "C:\Users\Sebastian\.conda\envs\tensorflow_src_1_14\lib\multiprocessing\connection.py", line 321, in _recv_bytes
    raise EOFError
EOFError

Process finished with exit code 1

Environment (please complete the following information):

• Unity Version: 2019.3.12f1
• OS + version: Windows 10
• ML-Agents version: 0.16.0
• TensorFlow version: 1.14.0
• Environment: GridWorld

[awjuliani]

Hi @seboz123

Are you able to connect to and train the environment using the mlagents-learn command? That can give some context as to whether it is an issue with the gym wrapper/baselines or if it is an issue with the environment itself.

[seboz123]

mlagents-learn works just fine for me

[awjuliani]

Can you share the code you are running? This error in the trace seems to be the essential one:

mlagents_envs.exception.UnityWorkerInUseException: Couldn't start socket communication because worker number 0 is still in use. You may need to manually close a previously opened environment or use a different worker number.

This suggests that the same port number is being used for more than one Unity environment, thus causing the issue.

[Dionysus326]

I also got the same error: TypeError: __init__() got multiple values for argument 'use_visual' Is this has something to do with the multi-agents? I deleted all the other agents except one but still got the same error

[awjuliani]

Hi @Dionysus326

There have been some changes to the gym wrapper recently which may have made the DQN tutorial no longer compatible. I'll log this as a bug, and have someone on the team take a look.

[Dionysus326]

Okay, thanks for the information. If you can update the tutorial would be nice though. Anyway, thanks a lot.

[Chris-YoungJin-Jung]

Apologies for the bump, but is there any update on this issue?

[Ademord]

I also come here to bump that the Python API and the Gym Wrapper don't get enough love. Baselines isn't supporting TF2 and the docs are obviously out of date (should update to stable-baselines3, for example).

[ervteng]

Thank you for your comments. We’ve documented your feedback and will prioritize when appropriate.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 28 days. It will be closed in the next 14 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had activity in the last 42 days. If this issue is still valid, please ping a maintainer. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 28 days. It will be closed in the next 14 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had activity in the last 42 days. If this issue is still valid, please ping a maintainer. Thank you for your contributions.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.