search

google-research / reincarnating_rl

[NeurIPS 2022] Open source code for reusing prior computational work in RL.
https://agarwl.github.io/reincarnating_rl
Apache License 2.0
91 stars 11 forks source link

TypeError: file must have 'read', 'readinto' and 'readline' attributes #4

Open goinouego opened 11 months ago

goinouego commented 11 months ago

Hello. This is my first post, so please excuse me if there are any mistakes.

The commands used are as follows.

cd reincarnating_rl
python -um reincarnating_rl.train \
  --agent qdagger_rainbow \
  --gin_files reincarnating_rl/configs/qdagger_rainbow.gin
  --base_dir /tmp/qdagger_rainbow \
  --teacher_checkpoint_dir $TEACHER_CKPT_DIR/Breakout/1 \
  --teacher_checkpoint_number 399
  --run_number=1 \
  --atari_roms_path=/tmp/atari_roms \
  --alsologtostderr

・'atari_roms_path' was not listed, so I added it myself.

When I run the code, the following error occurs.

 2023-10-04 13:09:32.049601: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-10-04 13:09:33.558113: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/mlab/tensei2/lib/python3.8/site-packages/jax/_src/api_util.py:229: SyntaxWarning: Jitted function has invalid argnames {'distill_type', 'distill_loss_coefficient', 'distill_best_action_only', 'distill_temperature'} in static_argnames. Function does not take these args.This warning will be replaced by an error after 2022-08-20 at the earliest.
  warnings.warn(f"Jitted function has invalid argnames {invalid_argnames} "
2023-10-04 13:09:38.179840: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-10-04 13:09:38.183743: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1960] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
I1004 13:09:38.941833 140563993347904 train.py:125] Setting random seed: 1
W1004 13:09:41.145547 140563993347904 run_experiment.py:267] DEPRECATION WARNING: Logger is being deprecated. Please switch to CollectorDispatcher!
/home/mlab/tensei2/lib/python3.8/site-packages/gym/envs/registration.py:593: UserWarning: WARN: The environment PongNoFrameskip-v0 is out of date. You should consider upgrading to version `v4`.
  logger.warn(
A.L.E: Arcade Learning Environment (version 0.7.5+db37282)
[Powered by Stella]
/home/mlab/tensei2/lib/python3.8/site-packages/gym/core.py:317: DeprecationWarning: WARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.
  deprecation(
/home/mlab/tensei2/lib/python3.8/site-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: WARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.
  deprecation(
I1004 13:09:43.154085 140563993347904 reincarnation_rainbow_agent.py:43] Creating ReincarnationRainbowAgent agent with the following parameters:
I1004 13:09:43.154490 140563993347904 reincarnation_rainbow_agent.py:45]         num_updates_per_persistent_step: 1
I1004 13:09:43.154817 140563993347904 reincarnation_rainbow_agent.py:47]         data_augmentation: False
I1004 13:09:43.155117 140563993347904 reincarnation_rainbow_agent.py:48]         network: <class 'dopamine.jax.networks.FullRainbowNetwork'>
I1004 13:09:43.752246 140563993347904 full_rainbow_agent.py:224] Creating QDaggerRainbowAgent agent with the following parameters:
I1004 13:09:43.753766 140563993347904 full_rainbow_agent.py:226]         double_dqn: False
I1004 13:09:43.754319 140563993347904 full_rainbow_agent.py:227]         noisy_networks: False
I1004 13:09:43.755004 140563993347904 full_rainbow_agent.py:228]         dueling_dqn: False
I1004 13:09:43.755769 140563993347904 full_rainbow_agent.py:229]         distributional: True
I1004 13:09:43.756666 140563993347904 full_rainbow_agent.py:230]         mse_loss: 0
I1004 13:09:43.757438 140563993347904 full_rainbow_agent.py:231]         num_atoms: 51
I1004 13:09:43.758368 140563993347904 full_rainbow_agent.py:232]         replay_scheme: prioritized
I1004 13:09:43.759295 140563993347904 full_rainbow_agent.py:233]         num_updates_per_train_step: 1
I1004 13:09:43.814832 140563993347904 xla_bridge.py:603] Unable to initialize backend 'cuda': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
I1004 13:09:43.815029 140563993347904 xla_bridge.py:603] Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
I1004 13:09:43.816573 140563993347904 xla_bridge.py:603] Unable to initialize backend 'tpu': INVALID_ARGUMENT: TpuPlatform is not available.
W1004 13:09:43.816766 140563993347904 xla_bridge.py:617] No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
I1004 13:09:43.989467 140563993347904 dqn_agent.py:278] Creating QDaggerRainbowAgent agent with the following parameters:
I1004 13:09:43.989721 140563993347904 dqn_agent.py:280]          gamma: 0.990000
I1004 13:09:43.989921 140563993347904 dqn_agent.py:281]          update_horizon: 3.000000
I1004 13:09:43.990098 140563993347904 dqn_agent.py:282]          min_replay_history: 20000
I1004 13:09:43.990217 140563993347904 dqn_agent.py:283]          update_period: 4
I1004 13:09:43.990343 140563993347904 dqn_agent.py:284]          target_update_period: 8000
I1004 13:09:43.990493 140563993347904 dqn_agent.py:285]          epsilon_train: 0.010000
I1004 13:09:43.990720 140563993347904 dqn_agent.py:286]          epsilon_eval: 0.001000
I1004 13:09:43.990818 140563993347904 dqn_agent.py:287]          epsilon_decay_period: 50000
I1004 13:09:43.990914 140563993347904 dqn_agent.py:288]          optimizer: adam
I1004 13:09:43.991011 140563993347904 dqn_agent.py:289]          seed: 1
I1004 13:09:43.991108 140563993347904 dqn_agent.py:290]          loss_type: mse
I1004 13:09:43.991207 140563993347904 dqn_agent.py:291]          preprocess_fn: <function preprocess_inputs_with_augmentation at 0x7fd6d206a940>
I1004 13:09:43.991539 140563993347904 dqn_agent.py:292]          summary_writing_frequency: 25000
I1004 13:09:43.991641 140563993347904 dqn_agent.py:293]          allow_partial_reload: False
I1004 13:09:45.225142 140563993347904 circular_replay_buffer.py:157] Creating a OutOfGraphPrioritizedReplayBuffer replay memory with the following parameters:
I1004 13:09:45.225919 140563993347904 circular_replay_buffer.py:160]     observation_shape: (84, 84)
I1004 13:09:45.226537 140563993347904 circular_replay_buffer.py:161]     observation_dtype: <class 'jax.numpy.uint8'>
I1004 13:09:45.227458 140563993347904 circular_replay_buffer.py:162]     terminal_dtype: <class 'numpy.uint8'>
I1004 13:09:45.228268 140563993347904 circular_replay_buffer.py:163]     stack_size: 4
I1004 13:09:45.229049 140563993347904 circular_replay_buffer.py:164]     replay_capacity: 1000000
I1004 13:09:45.229805 140563993347904 circular_replay_buffer.py:165]     batch_size: 32
I1004 13:09:45.230160 140563993347904 circular_replay_buffer.py:166]     update_horizon: 3
I1004 13:09:45.230264 140563993347904 circular_replay_buffer.py:167]     gamma: 0.990000
I1004 13:09:45.230487 140563993347904 circular_replay_buffer.py:168]     checkpoint_duration: 4
I1004 13:09:45.230724 140563993347904 circular_replay_buffer.py:169]     keep_every: None
I1004 13:10:13.732142 140563993347904 loss_helpers.py:83] Creating Adam optimizer with settings lr=0.000063, beta1=0.900000, beta2=0.999000, eps=0.000150
I1004 13:10:14.991514 140563993347904 atari_100k_rainbow_agent.py:159]   data_augmentation: False
I1004 13:10:14.992304 140563993347904 atari_100k_rainbow_agent.py:162]   data_augmentation: False
I1004 13:10:14.993070 140563993347904 qdagger_rainbow_agent.py:158] Creating QDaggerRainbowAgent agent with the following parameters:
I1004 13:10:14.993680 140563993347904 qdagger_rainbow_agent.py:160]      distill_decay_period: 2500000
I1004 13:10:14.994260 140563993347904 qdagger_rainbow_agent.py:161]      distill_loss_coefficient: 1.0000
I1004 13:10:14.994835 140563993347904 qdagger_rainbow_agent.py:162]      distill_best_action_only: 0.0000
I1004 13:10:14.995398 140563993347904 qdagger_rainbow_agent.py:163]      distill_temperature: 0.1000
I1004 13:10:14.995963 140563993347904 qdagger_rainbow_agent.py:164]      method_type: 1
Traceback (most recent call last):
  File "/home/mlab/.pyenv/versions/3.8.0/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/mlab/.pyenv/versions/3.8.0/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/mlab/.vscode/extensions/ms-python.python-2023.6.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/home/mlab/.vscode/extensions/ms-python.python-2023.6.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/home/mlab/.vscode/extensions/ms-python.python-2023.6.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/home/mlab/.vscode/extensions/ms-python.python-2023.6.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/home/mlab/.vscode/extensions/ms-python.python-2023.6.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/home/mlab/.vscode/extensions/ms-python.python-2023.6.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/home/mlab/tensei2/reincarnating_rl-master/train.py", line 173, in <module>
    app.run(main)
  File "/home/mlab/tensei2/lib/python3.8/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/home/mlab/tensei2/lib/python3.8/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/home/mlab/tensei2/reincarnating_rl-master/train.py", line 161, in main
    runner = runner_fn(
  File "/home/mlab/tensei2/lib/python3.8/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/home/mlab/tensei2/lib/python3.8/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/home/mlab/tensei2/lib/python3.8/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/home/mlab/tensei2/reincarnating_rl-master/reincarnating_rl/run_experiment.py", line 301, in __init__
    super().__init__(
  File "/home/mlab/tensei2/lib/python3.8/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/home/mlab/tensei2/lib/python3.8/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/home/mlab/tensei2/lib/python3.8/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/home/mlab/tensei2/reincarnating_rl-master/reincarnating_rl/run_experiment.py", line 76, in __init__
    super().__init__(
  File "/home/mlab/tensei2/lib/python3.8/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/home/mlab/tensei2/lib/python3.8/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/home/mlab/tensei2/lib/python3.8/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/home/mlab/tensei2/lib/python3.8/site-packages/dopamine/discrete_domains/run_experiment.py", line 235, in __init__
    self._initialize_checkpointer_and_maybe_resume(checkpoint_file_prefix)
  File "/home/mlab/tensei2/reincarnating_rl-master/reincarnating_rl/run_experiment.py", line 116, in _initialize_checkpointer_and_maybe_resume
    experiment_data = self._checkpointer.load_checkpoint(checkpoint_version)
  File "/home/mlab/tensei2/lib/python3.8/site-packages/dopamine/discrete_domains/checkpointer.py", line 214, in load_checkpoint
    return self._load_data_from_file(checkpoint_file)
  File "/home/mlab/tensei2/lib/python3.8/site-packages/dopamine/discrete_domains/checkpointer.py", line 193, in _load_data_from_file
    return pickle.load(fin)
TypeError: file must have 'read', 'readinto' and 'readline' attributes
  In call to configurable 'Runner' (<class 'dopamine.discrete_domains.run_experiment.Runner'>)
  In call to configurable 'RunnerWithTeacher' (<class 'reincarnating_rl.run_experiment.RunnerWithTeacher'>)
  In call to configurable 'ReincarnationRunner' (<class 'reincarnating_rl.run_experiment.ReincarnationRunner'>)

I get the error: Typeerror: file must have 'read', 'readinto' and 'readline' attributes.

My system info: ・Ubuntu 18.04 ・Python 3.8.0 ・I installed dependencies from requirements.txt. ・All checkpoint files have been downloaded and placed in a directory named DQN_400 at the same level as the README file.

I would appreciate it if you could reply.

sdpkjc commented 10 months ago

I've also come across some quirky bugs in the latest version 😬, which I suspect got slipped in during code refactoring.

Going through the init commit from git, it appeared a tad more accurate 🕵️.

May I suggest you to cast your eyes on distillation_dqn_agent.py and persistent_dqn_agent.py? They might just lend you a helping hand 🤝.

By the way, CleanRL's QDagger(code & docs) is quite straightforward to reproduce🚀. I highly recommend giving it a try!

agarwl commented 7 months ago

Yes, this might have happened during refactoring and I'd also suggest looking at CleanRL's code for a runnable version. The code here can be useful as a reference for implementation.