I'm running into out of storage issues while training (the "PPO" algorithm) as follows:
2022-08-24 11:57:32,014 WARNING worker.py:1404 -- Traceback (most recent call last):
File "python/ray/_raylet.pyx", line 896, in ray._raylet.spill_objects_handler
File "python/ray/_raylet.pyx", line 899, in ray._raylet.spill_objects_handler
File "/home/xxx/anaconda3/envs/mujoco_env/lib/python3.7/site-packages/ray/external_storage.py", line 662, in spill_objects
return _external_storage.spill_objects(object_refs, owner_addresses)
File "/home/xxx/anaconda3/envs/mujoco_env/lib/python3.7/site-packages/ray/external_storage.py", line 303, in spill_objects
return self._write_multiple_objects(f, object_refs, owner_addresses, url)
File "/home/xxx/anaconda3/envs/mujoco_env/lib/python3.7/site-packages/ray/external_storage.py", line 151, in _write_multiple_objects
written_bytes = f.write(payload)
OSError: [Errno 28] No space left on device
An unexpected internal error occurred while the IO worker was spilling objects: [Errno 28] No space left on device
Approximately how much space is required for the "apex" project to train. thanks
I'm running into out of storage issues while training (the "PPO" algorithm) as follows:
2022-08-24 11:57:32,014 WARNING worker.py:1404 -- Traceback (most recent call last): File "python/ray/_raylet.pyx", line 896, in ray._raylet.spill_objects_handler File "python/ray/_raylet.pyx", line 899, in ray._raylet.spill_objects_handler File "/home/xxx/anaconda3/envs/mujoco_env/lib/python3.7/site-packages/ray/external_storage.py", line 662, in spill_objects return _external_storage.spill_objects(object_refs, owner_addresses) File "/home/xxx/anaconda3/envs/mujoco_env/lib/python3.7/site-packages/ray/external_storage.py", line 303, in spill_objects return self._write_multiple_objects(f, object_refs, owner_addresses, url) File "/home/xxx/anaconda3/envs/mujoco_env/lib/python3.7/site-packages/ray/external_storage.py", line 151, in _write_multiple_objects written_bytes = f.write(payload) OSError: [Errno 28] No space left on device An unexpected internal error occurred while the IO worker was spilling objects: [Errno 28] No space left on device
Approximately how much space is required for the "apex" project to train. thanks