JudasDie
commented
1 year ago hi！When i run tune-sot.py, it reports:
(pid=84596) 2022-12-01 19:00:31,528 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84596) Traceback (most recent call last): (pid=84596) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84596) actor_class = pickle.loads(pickled_class) (pid=84596) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84596) import(name) (pid=84596) ModuleNotFoundError: No module named 'tracker' (pid=84597) 2022-12-01 19:00:31,513 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84597) Traceback (most recent call last): (pid=84597) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84597) actor_class = pickle.loads(pickled_class) (pid=84597) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84597) import(name) (pid=84597) ModuleNotFoundError: No module named 'tracker' (pid=84599) 2022-12-01 19:00:31,525 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84599) Traceback (most recent call last): (pid=84599) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84599) actor_class = pickle.loads(pickled_class) (pid=84599) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84599) import(name) (pid=84599) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:31,564 ERROR trial_runner.py:523 -- Trial fitness_5f069808: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84596, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. 2022-12-01 19:00:31,570 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 676d0b440100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
2022-12-01 19:00:31,586 ERROR trial_runner.py:523 -- Trial fitness_5f0f687a: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84599, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. 2022-12-01 19:00:31,616 ERROR trial_runner.py:523 -- Trial fitness_5f3270ea: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84601, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. (pid=84601) 2022-12-01 19:00:31,567 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84601) Traceback (most recent call last): (pid=84601) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84601) actor_class = pickle.loads(pickled_class) (pid=84601) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84601) import(name) (pid=84601) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:31,857 ERROR trial_runner.py:523 -- Trial fitness_5f40249c: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84626, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. 2022-12-01 19:00:31,860 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 552068930100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
(pid=84626) 2022-12-01 19:00:31,850 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84626) Traceback (most recent call last): (pid=84626) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84626) actor_class = pickle.loads(pickled_class) (pid=84626) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84626) import(name) (pid=84626) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:31,922 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID adb556570100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
2022-12-01 19:00:31,926 ERROR trial_runner.py:523 -- Trial fitness_5f744182: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84611, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. 2022-12-01 19:00:31,949 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 2f6e56200100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
2022-12-01 19:00:31,954 ERROR trial_runner.py:523 -- Trial fitness_5f78b94c: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84610, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. (pid=84610) 2022-12-01 19:00:31,941 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84610) Traceback (most recent call last): (pid=84610) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84610) actor_class = pickle.loads(pickled_class) (pid=84610) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84610) import(name) (pid=84610) ModuleNotFoundError: No module named 'tracker' (pid=84611) 2022-12-01 19:00:31,919 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84611) Traceback (most recent call last): (pid=84611) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84611) actor_class = pickle.loads(pickled_class) (pid=84611) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84611) import(name) (pid=84611) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:31,977 ERROR trial_runner.py:523 -- Trial fitness_5f7c0872: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84629, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. 2022-12-01 19:00:31,985 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID c39997530100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
(pid=84629) 2022-12-01 19:00:31,971 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84629) Traceback (most recent call last): (pid=84629) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84629) actor_class = pickle.loads(pickled_class) (pid=84629) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84629) import(name) (pid=84629) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:32,220 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 98ba875f0100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
2022-12-01 19:00:32,225 ERROR trial_runner.py:523 -- Trial fitness_5f80b8b8: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84607, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. (pid=84607) 2022-12-01 19:00:32,217 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84607) Traceback (most recent call last): (pid=84607) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84607) actor_class = pickle.loads(pickled_class) (pid=84607) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84607) import(name) (pid=84607) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:32,300 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 842595fe0100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
2022-12-01 19:00:32,303 ERROR trial_runner.py:523 -- Trial fitness_5fa5f254: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84608, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. (pid=84608) 2022-12-01 19:00:32,297 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84608) Traceback (most recent call last): (pid=84608) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84608) actor_class = pickle.loads(pickled_class) (pid=84608) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84608) import(name) (pid=84608) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:32,625 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 688d7fcd0100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
2022-12-01 19:00:32,625 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 3261819f0100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
I have run many times, somtimes it shows ModuleNotFoundError: No module named 'utils' I have tied many ways, like changing the version of ray for 0.8.4 to 0.6.0 or appending the utils to system path. it still didn't work. I'm not sure what causes this problem.
thank you, expect your reply!
I have met this problem years before. And rerunning several times could solve the problem.
hi！When i run tune-sot.py, it reports:
(pid=84596) 2022-12-01 19:00:31,528 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84596) Traceback (most recent call last): (pid=84596) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84596) actor_class = pickle.loads(pickled_class) (pid=84596) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84596) import(name) (pid=84596) ModuleNotFoundError: No module named 'tracker' (pid=84597) 2022-12-01 19:00:31,513 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84597) Traceback (most recent call last): (pid=84597) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84597) actor_class = pickle.loads(pickled_class) (pid=84597) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84597) import(name) (pid=84597) ModuleNotFoundError: No module named 'tracker' (pid=84599) 2022-12-01 19:00:31,525 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84599) Traceback (most recent call last): (pid=84599) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84599) actor_class = pickle.loads(pickled_class) (pid=84599) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84599) import(name) (pid=84599) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:31,564 ERROR trial_runner.py:523 -- Trial fitness_5f069808: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84596, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. 2022-12-01 19:00:31,570 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 676d0b440100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
2022-12-01 19:00:31,586 ERROR trial_runner.py:523 -- Trial fitness_5f0f687a: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84599, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. 2022-12-01 19:00:31,616 ERROR trial_runner.py:523 -- Trial fitness_5f3270ea: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84601, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. (pid=84601) 2022-12-01 19:00:31,567 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84601) Traceback (most recent call last): (pid=84601) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84601) actor_class = pickle.loads(pickled_class) (pid=84601) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84601) import(name) (pid=84601) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:31,857 ERROR trial_runner.py:523 -- Trial fitness_5f40249c: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84626, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. 2022-12-01 19:00:31,860 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 552068930100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
(pid=84626) 2022-12-01 19:00:31,850 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84626) Traceback (most recent call last): (pid=84626) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84626) actor_class = pickle.loads(pickled_class) (pid=84626) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84626) import(name) (pid=84626) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:31,922 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID adb556570100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
2022-12-01 19:00:31,926 ERROR trial_runner.py:523 -- Trial fitness_5f744182: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84611, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. 2022-12-01 19:00:31,949 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 2f6e56200100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
2022-12-01 19:00:31,954 ERROR trial_runner.py:523 -- Trial fitness_5f78b94c: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84610, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. (pid=84610) 2022-12-01 19:00:31,941 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84610) Traceback (most recent call last): (pid=84610) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84610) actor_class = pickle.loads(pickled_class) (pid=84610) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84610) import(name) (pid=84610) ModuleNotFoundError: No module named 'tracker' (pid=84611) 2022-12-01 19:00:31,919 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84611) Traceback (most recent call last): (pid=84611) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84611) actor_class = pickle.loads(pickled_class) (pid=84611) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84611) import(name) (pid=84611) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:31,977 ERROR trial_runner.py:523 -- Trial fitness_5f7c0872: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84629, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. 2022-12-01 19:00:31,985 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID c39997530100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
(pid=84629) 2022-12-01 19:00:31,971 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84629) Traceback (most recent call last): (pid=84629) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84629) actor_class = pickle.loads(pickled_class) (pid=84629) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84629) import(name) (pid=84629) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:32,220 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 98ba875f0100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
2022-12-01 19:00:32,225 ERROR trial_runner.py:523 -- Trial fitness_5f80b8b8: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84607, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. (pid=84607) 2022-12-01 19:00:32,217 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84607) Traceback (most recent call last): (pid=84607) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84607) actor_class = pickle.loads(pickled_class) (pid=84607) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84607) import(name) (pid=84607) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:32,300 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 842595fe0100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
2022-12-01 19:00:32,303 ERROR trial_runner.py:523 -- Trial fitness_5fa5f254: Error processing event. Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial result = self.trial_executor.fetch_result(trial) File "/root/miniconda3/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/root/miniconda3/lib/python3.8/site-packages/ray/worker.py", line 1538, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=84608, ip=172.17.0.7) File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method. (pid=84608) 2022-12-01 19:00:32,297 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc. (pid=84608) Traceback (most recent call last): (pid=84608) File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs (pid=84608) actor_class = pickle.loads(pickled_class) (pid=84608) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport (pid=84608) import(name) (pid=84608) ModuleNotFoundError: No module named 'tracker' 2022-12-01 19:00:32,625 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 688d7fcd0100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
2022-12-01 19:00:32,625 WARNING worker.py:1134 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 3261819f0100. Traceback: Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs actor_class = pickle.loads(pickled_class) File "/root/miniconda3/lib/python3.8/site-packages/ray/cloudpickle/cloudpickle.py", line 1096, in subimport import(name) ModuleNotFoundError: No module named 'tracker'
I have run many times, somtimes it shows ModuleNotFoundError: No module named 'utils' I have tied many ways, like changing the version of ray for 0.8.4 to 0.6.0 or appending the utils to system path. it still didn't work. I'm not sure what causes this problem.
thank you, expect your reply!