kachadx
commented
3 months ago 使用环境变量RAY_max_grpc_message_size=1073741824
Closed kachadx closed 3 months ago
kachadx
commented
3 months ago 使用环境变量RAY_max_grpc_message_size=1073741824
ray.tune.error.TuneError: The Trainable/training function is too large for grpc resource limit. Check that its definition is not implicitly capturing a large array or other object in scope. Tip: use tune.with_parameters() to put large objects in the Ray object store. Original exception: Traceback (most recent call last): File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray\tune\experiment\experiment.py", line 150, in init self._run_identifier = Experiment.register_if_needed(run) File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray\tune\experiment\experiment.py", line 352, in register_if_needed register_trainable(name, run_object) File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray\tune\registry.py", line 112, in register_trainable _global_registry.register(TRAINABLE_CLASS, name, trainable) File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray\tune\registry.py", line 239, in register self.flush_values() File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray\tune\registry.py", line 277, in flush_values _internal_kv_put( File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray_private\client_mode_hook.py", line 103, in wrapper return func(*args, **kwargs) File "C:\ProgramData\anaconda3\envs\myenv\lib\site-packages\ray\experimental\internal_kv.py", line 94, in _internal_kv_put return global_gcs_client.internal_kv_put(key, value, overwrite, namespace) == 0 File "python\ray_raylet.pyx", line 2690, in ray._raylet._auto_reconnect.wrapper File "python\ray_raylet.pyx", line 2667, in ray._raylet._auto_reconnect.wrapper File "python\ray_raylet.pyx", line 2804, in ray._raylet.GcsClient.internal_kv_put File "python\ray_raylet.pyx", line 580, in ray._raylet.check_status ray.exceptions.RpcError: Sent message larger than max (845965369 vs. 536870912)