search

ray-project / kuberay

A toolkit to run Ray applications on Kubernetes
Apache License 2.0
963 stars 328 forks source link

[Bug] Minimum CPU and Memory requirements for KubeRay Head and worker pods #2186

Open rajendra-avesha opened 2 weeks ago

rajendra-avesha commented 2 weeks ago

Search before asking

KubeRay Component

ray-operator

What happened + What you expected to happen

I am trying to find the minimum CPU and memory requirements for Ray Head and Worker Pods of Ray Cluster to having RayServe functional

Currently I tried with Head CPU : 500m Head Memory : 500Mi and Worker CPU : 1000m and Worker memory : 1Gi

Reproduction script

2024-06-11 04:31:39,264 INFO usage_lib.py:449 -- Usage stats collection is enabled by default without user confirmation because this terminal is detected to be non-interactive. To disable this, add --disable-usage-stats to the command that starts the cluster, or run the following command: ray disable-usage-stats before starting the cluster. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details. 2024-06-11 04:31:39,265 INFO scripts.py:744 -- Local node IP: 10.11.1.160 Traceback (most recent call last): File "/home/ray/anaconda3/bin/ray", line 8, in sys.exit(main()) File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/scripts/scripts.py", line 2498, in main return cli() File "/home/ray/anaconda3/lib/python3.8/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/home/ray/anaconda3/lib/python3.8/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/home/ray/anaconda3/lib/python3.8/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ray/anaconda3/lib/python3.8/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/ray/anaconda3/lib/python3.8/site-packages/click/core.py", line 783, in invoke return __callback(args, *kwargs) File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/autoscaler/_private/cli_logger.py", line 856, in wrapper return f(args, **kwargs) File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/scripts/scripts.py", line 771, in start node = ray._private.node.Node( File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/node.py", line 309, in init self.start_ray_processes() File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/node.py", line 1362, in start_ray_processes ) = ray._private.services.determine_plasma_store_config( File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/services.py", line 2035, in determine_plasma_store_config raise ValueError( ValueError: Attempting to cap object store memory usage at 74843750 bytes, but the minimum allowed is 78643200 bytes.

Anything else

No response

Are you willing to submit a PR?