Open cometta opened 2 months ago
Hi @cometta , welcome!
Are you on MacOS by chance? Object spilling is handed a little differently on MacOS.
@devin-petersohn , i tested on intel VM, run on Ubuntu
Ok great, in this case it should be straightforward to just use MODIN_MEMORY
environment variable or config. Link to how you can use these is here: https://modin.readthedocs.io/en/stable/flow/modin/config.html#modin-configs-list
Let me know if that helps!
i tried os.environ["MODIN_MEMORY"] = "30000000000"
but when i monitor memory , the usage is still use more than 64GB and then i get ray.exceptions.WorkerCrashedError: The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
.does Modin use so much memory to load parquet files?
@cometta, take a look at this comment https://github.com/modin-project/modin/issues/7020#issuecomment-2003264663.
my code hang , can advice what i miss out?
modin 0.31.0 modin-spreadsheet 0.1.2 ray 2.32.0
output
extra info, i see memory usage is maxed out. 64GB. this might be related and caused the hang. any option i need to set to modin ?