bystrogenomics / bystro

High Dimensional Machine Learning Toolkit with applications in Genetics & Proteomics
Apache License 2.0
43 stars 16 forks source link

[scheduler] Periodically touch beanstalkd python jobs #537

Closed akotlar closed 2 months ago

akotlar commented 3 months ago

Because of https://github.com/ray-project/ray/issues/46366, we use Python threading threads rather than ray workers to spawn the ray handler function. We will further investigate Ray memory usage characteristics with Arrow Datasets and make a longer term decision on whether to drop Ray entirely, or find a workaround for memory usage.