search

rapidsai / dask-cuda

Utilities for Dask and CUDA interactions
https://docs.rapids.ai/api/dask-cuda/stable/
Apache License 2.0
286 stars 91 forks source link

New version bug #1288

Open raphael7777777 opened 10 months ago

raphael7777777 commented 10 months ago

I had a code which ran a delayed function as a task. I would run this task with different inputs various times. For this reason i included gc.collects inside the function. When running, i could see in the dask dashboard that every task was taking progressively longer for no apparent reason. When i returned to the old version 23.8, everything worked normally. Even without the cg.collects inside the code, the running tasks showed some tasks (probably i assume when gc.collect ran by itself automatically) tasks would take a lot longer. Even thkugh most were taking the expected time, every 20 tasks there was one much longer, and also progressively longer

pentschev commented 10 months ago

Could you please share more details on the problem and your environment? Ideally a minimal reproducer, combined with the output of nvidia-smi and conda list (if using conda) or pip list (if using pip)?

raphael7777777 commented 10 months ago

Working on colab with a GPU A100. The problem i have already described above, unfortunately i cant show any code because its a proprietary algorithm. Its been solved by downgrading back to old version.

pentschev commented 10 months ago

Unfortunately that description is insufficient to provide any help. We don't need any proprietary pieces of your code, but if you would like us to be able to look at your problem we're going to need more than that, a minimal reproducer without any of your proprietary pieces would suffice.