search

NOC-OI / msm-os

A library to streamline the transfer, update, and deletion of Zarr files within object store environments.
MIT License
2 stars 1 forks source link

latest update seems to slow stuff down? #18

Open thopri opened 3 weeks ago

thopri commented 3 weeks ago

Hi,

I have pulled the latest update (with the skip verification check option) and reinstalled. I think it updated but the version number is the same? anyway I am finding it a lot slower than it was before even with the -si flag enabled.

Has anyone else found the same? It could environmental (more people using JASMIN right now) but would be good it see if anyone is in a similar position as I was hoping it would be quicker!

thopri commented 3 weeks ago

Oops meant to add it was taking like 10 mins for a monthly mean of grid T (eORCA12) but is now more like an hour.

EDIT: I have tried using an older version of msm-os the one I was using previously, it is also slow so I guess its something else slowing things down.... Will update if I find out what.

soutobias commented 2 days ago

Hi @thopri , I am sorry. I only saw your message today. Github only sends me notifications when someone cites me. Did you manage to identify what was the problem? I am not changing the version now because we are still in the dev mode (not in the main branch). It should be much faster now because you are skiping all the integrity checks. I will add some profile analysis in the code to identify the bottlenecks. Could you try to do the same? Like adding cProfile when you are running the script.

I will work in the next few days in the parallelisation. It will speed up the process.