[FEATURE]: Increase performance/speed to Builder.build

ncar-xdev / ecgtools

ESM Catalog Generation tools

https://ecgtools.readthedocs.io

Apache License 2.0

9 stars 11 forks source link

[FEATURE]: Increase performance/speed to Builder.build #194

Open jsta opened 1 week ago

jsta commented 1 week ago

Is your feature request related to a problem?

I am trying to run Builder.build on a very large archive path and it takes a very long time.

Describe the solution you'd like

Is there a possibility of speeding up Builder.build by e.g. parallelizing the directory traversals?

Describe alternatives you've considered

No response

Additional context

No response

kuchaale commented 1 week ago

I am able to parse 6776 assets in 12 minutes using 4 workers which I find reasonable according to the fact that I open the NetCDF files to parse their attributes. Have you tried e.g. joblib_parallel_kwargs=dict(n_jobs=-1)?