search

ecmwf / eccodes-python

Python interface to the ecCodes GRIB/BUFR decoder/encoder
Apache License 2.0
113 stars 33 forks source link

Enabling thread safety using pip #83

Closed dhah229 closed 6 months ago

dhah229 commented 10 months ago

Is your feature request related to a problem? Please describe.

As mentioned in https://github.com/ecmwf/cfgrib/issues/110#issuecomment-705544248, I am unable to open multiple datasets with xarray.open_mfdataset() with parallel=True. From what I understand, we can install eccodes from conda to fix this problem. Is it possible to install thread safety enabled eccodes without using conda? I am currently using a docker container to process my work. If not what other alternatives do I have?

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Organisation

No response

martincjespersen commented 6 months ago

I am also in the need for being able to use parallel=True without using conda. Is there any status on this @shahramn ?

shahramn commented 6 months ago

If you build your own ecCodes, enable the mutexes for thread safety cmake /path/to/eccodes -D ENABLE_ECCODES_THREADS=ON

See https://confluence.ecmwf.int/display/ECC/ecCodes+installation