HDF5-DIAG: Error detected in HDF5

[IreneNA95]

Hi everyone. I am having a problem when trying to run a simulation using reader_netCDF_CF_generic.Reader. My initial call to the reader incorporates the current velocity field, and subsequently, imposes the waves. I concatenate several files corresponding to 92 days as follows:

However, when I run the simulation, I get the following error:

HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:

When I run the simulation with one reader at a time, it works as intended. Strikingly, the number of concatenated files seems to be related to the problem, since when fewer files are concatenated, for example "202103*.nc" (which calls 31 files per variable instead of 92), the simulation also runs successfully with both sets of data. I checked that the number of files of currents and waves is the same in both cases when reading the two sets of files.

Could someone give me some advice on how to fix this issue?

[gauteh]

Maybe your netcdf and hdf5 library are built or run with out thread-safety? How have you installed these packages?

tor. 18. apr. 2024, 12:40 skrev IreneNA @.***>:

Hi everyone. I am having a problem when trying to run a simulation using reader_netCDF_CF_generic.Reader. My initial call to the reader incorporates the current velocity field, and subsequently, imposes the waves. I concatenate several files corresponding to 92 days as follows:

image.png (view on web) https://github.com/OpenDrift/opendrift/assets/132444113/6f347ec4-500d-427c-869a-4ba36b2764c4

However, when I run the simulation, I get the following error:

HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1: image.png (view on web) https://github.com/OpenDrift/opendrift/assets/132444113/7b01af5c-28db-4df6-bd92-242e5d3a6196

When I run the simulation with one reader at a time, it works as intended. Strikingly, the number of concatenated files seems to be related to the problem, since when fewer files are concatenated, for example "202103*.nc" (which calls 31 files per variable instead of 92), the simulation also runs successfully with both sets of data. I checked that the number of files of currents and waves is the same in both cases when reading the two sets of files.

Could someone give me some advice on how to fix this issue?

— Reply to this email directly, view it on GitHub https://github.com/OpenDrift/opendrift/issues/1282, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAN364SO4FYPTDBB7LXBY3Y56PKTAVCNFSM6AAAAABGNABAZGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI2TAMZUHE2DIMA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[IreneNA95]

Hi, Thank you for your quick response. How could I check that? I apologize if the question is too basic, I am still a novice of the software. I checked the configuration, and it seems that the version of the libraries is not thread-safe.

The netcdf4 and hdf5 were already installed on our server when I started using OpenDrift.

[gauteh]

Hi,

That can be difficult for us to debug. Maybe you can setup your own micromamba like in the opendrift instructions and see if there is a difference.

Gaute

tor. 18. apr. 2024, 13:00 skrev IreneNA @.***>:

Hi, Thank you for your quick response. How could I check that? I apologize if the question is too basic, I am still a novice of the software. I checked the configuration, and it seems that the version of the libraries is not thread-safe.

image.png (view on web) https://github.com/OpenDrift/opendrift/assets/132444113/325f7236-1004-4028-924b-a616896a9603

image.png (view on web) https://github.com/OpenDrift/opendrift/assets/132444113/26d2a40f-6334-4a50-8b17-a1849e7ad6d7

The netcdf4 and hdf5 were already installed on our server when I started OpenDrift.

— Reply to this email directly, view it on GitHub https://github.com/OpenDrift/opendrift/issues/1282#issuecomment-2063594998, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAN36ZLQ2S6TRR32S2AD5DY56RUBAVCNFSM6AAAAABGNABAZGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRTGU4TIOJZHA . You are receiving this because you commented.Message ID: @.***>

[IreneNA95]

Hi all, I discovered that the error was related to some internal limit on the maximum number of NetCDF files that may be simultaneously opened or listed. In my case, I had one file per day for a year, but the list limit was 50 files. So I solved the issue by concatenating the daily files into monthly files. This obviously resulted in larger files, but file size does not seem to be a limit for OpenDrift. I guess having more than 50 monthly files would lead to the same problem, but that is not my case (for now). This solution may not apply to other people having the same issue, as it probably has more to do with the libraries I had already installed than to an error in the OpenDrift code itself. But I write it in case it might help someone else. I close this issue 🙂 Irene

[knutfrode]

Good, thank you for sharing this, it might be useful reference for others in the future.