search

HDFGroup / h5pyd

h5py distributed - Python client library for HDF Rest API
Other
109 stars 39 forks source link

WIP: New HTTPConn for each obj in file #195

Closed mattjala closed 1 month ago

mattjala commented 1 month ago

MultiManager generally needs multiple HttpConns in order to provide a performance boost. For multiple dsets in a single file (a common use case), the MultiManager's effectiveness was limited.

In order to avoid issues in tests with group creation/deletion not being detected, the cache must be disabled when those files are opened. For that reason, this should probably be considered a breaking API change.

mattjala commented 1 month ago

Testing on the icesat2 benchmark didn't show any changes in performance from this. It's simpler just to keep one connection per file.