search

HDFGroup / h5pyd

h5py distributed - Python client library for HDF Rest API
Other
109 stars 39 forks source link

Field selection for simple compound types #173

Closed mattjala closed 3 months ago

mattjala commented 3 months ago

This currently has a massive performance issue with field reads. Reading 144 bytes in a field selection takes over a minute. The stall occurs when rsp.iter_content is invoked in base.py. Other calls that try to directly interact with the response, like rsp.text or rsp.content have a similar delay.

I tried manually specifying the utf-8 encoding for the response, since apparently having to try a bunch of encodings when one isn't specifying can sometimes cause slowdown like this, but it had no effect.

jreadey commented 3 months ago

It seems that h5pyd is trying to read more bytes than was actually sent. See this PR: https://github.com/HDFGroup/hsds/pull/331 - that should fix the issue.

mattjala commented 3 months ago

It seems that h5pyd is trying to read more bytes than was actually sent. See this PR: HDFGroup/hsds#331 - that should fix the issue.

This fixed the slowdown - this PR is ready for review now.