Support HTTP requests with data larger than 100mb

[jreadey]

Currently very large read and writes will fail if it's larger than than the default 100mb request size limit in HSDS. Take advantage of the HSDS streaming feature to get around this.

[ron-kuhn]

@jreadey, FYI... I was have problems writing 1920x1080x16bit images (significantly < 100mb) to a HSDS dataset using vol-rest. I decrease the size of the images to 192x108x16bits and it worked. Didn't investigate further.

[ron-kuhn]

@jreadey, can you tell me where you are on this issue? I had to decrease the size of my image (shown above) to get this to work. This is OK for my work but I need to be able to support real images from the instrument. Is compression supported? Currently, my chunk size is one image.

[ron-kuhn]

@jreadey, I turned on debug for CURL in vol-rest and got the following during the PUT of the pixel data

PUT /datasets/d-810209f2-7efe76ec-795d-858051-3ed3a3/value?select=[0:1:1,0:1:1,0:1080:1,0:1920:1] HTTP/1.1 Host: localhost:5101 Authorization: Basic ZG9yYS1hZG1pbjpkb3JhMTIzNA== User-Agent: libhdf5/1.14.0 (Windows; HDF5 REST VOL v1.0.0) Accept: / X-Hdf-domain: /cell_data/20230412_195343_w0000p0000f0000.imgh5 Content-Type: application/octet-stream Content-Length: 4147200

• STATE: DO => DID handle 0x1f1b86c0ee8; line 2223 (connection #1)
• STATE: DID => PERFORMING handle 0x1f1b86c0ee8; line 2342 (connection #1)
• HTTP 1.1 or later with persistent connection < HTTP/1.1 200 OK < Server: Highly Scalable Data Service (HSDS) < X-XSS-Protection: 1; mode=block < Content-Type: application/json; charset=utf-8 < Content-Length: 2 < Date: Wed, 12 Apr 2023 19:58:22 GMT <
• Recv failure: Connection was reset
• [CONN-1] readwrite_data() -> 56
• [CONN-1] Curl_readwrite() -> 56
• multi_done: status: 56 prem: 1 done: 0
• multi_done, not re-using connection=1, forbid=0, close=1, premature=1, conn_multiplex=0
• The cache now contains 0 members
• Curl_disconnect(conn #1, dead=1)
• Closing connection 1
• Expire cleared (transfer 0x1f1b86c0ee8)

[mattjala]

After #26, the VOL should support arbitrarily large read/writes. It's likely that the bad behavior above 64kb was the source of @ron-kuhn's issue with large images.