region selection failed on size of 10000

[mikejiang]

It worked ok for the smaller region size.

(xstart, xend)
Out[155]: (15986, 25986)
(ystart, yend)
Out[156]: (59448, 69448)
vals3 = ds_remote[xstart:xend, ystart:yend]
Traceback (most recent call last):
  File "/home/wjiang2/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2910, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-153-2ab7845b65c0>", line 1, in <module>
    vals3 = ds_remote[xstart:xend, ystart:yend]
  File "/home/wjiang2/.local/lib/python3.6/site-packages/h5pyd/_hl/dataset.py", line 759, in __getitem__
    page_arr = numpy.reshape(arr1d, page_mshape)
  File "/app/python3/3.6.0/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 232, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "/app/python3/3.6.0/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
ValueError: cannot reshape array of size 10498 into shape (10000,10000)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/wjiang2/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2850, in run_ast_nodes
    if self.run_code(code, result):
  File "/home/wjiang2/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2927, in run_code
    self.showtraceback(running_compiled_code=True)
TypeError: showtraceback() got an unexpected keyword argument 'running_compiled_code'

[jreadey]

@mikejiang - can you provide the code that led up to the exception?

[mikejiang]

f_remote = h5pyd.File("/home/wjiang2/10x", "r")#bug in h5pyd: mode r is still writable
ds_remote = f_remote["/data"]
xstart, xend = 15986, 25986
ystart, yend = 59448, 69448
vals3 = ds_remote[xstart:xend, ystart:yend]

[jreadey]

This was failing because the selection size of 10000^2 * 4 ~ 400MB caused the container to run out of memory.
I've put in a check on the server that sends a 413 response (request too large) if the request size is too large (set to 100MB for now). When h5pyd gets the 413, it breaks up the read into smaller requests.

Looks like the above code is working now.