Unable to open object (Component not found)

bwulff commented 5 years ago

have a file f3.h5 with s single uint8 dataset called '/raw'
I start the h5serv docker image and it finds f3.h5 while building the toc

when I try to access the dataset in python, I get a KeyError:

>>> import h5pyd as h5py
>>> a = h5py.File('f3.hdfgroup.org',mode='r',endpoint='http://localhost:5001')
>>> a['/raw']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/h5pyd/_hl/group.py", line 397, in __getitem__
parent_uuid, link_json = self.get_link_json(name)
File "/usr/local/lib/python2.7/dist-packages/h5pyd/_hl/group.py", line 303, in get_link_json
raise KeyError("Unable to open object (Component not found)")
KeyError: 'Unable to open object (Component not found)'

...am I missing something? I looked at the read_example.py in the examples folder of this repo and it seemed to me it should work like above.

When I try to get the items that are in there I get

>>> a.items()
[(u'f3', <HDF5 group "/" (1 members)>)]

but both seem to be infinite loops:

>>> a.items()
[(u'f3', <HDF5 group "/" (1 members)>)]
>>> a['f3'].items()
[(u'f3', <HDF5 group "/" (1 members)>)]
>>> a['f3']['f3'].items()
[(u'f3', <HDF5 group "/" (1 members)>)]
>>> a['f3']['f3']['f3'].items()
[(u'f3', <HDF5 group "/" (1 members)>)]
>>> a['f3']['f3']['f3']['f3'].items()
[(u'f3', <HDF5 group "/" (1 members)>)]

>>> a.items()
[(u'f3', <HDF5 group "/" (1 members)>)]
>>> a['/'].items()
[(u'f3', <HDF5 group "/" (1 members)>)]
>>> a['/']['/'].items()
[(u'f3', <HDF5 group "/" (1 members)>)]
>>> a['/']['/']['/'].items()
[(u'f3', <HDF5 group "/" (1 members)>)]
>>> a['/']['/']['/']['/'].items()
[(u'f3', <HDF5 group "/" (1 members)>)]
>>>

bwulff commented 5 years ago

is this maybe in issue with hdfserv rather then h5pyd?

bwulff commented 5 years ago

Ok, another try:

cloned h5serv
compiled the Docker image using the Dockerfile included in the h5serv project
there is a demo file (tall.h5) in there which is put into the image --> when you start the image without mounting an external data directory, h5serv uses the data directory in the image, containing tall.h5
installed h5json und h5pyd

in the console:

/code/h5serv/data/public $ h5ls -r tall.h5 
/                        Group
/g1                      Group
/g1/g1.1                 Group
/g1/g1.1/dset1.1.1       Dataset {10, 10}
/g1/g1.1/dset1.1.2       Dataset {20}
/g1/g1.2                 Group
/g1/g1.2/extlink         External Link {somefile//somepath}
/g1/g1.2/g1.2.1          Group
/g1/g1.2/g1.2.1/slink    Soft Link {somevalue}
/g2                      Group
/g2/dset2.1              Dataset {10}
/g2/dset2.2              Dataset {3, 5}

...so it seems to be the file used in the h5pyd tests

then in python:

Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import h5pyd as h5py
>>> db = h5py.File("tall.hdfgroup.org","r",endpoint="http://localhost:5000")
>>> db.items()
[(u'tall', <HDF5 group "/" (2 members)>), (u'public', <HDF5 group "/public" (1 members)>)]
>>> db['tall']
<HDF5 group "/" (2 members)>
>>> db['tall'].items()
[(u'tall', <HDF5 group "/" (2 members)>), (u'public', <HDF5 group "/public" (1 members)>)]
>>> db['tall'].keys()
[u'tall', u'public']
>>> db.name
'/'
>>> db.id.id
'0a112e14-4329-11e9-bfcf-0242ac110002'
>>> g2 = db['g2']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/h5pyd/_hl/group.py", line 450, in __getitem__
parent_uuid, link_json = self.get_link_json(name)
File "/usr/local/lib/python2.7/dist-packages/h5pyd/_hl/group.py", line 350, in get_link_json
raise KeyError("Unable to open object (Component not found)")
KeyError: 'Unable to open object (Component not found)'
>>> ds = db['g1/g1.1/dset1.1.1']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/h5pyd/_hl/group.py", line 450, in __getitem__
parent_uuid, link_json = self.get_link_json(name)
File "/usr/local/lib/python2.7/dist-packages/h5pyd/_hl/group.py", line 350, in get_link_json
raise KeyError("Unable to open object (Component not found)")
KeyError: 'Unable to open object (Component not found)'
>>> ds = db['/g1/g1.1/dset1.1.1']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/h5pyd/_hl/group.py", line 450, in __getitem__
parent_uuid, link_json = self.get_link_json(name)
File "/usr/local/lib/python2.7/dist-packages/h5pyd/_hl/group.py", line 350, in get_link_json
raise KeyError("Unable to open object (Component not found)")
KeyError: 'Unable to open object (Component not found)'
>>>

The above steps are taken from examples/read_example.py.

jreadey commented 5 years ago

I'm not sure what's going on. When I run h5serv on my desktop I get this:

>>> db = h5py.File("tall.hdfgroup.org", "r")
>>> db.items()
[(u'g1', <HDF5 group "/g1" (2 members)>), (u'g2', <HDF5 group "/g2" (2 members)>)]
>>> db['g1']
<HDF5 group "/g1" (2 members)>

I was going to try out using the Docker image, but it looks like the Dockerfile needs updating. There was some code re-arrangement that happened a while back, but the Dockerfile wasn't updated to reflect that.

If you are interested, you can try out the new server on KitaLab without any setup needed: https://www.hdfgroup.org/hdfkitalab/.

jreadey commented 5 years ago

I've checked in updates for h5serv Dockerfile: https://github.com/HDFGroup/h5serv/commit/f31087f0f82c87ec31febd52129b19ce38c80713.

bwulff commented 5 years ago

@jreadey I saw that in the Dockerfile the path to the server changed, but I didn't try with the python base image. Will get back to the project next week and then post an update here. Thanks for caring about this!

jonas2202 commented 5 years ago

@jreadey I have a encountered a similiar problem yesterday. When I start the h5serv docker image, it finds my file but then I get a KeyError:

f = h5pyd.File(r'D:\resource\cosimo\test.erfh5', "r", endpoint="http://172.25.138.241:5000")
meta_data = f['meta'].value

Traceback (most recent call last): File "C:\Users\schwinn\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\h5pyd_hl\group.py", line 301, in get_link_json rsp_json = self.GET(req) File "C:\Users\schwinn\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\h5pyd_hl\base.py", line 551, in GET raise IOError(rsp.status_code, rsp.reason) OSError: [Errno 404] Link [meta] of: 4240de76-4f07-11e9-8c8c-0242ac110002 not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "D:\programming\python\tests\databases\hdf5\test_hdfserv.py", line 5, in meta_data = f['meta'].value File "C:\Users\schwinn\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\h5pyd_hl\group.py", line 397, in getitem parent_uuid, link_json = self.get_link_json(name) File "C:\Users\schwinn\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\h5pyd_hl\group.py", line 303, in get_link_json raise KeyError("Unable to open object (Component not found)") KeyError: 'Unable to open object (Component not found)' OSError: [Errno 404] Link [meta] of: 4240de76-4f07-11e9-8c8c-0242ac110002 not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "D:/programming/python/tests/databases/hdf5/test_hdfserv", line 5, in meta_data = f['meta'].value File "C:\Users\schwinn\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\h5pyd_hl\group.py", line 397, in getitem parent_uuid, link_json = self.get_link_json(name) File "C:\Users\schwinn\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\h5pyd_hl\group.py", line 303, in get_link_json raise KeyError("Unable to open object (Component not found)") KeyError: 'Unable to open object (Component not found)'

asttra commented 3 years ago

I am also encountering this issue.

fabio-rosa-ors commented 3 years ago

I encountered the same issue with the image from dockerhub. @jreadey after a bit of debug I found that h5pyd.File.init setups an http connection corresponding to: http://127.0.0.1:5000/?getdnids=1&getobjs=T&include_attrs=T&domain=tall.test.hdfgroup.org here the 'domain' parameter is translated as is into the httpconn, so the response still refers to a DOMAIN json object without any uuid to let us browse deeper.

but if I translate the 'domain' parameter into a 'host' parameter inside the httpconn object I get the following request: http://127.0.0.1:5000/?getdnids=1&getobjs=T&include_attrs=T&host=tall.test.hdfgroup.org that produces a different json root object referring my hdf5 file

jreadey commented 1 year ago

Closing this since we are dropping support for h5serv.

HDFGroup / h5pyd

Unable to open object (Component not found) #60