Fix for search result handling

[hevp]

Patch fixes searching in general, since currently an ReverseLookupException is raised on any search:

• Split query results string on line feeds
• Remove empty results
• Create JSON structure with "count" and "values" properties

[TobiasWeigel]

Hello hevp (lacking another name),

thank you for your merge request. We have now looked into this and cannot reproduce the original cause where any search would raise a ReverseLookupException. Searching works fine in EUDAT operations. Can you be more specific to what you did and how your setup was? The proposed JSON structure would cause an API change and as such we cannot easily do it without good reasons and a possible migration procedure at EUDAT operations. Why do you see a separate "count" item as necessary? Usually, it would be up to the client to count items in the response.

Best, Tobias

[hevp]

Hi Tobias,

The response variable resp.content is not formatted as a serialized JSON string, but as a \r\n separated string of values. Therefore the JSON.loads call doesn't work in line 301.

[hevp]

To clarify, when running epicclient2.py from B2SAFE-core I get the following error:

$ ./epicclient2.py os credentials_tdr search URL *hello*
/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:821: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
Traceback (most recent call last):
  File "./epicclient2.py", line 284, in <module>
    _ARGS.func(_ARGS)
  File "./epicclient2.py", line 43, in search
    result = client.search_handle(**kvpairs)
  File "/usr/local/lib/python2.7/site-packages/b2handle-1.0.3-py2.7.egg/b2handle/handleclient.py", line 963, in search_handle
    list_of_handles = self.__searcher.search_handle(URL=URL, prefix=prefix, **key_value_pairs)
  File "/usr/local/lib/python2.7/site-packages/b2handle-1.0.3-py2.7.egg/b2handle/searcher.py", line 242, in search_handle
    return self.__search_handle(**args)
  File "/usr/local/lib/python2.7/site-packages/b2handle-1.0.3-py2.7.egg/b2handle/searcher.py", line 304, in __search_handle
    raise ReverseLookupException(msg=msg, query=query, response=resp)
b2handle.handleexceptions.ReverseLookupException: Error during Reverse Lookup: The response is not JSON..
    Query: ?URL=*hello*
    URL: https://epic3.storage.surfsara.nl/v2_test/handles/843?URL=*hello*
    HTTP Status Code: 200
    Response: 

The response from the request is not JSON, so it needs to be processed before it can be loaded with the JSON package.

[cookie33]

Hi,

Can you test it with HTTPS_verify set to either True or pointing to a correct certificate authentication.

And we just uploaded a new version of the epicclient2.py in a pull request. This has better errorhandling.

This is what I get:

$ ./epicclient2.py os ../scripts/tests/resources/epic2_credentials search URL *hello*
["841/5f6fb451-5841-11e4-9665-14109fe83170", "847/3d5b247c-62c8-11e5-a76a-000c2913a7b8", "847/3ed7690a-62c8-11e5-8ce2-000c2913a7b8", "847/5abe2b78-65e2-11e5-8918-000c2913a7b8"]

And with HTTPS_verify set to False:

$ ./epicclient2.py os ../scripts/tests/resources/epic2_credentials search URL *hello*
/usr/lib/python2.6/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
["841/5f6fb451-5841-11e4-9665-14109fe83170", "847/3d5b247c-62c8-11e5-a76a-000c2913a7b8", "847/3ed7690a-62c8-11e5-8ce2-000c2913a7b8", "847/5abe2b78-65e2-11e5-8918-000c2913a7b8"]

[hevp]

Hi,

With HTTPS_verify set to True I get the same error:

$ ./epicclient2.py os credentials_tdr search URL *test*
Traceback (most recent call last):
  File "./epicclient2.py", line 284, in <module>
    _ARGS.func(_ARGS)
  File "./epicclient2.py", line 43, in search
    result = client.search_handle(**kvpairs)
  File "/usr/local/lib/python2.7/site-packages/b2handle-1.0.3-py2.7.egg/b2handle/handleclient.py", line 963, in search_handle
    list_of_handles = self.__searcher.search_handle(URL=URL, prefix=prefix, **key_value_pairs)
  File "/usr/local/lib/python2.7/site-packages/b2handle-1.0.3-py2.7.egg/b2handle/searcher.py", line 242, in search_handle
    return self.__search_handle(**args)
  File "/usr/local/lib/python2.7/site-packages/b2handle-1.0.3-py2.7.egg/b2handle/searcher.py", line 304, in __search_handle
    raise ReverseLookupException(msg=msg, query=query, response=resp)
b2handle.handleexceptions.ReverseLookupException: Error during Reverse Lookup: The response is not JSON..
    Query: ?URL=*test*
    URL: https://epic3.storage.surfsara.nl/v2_test/handles/843?URL=*test*
    HTTP Status Code: 200
    Response: 0000a932-a592-11e5-a86b-0050569ed64e
0000d444-a40b-11e5-9d30-0050569ed64e
...

(followed by a long list of handles). The request works as expected, but the response text is a long string with line feeds in it.

[cookie33]

Hi,

There is a basic flaw.

Why are you using the epicclient2 on the "normal" ePIC API? Here you should use the epicclient.py. This is a mismatch.

https://epic3.storage.surfsara.nl/v2_test is for the ePIC API with epicclient.py https://epic3.storage.surfsara.nl:8001 is for the reverse lookup servlet with epicclient2.py

These API's are totally different. And also use different credentials etc..

[hevp]

Good question. If you point me to the right documentation, I might be able to find this out myself, but it's not mentioned anywhere as far as I know...

Thanks!

[cookie33]

To work with the b2handle library and handle http RESTFULL JSON API and reverse lookup servlet you need to use the epicclient2.py

The documentation for the b2handle library is here: http://eudat-b2safe.github.io/B2HANDLE/creatingclientcertificates.html Contact your handle provider to get the public key uploaded you create and to get the credentials for the reverse lookup.

[cookie33]

p.s. for the b2handle library and reverselookup servlet with epicclient2.py you need a credentials file similar like the one below:

{
    "handle_server_url": "https://fqdn:8001",
    "private_key": "/home/.../355_841_privkey.pem",
    "certificate_only": "/home/.../355_841_certificate_only.pem",
    "prefix": "841",
    "handleowner": "200:0.NA/841",
    "reverselookup_username": "841",
    "reverselookup_password": "<password>",
    "HTTPS_verify": "True"
}

[hevp]

Thank you cookie33, I will try and run it with this configuration.

This pull request can be closed for now.