Missing User-Agent headers breaks Cloudfront (was: requests.exceptions.JSONDecodeError ... (perhaps wrong api_endpoint ?))

[abubelinha]

Got error within the first lines of example.py:

My code:

def dspace_rest_python_example():
    """ Taken from https://github.com/the-library-code/dspace-rest-python/blob/main/example.py
        """
    from dspace_rest_client.client import DSpaceClient
    from dspace_rest_client.models import Community, Collection, Item, Bundle, Bitstream

    # Example variables needed for authentication and basic API requests
    # SET THESE TO MATCH YOUR TEST SYSTEM BEFORE RUNNING THE EXAMPLE SCRIPT
    url = 'https://demo.dspace.org/server/api'
    username = 'myusername'
    password = 'mypassword'

    # Instantiate DSpace client
    d = DSpaceClient(api_endpoint=url, username=username, password=password)

    # Authenticate against the DSpace client
    authenticated = d.authenticate()
    if not authenticated:
        print(f'Error logging in! Giving up.')
        exit(1)
    else:
        print('Login successful !!')

if __name__ == "__main__":
    dspace_rest_python_example()

My output:

C:\Python38\python test_dspace.py

Traceback (most recent call last):
  File "C:\Python38\lib\site-packages\requests\models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Python38\lib\site-packages\simplejson\__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "C:\Python38\lib\site-packages\simplejson\decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "C:\Python38\lib\site-packages\simplejson\decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dvd_dspace.py", line 65, in <module>
    dspace_rest_python_example()
  File "dvd_dspace.py", line 56, in dspace_rest_python_example
    authenticated = d.authenticate()
  File "C:\Python38\lib\site-packages\dspace_rest_client\client.py", line 116, in authenti
    r_json = r.json()
  File "C:\Python38\lib\site-packages\requests\models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Tried it in two different machines, same error:

• Windows 7 64bit, Python 3.8.7, dspace_rest_client-0.1.8 (above output)
• Windows 10, Python 3.9.9, dspace_rest_client-0.1.8 (same output)

I could login to the demo server (https://demo.dspace.org/) using the same credentials, so they must be correct. Actually these are provided in the top of that page.

As the script does not reach the 'if not authenticated' line , maybe the api_endpoint url value I am using is just wrong?

I figured the url out looking at the end of this page, but I may have misunderstood: https://wiki.lyrasis.org/display/DSDOC7x/REST+API

If that is the case, I would like to know what url should I use.

• Perhaps the package could also tell users the provided url is not a valid api endpoint, instead of raising an error?
• I also suggest adding a way to recognize unsupported DSpace versions (so users providing a valid -but old- DSpace endpoint do not get an error, but an informative message).

Thanks a lot in advance @abubelinha

[kshepherd]

Hi @abubelinha , thanks for logging this issue. I am looking into it - trying to figure out if 7.6.1 has some differences or if its due to some unexpected setup in the demo site (other servers are working ok). The fact that the 403s I'm seeing are HTML rather than JSON responses is interesting...

I also like the suggestions around endpoint and version validation, I will create some issues to describe these requirements

[abubelinha]

Thanks @kshepherd From your reply I should understand the url I used is the correct one, isn't it?

Do you know of other open demo servers I could use to try this package until this issue is solved?

Unfortunately my institution repository is an old DSpace version (and it doesn't have a demo testing server, anyway). I wish I could use this package to encourage them to upgrade it.

Off topic (would better fit into repository discussions): Does anyone know examples of DSpace api being used (python or not) to autopublish thousands of museum pieces as individual DSpace items? I.e. having a handle.net url for a pdf commenting each painting in a museum, or things like that. (of course, providing the institution has a database containing all info needed to produce the documents).

Thanks a lot for your help Edit: sorry I closed the issue by mistake and reopened

[kshepherd]

@abubelinha

Yes your base URL looks good, it just seems that either something is different about that site, or 7.6.1 (the latest version) changed an authentication response.

For the most control over a demonstration server, of whatever version you want to try, I would recommend using the docker images (see: https://wiki.lyrasis.org/display/DSPACE/Try+out+DSpace+7#TryoutDSpace7-InstallviaDocker)

If I run up any public demo servers myself I will be sure to let you know.

Regarding the other question - there are definitely scenarios like this, though usually for digital derivatives of the pieces themselves - I don't know if I've heard of a handle or DOI for a physical item but at the very least it could resolve to a catalogue entry with any digitised objects (photos, recordings, etc). I would recommend the dspace-community mailing list (https://groups.google.com/g/dspace-community) or code4lib community (https://code4lib.org/).

[kshepherd]

@abubelinha Good news! @misilot had a suggestion which helped to solve the problem -- there is a Cloudfront layer in front of the official demo server, and it was filtering out requests based on which user agent was set. Please see my new example.py script for usage of the new fake_user_agent argument to the client instantiation.

Something like this should now work, as at version 0.1.9 on PyPI (https://pypi.org/project/dspace-rest-client/0.1.9/) or the git tag here: https://github.com/the-library-code/dspace-rest-python/tree/dspace-rest-client-0.1.9

Thanks for reporting this issue, it will definitely help other users in the future who might have API endpoints behind other WAF proxies like Cloudfront.