OperationFailedError: JSON metadata could not be updated.

[garthg]

Hi @pdurbin ,

I have been using the Dataverse API, and overall it's been pretty successful, but I have a few stragglers that for some reason keep throwing this exception "OperationFailedError: JSON metadata could not be updated". I'm pretty stumped as to why it's breaking, so I'm opening this bug in hopes that you can help me solve it.

I'm using the Python Dataverse client, and the traceback reports the problem from dataset.py line 235. Here's the bottom of the trace:

File "/opt/src/src/dataverse/dataverse/dataset.py", line 235, in update_metadata raise OperationFailedError('JSON metadata could not be updated.')

Using the Dataset object from the Dataverse API, the error comes from my invocation of the following method: dataset_object.update_metadata(new_metadata)

I'm trying to update this Dataset: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi%3A10.7910/DVN/WZMIPM

The code is exactly the same for this one as for all the other successful ones, so I'm not sure what's different. Below is a JSON dump of the new_metadata object I'm passing in. If you have any idea what's breaking here, your help would be greatly appreciated!

{
  "id": 75977, 
  "metadataBlocks": {
    "citation": {
      "displayName": "Citation Metadata", 
      "fields": [
        {
          "multiple": false, 
          "typeClass": "primitive", 
          "typeName": "title", 
          "value": "Council; Council Files August 29, 1845, Case of Orrin de Wolf, GC3/series 378, Petition of Levi D. Smith"
        }, 
        {
          "multiple": true, 
          "typeClass": "compound", 
          "typeName": "author", 
          "value": [
            {
              "authorName": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "authorName", 
                "value": "Digital Archive of Massachusetts Anti-Slavery and Anti-Segregation Petitions, Massachusetts Archives, Boston MA"
              }
            }
          ]
        }, 
        {
          "multiple": true, 
          "typeClass": "compound", 
          "typeName": "datasetContact", 
          "value": [
            {
              "datasetContactEmail": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "datasetContactEmail", 
                "value": "dcarpenter@gov.harvard.edu"
              }
            }
          ]
        }, 
        {
          "multiple": true, 
          "typeClass": "compound", 
          "typeName": "dsDescription", 
          "value": [
            {
              "dsDescriptionValue": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "dsDescriptionValue", 
                "value": "<p>Petition subject: Execution case </p> <p>Original: <a href=\"http://nrs.harvard.edu/urn-3:FHCL:12233013\">http://nrs.harvard.edu/urn-3:FHCL:12233013</a> </p> <p>Date of creation: (unknown) </p> <p>Petition location: Stoneham </p> <p>Legislator, committee, or address that the petition was sent to: \"To be left at the office of the Hangman, 38 Cornhill Boston\" </p> <p>Selected signatures:<ol><li>Levi D. Smith</li><li>John Sprague</li><li>John J. Brown</li></ol> </p> <p>Total signatures: 21 </p> <p>Legal voter signatures (males not identified as non-legal): 21 </p> <p>Female only signatures: No </p> <p>Identifications of signatories: inhabitants </p> <p>Prayer format was <a href=\"http://en.wikipedia.org/wiki/Printing\">printed</a> vs. <a href=\"http://en.wikipedia.org/wiki/Manuscript\">manuscript</a>: Manuscript </p> <p>Additional non-petition or unrelated documents available at archive: additional documents available </p> <p>Additional archivist notes: Orrin de Wolf </p> <p>Location of the petition at the Massachusetts Archives of the Commonwealth: Governor Council Files, August 29, 1845, Case of Orrin de Wolf </p> <p>Acknowledgements: Supported by the National Endowment for the Humanities (PW-5105612), Massachusetts Archives of the Commonwealth, Radcliffe Institute for Advanced Study at Harvard University, Center for American Political Studies at Harvard University, Institutional Development Initiative at Harvard University, and Harvard University Library.</p>"
              }
            }
          ]
        }, 
        {
          "multiple": true, 
          "typeClass": "controlledVocabulary", 
          "typeName": "subject", 
          "value": [
            "Social Sciences"
          ]
        }, 
        {
          "multiple": true, 
          "typeClass": "compound", 
          "typeName": "keyword", 
          "value": [
            {
              "keywordValue": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordValue", 
                "value": "Manuscript"
              }, 
              "keywordVocabulary": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordVocabulary", 
                "value": "prayer-format"
              }
            }, 
            {
              "keywordValue": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordValue", 
                "value": "John J. Brown"
              }, 
              "keywordVocabulary": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordVocabulary", 
                "value": "signatory"
              }
            }, 
            {
              "keywordValue": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordValue", 
                "value": "John Sprague"
              }, 
              "keywordVocabulary": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordVocabulary", 
                "value": "signatory"
              }
            }, 
            {
              "keywordValue": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordValue", 
                "value": "Levi D. Smith"
              }, 
              "keywordVocabulary": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordVocabulary", 
                "value": "signatory"
              }
            }, 
            {
              "keywordValue": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordValue", 
                "value": "inhabitants"
              }, 
              "keywordVocabulary": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordVocabulary", 
                "value": "signatory-category"
              }
            }, 
            {
              "keywordValue": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordValue", 
                "value": "No"
              }, 
              "keywordVocabulary": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordVocabulary", 
                "value": "signatures-female-only"
              }
            }, 
            {
              "keywordValue": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordValue", 
                "value": "21"
              }, 
              "keywordVocabulary": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordVocabulary", 
                "value": "signatures-legal-voters"
              }
            }, 
            {
              "keywordValue": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordValue", 
                "value": "21"
              }, 
              "keywordVocabulary": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "keywordVocabulary", 
                "value": "signatures-total"
              }
            }
          ]
        }, 
        {
          "multiple": true, 
          "typeClass": "compound", 
          "typeName": "publication", 
          "value": [
            {
              "publicationCitation": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "publicationCitation", 
                "value": "D. Carpenter, N. Topich and G. Griffin. Digital Archive of Massachusetts Anti-Slavery and Anti-Segregation Petitions; Council; Council Files August 29, 1845, Case of Orrin de Wolf, GC3/series 378. Massachusetts Archives. Boston, Mass."
              }, 
              "publicationURL": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "publicationURL", 
                "value": "http://nrs.harvard.edu/urn-3:FHCL:12233013"
              }
            }
          ]
        }, 
        {
          "multiple": true, 
          "typeClass": "compound", 
          "typeName": "distributor", 
          "value": [
            {
              "distributorName": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "distributorName", 
                "value": "Massachusetts Archives. Boston, Mass."
              }
            }, 
            {
              "distributorName": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "distributorName", 
                "value": "Harvard Dataverse Network"
              }
            }
          ]
        }, 
        {
          "multiple": false, 
          "typeClass": "primitive", 
          "typeName": "dateOfDeposit", 
          "value": "2015-08-06"
        }
      ]
    }, 
    "geospatial": {
      "displayName": "Geospatial Metadata", 
      "fields": [
        {
          "multiple": true, 
          "typeClass": "compound", 
          "typeName": "geographicCoverage", 
          "value": [
            {
              "city": {
                "multiple": false, 
                "typeClass": "primitive", 
                "typeName": "city", 
                "value": "Stoneham"
              }, 
              "country": {
                "multiple": false, 
                "typeClass": "controlledVocabulary", 
                "typeName": "country", 
                "value": "United States"
              }
            }
          ]
        }
      ]
    }
  }, 
  "productionDate": "Production Date", 
  "versionMinorNumber": 1, 
  "versionNumber": 1, 
  "versionState": "DRAFT"
}

[pdurbin]

File "/opt/src/src/dataverse/dataverse/dataset.py", line 235, in update_metadata raise OperationFailedError('JSON metadata could not be updated.')

Ok, so this line: https://github.com/IQSS/dataverse-client-python/blob/d6e2199f21cddffac37d94d4e39caeaece70ee7e/dataverse/dataset.py#L235

I'm pretty sure this problem is on the server side. Or at least, I can easily reproduce a non-200 error so I opened an issue: https://github.com/IQSS/dataverse/issues/2441

[pdurbin]

@garthg I put in a fix at https://github.com/IQSS/dataverse/commit/9a34912ea805b0fa4379f5b24d8ef410fff8e16d but let's make sure https://github.com/IQSS/dataverse/issues/2441 passes QA before we close this issue. I could ask about getting a build put on https://beta.dataverse.org

[garthg]

@pdurbin thanks for responding so quickly. I think we can wait for #2441, no need to rush something out onto the beta server.

[pdurbin]

@garthg I'm happy for you to do a little testing for us if you feel like it. :)

https://beta.dataverse.org now has v. 4.2 build 13 which includes the bug fix if you'd like to try it out.

[garthg]

@pdurbin Okay! I'll check it out on the beta server if I get a chance.

[garthg]

Hi @pdurbin ,

I'm getting a different error now. Previously, I was obtaining the data-deposit URL from the Dataverse client and calling it directly (which is necessary because of https://github.com/IQSS/dataverse/issues/2122). I think that is no longer working correctly for me, because I'm getting a connection error: ConnectionError: ('Connection aborted.', gaierror(-2, 'Name or service not known'))

Here is the bottom of the Traceback:

File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 108, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', gaierror(-2, 'Name or service not known'))

And here is a representation of the line of my code where I make the request:

resp = requests.post('https://beta.harvard.edu/dvn/api/data-deposit/v1.1/swordv2/collection/dataverse/antislaverypetitionsma',
  data="<entry xmlns=...>...</entry>",
  headers={"Content-type": "application/atom+xml"},
  auth=(my_api_key, None))

Possibly I need to make a different URL request here? Any ideas? Thanks as always for your help!

[pdurbin]

beta.harvard.edu

You'll want to use beta.dataverse.org instead.

[garthg]

That URL was returned from dv_object.collection.get("href"), where dv_object is the Dataverse object found from my dataverse.Connection("beta.dataverse.org", my_api_key).get_dataverses()[0]. Is that not the correct way to obtain the URL? I'd prefer not to hardcode it if possible.

Update: an easier to follow example from the terminal

>>> import dataverse
>>> my_api_key="..."
>>> dataverse.Connection('beta.dataverse.org',  my_api_key).get_dataverses()[0].collection.get('href')
'https://beta.harvard.edu/dvn/api/data-deposit/v1.1/swordv2/collection/dataverse/antislaverypetitionsma'

Update 2: It also appears to give the same error if I just replace beta.harvard.edu with beta.dataverse.org in the URL. Here's my invocation, giving the same error:

resp = requests.post('https://beta.dataverse.org/dvn/api/data-deposit/v1.1/swordv2/collection/dataverse/antislaverypetitionsma',
  data="<entry xmlns=...>...</entry>",
  headers={"Content-type": "application/atom+xml"},
  auth=(my_api_key, None))

[pdurbin]

Ah ha. Here's the problem:

[root@dvn-beta-1 bin]# ./asadmin list-jvm-options | grep fqdn
-Ddataverse.fqdn=beta.dataverse.org
-Ddataverse.fqdn=beta.harvard.edu
[root@dvn-beta-1 bin]# 

[pdurbin]

Ok, fixed. Now the SWORD operations return the proper hostname:

[root@dvn-beta-1 bin]# ./asadmin list-jvm-options | grep fqdn
-Ddataverse.fqdn=beta.dataverse.org
-Ddataverse.fqdn=beta.harvard.edu
[root@dvn-beta-1 bin]# curl -s -u $API_TOKEN: https://beta.dataverse.org/dvn/api/data-deposit/v1.1/swordv2/service-document | xmllint -format - | xpath '//service/workspace/collection/@href'
Found 1 nodes:
-- NODE --
 href="https://beta.harvard.edu/dvn/api/data-deposit/v1.1/swordv2/collection/dataverse/root"
[root@dvn-beta-1 bin]# ./asadmin delete-jvm-options "-Ddataverse.fqdn=beta.harvard.edu"
Deleted 1 option(s)
Command delete-jvm-options executed successfully.
[root@dvn-beta-1 bin]# ./asadmin list-jvm-options | grep fqdn
-Ddataverse.fqdn=beta.dataverse.org
[root@dvn-beta-1 bin]# curl -s -u $API_TOKEN: https://beta.dataverse.org/dvn/api/data-deposit/v1.1/swordv2/service-document | xmllint -format - | xpath '//service/workspace/collection/@href'
Found 1 nodes:
-- NODE --
 href="https://localhost/dvn/api/data-deposit/v1.1/swordv2/collection/dataverse/root"
[root@dvn-beta-1 bin]# ./asadmin stop-domain
Waiting for the domain to stop ....
Command stop-domain executed successfully.
[root@dvn-beta-1 bin]# ./asadmin start-domain
Waiting for domain1 to start ..................................................
Successfully started the domain : domain1
domain  Location: /usr/local/glassfish4/glassfish/domains/domain1
Log File: /usr/local/glassfish4/glassfish/domains/domain1/logs/server.log
Admin Port: 4848
Command start-domain executed successfully.
[root@dvn-beta-1 bin]# curl -s -u $API_TOKEN: https://beta.dataverse.org/dvn/api/data-deposit/v1.1/swordv2/service-document | xmllint -format - | xpath '//service/workspace/collection/@href'
Found 1 nodes:
-- NODE --
 href="https://beta.dataverse.org/dvn/api/data-deposit/v1.1/swordv2/collection/dataverse/root"
[root@dvn-beta-1 bin]# 

Sorry, @garthg! Please try again at your convenience.

[garthg]

Hi @pdurbin ,

It looks like my test against beta.dataverse.org successfully updated this dataset! Published data here: https://beta.dataverse.org/dataset.xhtml?persistentId=doi:10.5072/FK2/BWSZDW .

Do you know when this fix is expected to be deployed in production, so I can run the updates against my production Dataverse?

Thank you as always for your help getting this sorted out!

Garth

[pdurbin]

@garthg fantastic. Thanks for testing. That bug fix is in the 4.2 branch so I'd keep an eye on when the 4.2 milestone is closed: https://github.com/IQSS/dataverse/milestones

Speaking of closed, if it's ok with you we can close this issue since https://github.com/IQSS/dataverse/issues/2441 will go through QA. The problem was with the server, not this Python client.

[garthg]

@pdurbin Okay, I'll track the 4.2 milestone, which looks to be targeting September 9th. I'll close this bug. Thanks for your help!