oai_dc export fails, OAI-PMH error

[luopc-top]

Hi, I find that oai-pmh api return error xml data with the url https://dataverse.harvard.edu/oai?verb=ListRecords&resumptionToken=MTo1NTAwfDI6fDM6fDQ6fDU6b2FpX2Rj see the figure below.

[madryk]

Probably there is some error in oai_dc format when retrieving dataset with doi: doi:10.7910/DVN/3FPJEA

https://dataverse.harvard.edu/oai?verb=GetRecord&identifier=doi:10.7910/DVN/3FPJEA&metadataPrefix=oai_dc

returned xml is cut after opening of <metadata> tag

[pdurbin]

@madryk good catch, if I go to https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/3FPJEA and try various export formats like...

• Dublin Core
• DDI
• DDI HTML Codebook
• OpenAIRE

... I get "Export Failed".

If I download the dataset as JSON I see that the title has Unicode Character 'END OF TEXT' (U+0003) at the end:

But is this the only dataset causing problems for @luopc-top ?

[luopc-top]

I just found out by accident, I haven't tried other datasets.

[luopc-top]

I also found that the exported json data for dataset doi:10.7910/DVN/3FPJEA has some errors. see https://dataverse.harvard.edu/api/datasets/export?exporter=dataverse_json&persistentId=doi%3A10.7910/DVN/29779

[pdurbin]

@luopc-top @madryk I'm curious if you've experimented with this any further, especially with the Unicode character 'END OF TEXT' (U+0003) thing I mentioned.

[madryk]

I didn't explore it further. The only thing I can add is that I encountered some similar issue in not Dataverse project. In some way we had https://www.fileformat.info/info/unicode/char/0002/index.htm character in our data, which also was making issues when creating xml's. What we did to fix it:

• wrote flyway java migration that removes unwanted characters
• for every string that could be provided by user or imported from external source we run algorithm for removing unwanted characters.

Algorithm which removes unwanted character was based on: https://stackoverflow.com/questions/6198986/how-can-i-replace-non-printable-unicode-characters-in-java (answer https://stackoverflow.com/a/18603020) but instead of replacing to ? character we just remove them and we left newline and tab character (\n \t) intact.