jggautier
commented
4 years ago This might be related to invisible characters in the metadata of datasets. I was asked to try to find special characters that are breaking harvesting, and I've found that datasets with the hexadecimal character 0x0C in their description fields are breaking metadata export in general (in the UI, over API and over OAI-PMH) and making it impossible to edit the dataset metadata, at least in the UI - nothing happens when I click the dataset's Edit Metadata button.
You can find 35 datasets with this character in their description fields by querying the database, copying and pasting:
select datasetversion.dataset_id, concat('https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:', authority, '/', identifier) as dataset_url,
datasetversion.versionstate, versionnumber, lastupdatetime, datasetfieldvalue.value, dataverse.name
from datasetfield datasetfield1, datasetfield datasetfield2, datasetfieldcompoundvalue, datasetfieldvalue, datasetversion, datasetfieldtype, dvobject, dataset, dataverse
where
-- Join all of the tables
datasetfield1.id = datasetfieldvalue.datasetfield_id and
datasetfield1.parentdatasetfieldcompoundvalue_id = datasetfieldcompoundvalue.id and
datasetfield2.id = datasetfieldcompoundvalue.parentdatasetfield_id and
datasetfield2.datasetversion_id = datasetversion.id and
datasetfield2.datasetfieldtype_id = datasetfieldtype.id and
dvobject.id = datasetversion.dataset_id and
dataset.id = dvobject.id and
dataverse.id = dvobject.owner_id and
-- Search in the description field for the special character
datasetfield1.datasetfieldtype_id = 17 and
datasetfieldvalue.value ~* ' ' and
-- Exclude harvested datasets
harvestingclient_id is null and
-- Exclude any metadata stored in metadata templates
datasetfield1.template_id is null
order by lastupdatetime descThe where clause with ~* ' ' is regex. The invisible character is between the two apostrophes. If you copy and paste into a text editor that shows hidden characters, you should see <0x0C>. I read this character can be used for html colors, so probably people copied text from a webpage and pasted into the metadata field. I can't remember or find a better way to query for these types of characters, for a better idea of how many datasets this is affecting - that is, how many have un-editable metadata and un-exportable metadata. If someone else has suggestions, please share. :)
These three datasets, published earlier this year, are among the 35 :
- https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/XLQYWH
- https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/KHDZND
- https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/F6CVMG
I can reproduce this problem in Demo Dataverse, so this issue should probably be moved to the general Dataverse GitHub repo, or maybe information in this issue should be moved into an existing issue in the general Dataverse GitHub repo. This doesn't seem to affect dataset publishing, so doesn't seem to fit the scope of https://github.com/IQSS/dataverse/issues/3328.
JingMa87
sbarbosadataverse
Specifically, UNC Dataverse and Demo Dataverse are unable to harvest all records in Harvard Dataverse's "IQSS" set, and Demo Dataverse is unable to harvest all records in Harvard Dataverse's default set. Both the IQSS set and the default set contain metadata records of all datasets deposited in Harvard Dataverse.
UNC's repository (Dataverse version 4.16) has harvested about 20,000 of Harvard Dataverse's 32,000+ datasets from the IQSS set using the oai_ddi metadata format. Don Sizemore let me know that the repository's superuser dashboard reports that the last harvesting attempt was Apr 29, 2018 and that it's been "INPROGRESS" since.
Demo Dataverse (version 4.20) harvested fewer than 14,000 of Harvard Dataverse's datasets from the IQSS set using the dataverse_json format. At the time, May 11, 2020, its superuser dashboard reported that the attempt on May 11, 2020 FAILED. Then I set Demo Dataverse to harvest from the default set, using dataverse_json format, and it also failed, again harvesting fewer than 14,000.