Extract Study Url from NESSTAR records

[cessda-bitbucket-importer]

Original report on BitBucket by John Shepherdson (GitHub: john-shepherdson).

---

Not in expected field. See email correspondence between Ornulf Risnes and John Shepherdson.

[cessda-bitbucket-importer]

Original comment by John Shepherdson (GitHub: john-shepherdson).

---

Hi John,

I believe I have been able to confuse us both in several different ways during this discussion about constructing a synthetic study URL to Nesstar held studies.

The main cause of confusion can probably be clarified by the following statement:

If landing page URLs (e.g. synthetic Nesstar WebView-urls) are part of the DDI payload already, no modifications to the OAI-PMH-Nesstar front are required.

The payload is already there.

We have now set up a preliminary OAI-PMH-front for NSD's main survey repository (http://nsddata.nsd.uib.no).

The OAI-PMH-front-service will get a domain name (likely oai-pmh.nsd.no), but for now it has an IP-address only.

http://129.177.90.182/oai-pmh/?verb=GetRecord&metadataPrefix=oai_ddi&identifier=http://nsddata.nsd.uib.no:80/obj/fStudy/NSD2200-3

NB! Right now you won't be able to access this URL due to University firewalls. I will address this.

But attached you will find the OAI-PMH-xml response payload for that particular OAI-PMH-request. I also attach a screenshot from my browser as evidence that I'm not making this up.

Now - in the attached xml response, you will find the URI to the landing page for this particular study under:

stdyDscr/method/dataAccs/setAvail/accsPlac

...approximately on line 223 in the xml-document when viewed in a text editor:

doi:10.18712/NSD-NSD2200-3-v1

To sum up: The good:

• Using verb=GetRecord and metadataPrefix=oai_ddi will give the harvesters utilising this OAI-PMH-servers the content we're after.

• No modifications are needed to the Nesstar-OAI-PMH-server to achieve this

The bad:

• While the -element indeed is/was in the agreed CESSDA Metadata Template for Nesstar Publisher, the use of this field for this purpose doesn't seem to be widespread among CESSDA archives. The Slovenians use it (but not in a sufficiently precise manner). The Germans and the British don't.

This means that CESSDA archives with Nesstar Servers, and that have been following our practice by using the -element for landing page info can use an unmodified version of the Nesstar OAI-PMH-server to expose this info.

It is however unclear how many archives this "convenience" will benefit.

• Harvesters who ask for the metadataPrefix=oai_dc (dublin core), won't see this at all, and only have the direct object link to the "raw" Nesstar study object in the identifier field:

http://nsddata.nsd.uib.no:80/obj/fStudy/NSD2200-3 This means that we are unable to present a unified view of the identifier (or what we mean by identifier) through both DDI and DC at a time. Note that this exact same problem will also occur if we use the CMM-suggestion: instead of the CESSDA Template element that we use: Perhaps not a big problem if the CESSDA-catalogue-harvesters rely on DDI (and not DC) in their harvesting. To sum up - if we ignore the DC-problem (and the fact that the lack of domain name and firewall-whitelisting prevents you from seeing the service), we should very soon have a situtation where you will be able to harvest NSDs (main) Nesstar-server and extract the proper landing page URL from the oai-pmh(-ddi)-payload. best regards, Ørnulf

[cessda-bitbucket-importer]

Original comment by Moses Mansaray (GitHub: doraVentures).

---

There is a lot in the above description @‌jws_mo

In summary I understand it to suggest to use: <stdyDscr><method><dataAccs><setAvail><accsPlac> for the study url instead of what CMM suggested which in code currently is:

Can you please confirm? Maybe update the nesstar CMM mapping document and send me a link to it.

Thank you

[cessda-bitbucket-importer]

Original comment by John Shepherdson (GitHub: john-shepherdson).

---

Please try extracting studyURL for NESSTAR endpoints from Can you check which endpoints are using that field consistently (the above suggests that UKDS and GESIS don't).

[cessda-bitbucket-importer]

Original comment by Moses Mansaray (GitHub: doraVentures).

---

Analysis of StudyURL current state for Nesstar records

Source: https://datacatalogue-dev.cessda.eu/es/

• ADP - every Records harvested StudyUrl is: http://www.adp.fdv.uni-lj.si/opisi/
• NSD - All Records harvested are missing StudyUrl
• Progedo - All Records harvested are missing StudyUrl
• SoDaNet - 45 Records harvested StudyUrl seems to be unique and takes me to the records landing page, an instance of junk entry, the rest of the records are missing studyUrl from the current cmm expected Xpath

I have extracted and converted this to a flattened table in excel for @‌jws_mo to further analyse at will, see attached document. studyurl_checks_result.xlsx

To conclude

The correct CMM xPath at least does not seem to be consistently used and at most not used at all:

• ADP - used but not a Unique Resource Identifier of a study
• NSD - not all
• PROGE - not all
• SoDaNet - Used correctly for some Records, missing for others.

[cessda-bitbucket-importer]

Original comment by Moses Mansaray (GitHub: doraVentures).

---

I will move onto replace the xpath to use <stdyDscr><method><dataAccs><setAvail><accsPlac> and report back the difference/impact.

[cessda-bitbucket-importer]

Original comment by John Shepherdson (GitHub: john-shepherdson).

---

Thanks. I'll have a look at the spreadsheet.

[cessda-bitbucket-importer]

Original comment by Moses Mansaray (GitHub: doraVentures).

---

Code changes done, doing a full re-index locally to further verify impact. @‌jws_mo So far I know from manual checks that:

Progedo http://nesstar.sciences-po.fr/oai-pmh

• Does not consistentyly have for all records eg
  • http://nesstar.sciences-po.fr/oai-pmh/?verb=GetRecord&metadataPrefix=oai_ddi&identifier=http%3A%2F%2Fnesstar.sciences-po.fr%3A80%2Fobj%2FfStudy%2Ffr.cdsp.ddi.OV70 (Interestingly this example record has a questionable title: Temp#119 )
• Records with does not contain a unique identifier to a record example:
  • http://nesstar.sciences-po.fr/oai-pmh/?verb=GetRecord&metadataPrefix=oai_ddi&identifier=http%3A%2F%2Fnesstar.sciences-po.fr%3A80%2Fobj%2FfStudy%2Ffr.cdsp.ddi.AGORA1977

ADP https://nesstar2.adp.fdv.uni-lj.si/oai-pmh

• Has Records with does not contain a unique identifier to a record example:
  • https://nesstar2.adp.fdv.uni-lj.si/oai-pmh/?verb=GetRecord&identifier=http%3A%2F%2Fnesstar2.adp.fdv.uni-lj.si%3A80%2Fobj%2FfStudy%2Fads&metadataPrefix=oai_ddi and
  • https://nesstar2.adp.fdv.uni-lj.si/oai-pmh/?verb=GetRecord&identifier=http%3A%2F%2Fnesstar2.adp.fdv.uni-lj.si%3A80%2Fobj%2FfStudy%2Fakp93&metadataPrefix=oai_ddi

NSD https://oai-pmh.nsd.no/oai-pmh

• Has Records with that do contain a unique identifier to a record example:
  • https://oai-pmh.nsd.no/oai-pmh?verb=GetRecord&identifier=http%3A%2F%2Fnsddata.nsd.uib.no%3A80%2Fobj%2FfStudy%2FNSD0054&metadataPrefix=oai_ddi and
  • https://oai-pmh.nsd.no/oai-pmh?verb=GetRecord&identifier=http%3A%2F%2Fnsddata.nsd.uib.no%3A80%2Fobj%2FfStudy%2FNSD0019-2&metadataPrefix=oai_ddi

Sodanet http://nesstar-server.sodanet.gr/oai-pmh

• Does not have a for the many records i tried eg:
  • http://nesstar-server.sodanet.gr/oai-pmh/?verb=GetRecord&identifier=http%3A%2F%2Fnesstar-server.sodanet.gr%3A80%2Fobj%2FfStudy%2FZA5897-GR&metadataPrefix=oai_ddi
  • http://nesstar-server.sodanet.gr/oai-pmh/?verb=GetRecord&identifier=http%3A%2F%2Fnesstar-server.sodanet.gr%3A80%2Fobj%2FfStudy%2FZA5895-GR&metadataPrefix=oai_ddi
  • http://nesstar-server.sodanet.gr/oai-pmh/?verb=GetRecord&identifier=http%3A%2F%2Fnesstar-server.sodanet.gr%3A80%2Fobj%2FfStudy%2FZA5785&metadataPrefix=oai_ddi

[cessda-bitbucket-importer]

Original comment by Moses Mansaray (GitHub: doraVentures).

---

Checks after Code

In brief @‌jws_mo All SPs will need take action to ensure they:

• Have studyUrl in the this path: <stdyDscr><method><dataAccs><setAvail><accsPlac>
• With a URI attribute in the above element
• Each URI should be that for the record; a Unique Resource Identifier

ADP

• Has Records with <accsPlac> does not contain a unique identifier to a record example:
• Most URIs are: https://www.adp.fdv.uni-lj.si/opisi/
• Some Blanks
• A few URI links to
  • http://www.adp.fdv.uni-lj.si/podatki/
  • http://www.gesis.org/eurobarometer-data-service/home/
  • http://www.gesis.org/eurobarometer
  • http://zacat.gesis.org/webview/
  • http://zacat.gesis.org/webview/index.jsp

NSD

• Majority of the Records have <accsPlac> that do contain a unique identifier to a record
• Some Records do not
• Some Records have the same study url eg http://www.nsd.uib.no/data/ny_individ/norStudy/norStudy.cfm?norStudyID=197
  • This uri results to an Error 404

Progedo

• Does not consistently have <accsPlac> for all Records
• Records with <accsPlac> does not contain a unique identifier to a Record. Example repeated URI:
  • https://quetelet.casd.eu
  • https://quetelet.casd.eu/fr

Sodanet

• Does not have a <accsPlac> for the many records I tried

CSDA: http://nesstar.soc.cas.cz/oai-pmh

• Connection timeouts
• No records harvested

Unidata: http://149.132.157.156/oai-pmh

• Connection timeouts
• No records harvested

For more see attached spreadsheet “studyurl_checks_result_after_change.xlsx”

[cessda-bitbucket-importer]

Original comment by Moses Mansaray (GitHub: doraVentures).

---

[cessda-bitbucket-importer]

Original comment by Moses Mansaray (GitHub: doraVentures).

---

@‌jws_mo PRs:

• link to pull request removed
• link to pull request removed

I will merge these to get them to DEV for checks; sonar and full index clear out and re-harvest and re-ingest . Please feel free to feedback if anything.

[cessda-bitbucket-importer]

Original comment by Moses Mansaray (GitHub: doraVentures).

---

Hi @‌jws_mo, the coverage issue I mentioned on slack is now a blocker.

https://sonarqube.cessda.eu/component_measures?id=eu.cessda.pasc%3Apasc-osmh-handler-nesstar&metric=new_coverage&view=list

The new code coverage = 0% would be because no coverage is being detected project wide for most of the java applications.

This will now BLOCK all further builds and tickets from being verified on DEV. This would need to be resolved asap.

[cessda-bitbucket-importer]

Original comment by Moses Mansaray (GitHub: doraVentures).

---

Waiting for sonar to be fixed for code coverage

[cessda-bitbucket-importer]

Original comment by John Shepherdson (GitHub: john-shepherdson).

---

Waiting for admin access to SonarQube.

[cessda-bitbucket-importer]

Original comment by John Shepherdson (GitHub: john-shepherdson).

---

@‌matthew-morris-cessda Matthew, please have a look at this as a matter urgency. We need these builds to succeed so I can evaluate the changes made by Moses and Ashley.

[cessda-bitbucket-importer]

Original comment by Matthew Morris (GitHub: matthew-morris-cessda).

---

I fixed the code coverage issue and builds are passing again

[cessda-bitbucket-importer]

Original comment by Moses Mansaray (GitHub: doraVentures).

---

Nesstar now extracts studyUrl from the @‌URI attribute of the following element:

+  static final String STUDY_URL_XPATH = "//ddi:codeBook/stdyDscr/dataAccs/setAvail/accsPlac";

‌

Actual above to reflect this path requested here: stdyDscr><method><dataAccs><setAvail><accsPlac>

This code is now in dev and ES has had a fresh re-index.

See comment of analysis after change here: https://github.com/cessda/cessda.pasc.version2/issues/94#comment-51834673

[cessda-bitbucket-importer]

Original comment by Moses Mansaray (GitHub: doraVentures).

---

Not in expected field. See email correspondence between Ornulf Risnes and John Shepherdson.

[cessda-bitbucket-importer]

Original comment by John Shepherdson (GitHub: john-shepherdson).

---

Assigned to Metadata Office. See https://github.com/cessda/cessda.metadata.office/issues/9

[cessda-bitbucket-importer]

Original comment by John Shepherdson (GitHub: john-shepherdson).

---

Not in expected field. See email correspondence between Ornulf Risnes and John Shepherdson.

[cessda-bitbucket-importer]

Original comment by Ørnulf Risnes.

---

Have discussed options with Anne Marie T Laundal (NSD).

‌

Could I ask; where does the harvester currently look for DOI-info?

Is it here?:

[cessda-bitbucket-importer]

Original comment by Taina Jääskeläinen.

---

Doi info is harvested from

urn:nbn:fi:fsd:FSD3205 It is a repeatable element, so it is good to record the PID type in _agency_ attribute as CESSDA allows only the four \(doi, urn, Handle, ARK\). PID type can differentiate it from other, possibly in-house identifiers, whenever needed.