search

cessda / cessda.cdc.versions

Issue track and wiki for the CESSDA Data Catalogue
https://datacatalogue.cessda.eu/
Apache License 2.0
0 stars 0 forks source link

Records not available for some endpoints where default languages has been set #202

Closed cessda-bitbucket-importer closed 3 years ago

cessda-bitbucket-importer commented 4 years ago

Original report on BitBucket by John Shepherdson (GitHub: john-shepherdson).

See also #192

cessda-bitbucket-importer commented 4 years ago

Original comment by John Shepherdson (GitHub: john-shepherdson).

ADP set to sl

APIS set to pt

PROGEDO set to fr

SODA set to fr

CSDA set to cs

SoDaNet set to el

cessda-bitbucket-importer commented 4 years ago

Original comment by John Shepherdson (GitHub: john-shepherdson).

ES indices exist for all above languages, but cs and sl are empty. Greek and French records are accessible via UI.

settings_cmmstudy_xx.json files exist for all of them in Indexer codebase.

They are present in the list of languages in application.yml and imports and language structures in language.js and xx.js files in the locales directory of the Searchkit codebase.

‌

‌

cessda-bitbucket-importer commented 4 years ago

Original comment by John Shepherdson (GitHub: john-shepherdson).

APIS studies now appear in the English section (in a mixture of English and Portuguese), publisher is 'Arquivo Português de Informação Social'

cessda-bitbucket-importer commented 4 years ago

Original comment by Matthew Morris (GitHub: matthew-morris-cessda).

@john-shepherdson Is this still valid?

cessda-bitbucket-importer commented 4 years ago

Original comment by John Shepherdson (GitHub: john-shepherdson).

2020-10-20 09:35:55.626 INFO                (HarvesterRunner.java:103) - Total number of records: 33491
2020-10-20 09:35:55.640 INFO              (MicrometerMetrics.java:122) - [cs] Current records [0]
2020-10-20 09:35:55.643 INFO              (MicrometerMetrics.java:122) - [da] Current records [2193]
2020-10-20 09:35:55.646 INFO              (MicrometerMetrics.java:122) - [de] Current records [6257]
2020-10-20 09:35:55.647 INFO              (MicrometerMetrics.java:122) - [el] Current records [0]
2020-10-20 09:35:55.650 INFO              (MicrometerMetrics.java:122) - [en] Current records [17812]
2020-10-20 09:35:55.653 INFO              (MicrometerMetrics.java:122) - [et] Current records [0]
2020-10-20 09:35:55.655 INFO              (MicrometerMetrics.java:122) - [fi] Current records [1565]
2020-10-20 09:35:55.657 INFO              (MicrometerMetrics.java:122) - [fr] Current records [313]
2020-10-20 09:35:55.659 INFO              (MicrometerMetrics.java:122) - [hu] Current records [0]
2020-10-20 09:35:55.663 INFO              (MicrometerMetrics.java:122) - [it] Current records [0]
2020-10-20 09:35:55.666 INFO              (MicrometerMetrics.java:122) - [nl] Current records [4193]
2020-10-20 09:35:55.668 INFO              (MicrometerMetrics.java:122) - [no] Current records [0]
2020-10-20 09:35:55.670 INFO              (MicrometerMetrics.java:122) - [pt] Current records [0]
2020-10-20 09:35:55.672 INFO              (MicrometerMetrics.java:122) - [sk] Current records [5]
2020-10-20 09:35:55.676 INFO              (MicrometerMetrics.java:122) - [sl] Current records [0]
2020-10-20 09:35:55.679 INFO              (MicrometerMetrics.java:122) - [sr] Current records [0]
2020-10-20 09:35:55.680 INFO              (MicrometerMetrics.java:122) - [sv] Current records [694]
2020-10-20 09:36:02.357 INFO              (MicrometerMetrics.java:180) - [AUSSDA] Current records: [730]
2020-10-20 09:36:02.357 INFO              (MicrometerMetrics.java:180) - [SASD] Current records: [8]
2020-10-20 09:36:02.357 INFO              (MicrometerMetrics.java:180) - [SND] Current records: [1388]
2020-10-20 09:36:02.358 INFO              (MicrometerMetrics.java:180) - [DNA] Current records: [3414]
2020-10-20 09:36:02.360 INFO              (MicrometerMetrics.java:180) - [UniData] Current records: [64]
2020-10-20 09:36:02.360 INFO              (MicrometerMetrics.java:180) - [DANS] Current records: [4193]
2020-10-20 09:36:02.361 INFO              (MicrometerMetrics.java:180) - [APIS] Current records: [15]
2020-10-20 09:36:02.361 INFO              (MicrometerMetrics.java:180) - [GESIS] Current records: [12216]
2020-10-20 09:36:02.361 INFO              (MicrometerMetrics.java:180) - [ProgedoSciencesPo] Current records: [313]
2020-10-20 09:36:02.362 INFO              (MicrometerMetrics.java:180) - [FSD] Current records: [3117]
2020-10-20 09:36:02.362 INFO              (MicrometerMetrics.java:180) - [UKDS] Current records: [8033]
2020-10-20 09:36:02.363 INFO              (ConsumerScheduler.java:139) - [Full Run] Consume and Ingest All SPs Repos:
Ended at: [2020-10-20T09:36:02.362834Z]

‌

APIS studies now appear in the English section (in a mixture of English and Portuguese), publisher is ‘Portuguese Archive of Social Information (APIS)’

It looks like 5 endpoints are currently unavailable, so keep issue on hold until they come back on line and can be checked.

cessda-bitbucket-importer commented 3 years ago

Original comment by Taina Jääskeläinen.

We seem to be missing CSDA which had a functional endpoint previously. Is the endpoint still unavailable? Any news on the issue?

cessda-bitbucket-importer commented 3 years ago

Original comment by John Shepherdson (GitHub: john-shepherdson).

CSDA are using NESSTAR. It was unavailable/unresponsive last night and as a result the attempt to harvest it timed out.

2021-03-02 22:38:03.531 ERROR (RemoteHarvesterConsumerService.java:99) - [CSDA] ListRecordHeaders failed: java.net.http.HttpTimeoutException: request timed out

cessda-bitbucket-importer commented 3 years ago

Original comment by John Shepherdson (GitHub: john-shepherdson).

CSDA endpoint is available again and has been reharvested by staging. However, metadata is not in English, despite what is says in the UI.

‌

Harvesting config includes correct default language.

Repo(url=http://nesstar.soc.cas.cz/oai-pmh, code=CSDA, name=Czech Social Science Data Archive (CSDA), handler=NESSTAR, preferredMetadataParam=oai_ddi, setSpec=null, defaultLanguage=cs)

cessda-bitbucket-importer commented 3 years ago

Original comment by Taina Jääskeläinen.

Oh dear, we cannot have the data in the English catalogue. How come it goes to English even with default language? Can we set it to be cs, using the tools mentioned above in this issue? But I may not get the technical point!

cessda-bitbucket-importer commented 3 years ago

Original comment by John Shepherdson (GitHub: john-shepherdson).

CSDA has a misconfiguration Content type text/xml; charset=utf-8

Should be Content type text/xml; charset=UTF-8

Screenshot 2021-03-03 at 16.33.12.png

cessda-bitbucket-importer commented 3 years ago

Original comment by John Shepherdson (GitHub: john-shepherdson).

@‌TainaFSD we have set the default language to ‘cs’ for CSDA.

cessda-bitbucket-importer commented 3 years ago

Original comment by Taina Jääskeläinen.

Let me know when I can check in staging.

Should I make a metadata issue of their misconfiguration?

I will close the issue in the metadata issue tracker of their endpoint not being available https://github.com/cessda/cessda.metadata.office/issues/70.

‌

cessda-bitbucket-importer commented 3 years ago

Original comment by John Shepherdson (GitHub: john-shepherdson).

You can now check the Czech in staging.

Please make a new issue for CSDA misconfiguration and close this one.

cessda-bitbucket-importer commented 3 years ago

Original comment by Taina Jääskeläinen.

I now made an issue for CSDA in the metadata office issue tracker: #83 https://github.com/cessda/cessda.metadata.officeissues?status=new&status=open

Closing this issue.

But the CSDA data still appears in the English catalogue in staging, and Czech is not in the language drop-down list.