BI-Library: CDS. New items in the institutional repository

[ErnestaP]

KPIs descriptions can be found here

• [x] The queries are not full. Ask Salome, Paulina, and Lydia for clarification. We might use the output from the CDS script. More information in the ticket link below. Queries: Records harvested by INSPIRE through arXiv: 037:'arXiv' and year:YYYY not 980:hidden Year 2021: 037:'arXiv' and year:2021 not 980:hidden https://cds.cern.ch/search?ln=en&p=037%3A%27arXiv%27+and+year%3A2021+not+980%3Ahidden&action_search=Search&op1=a&m1=a&p1=&f1=&c=CERN+Document+Server&sf=&so=d&rm=&rg=10&sc=1&of=hb&wl=0 Year 2022: 037:'arXiv' and year:2022 not 980:hidden https://cds.cern.ch/search?ln=en&p=037%3A%27arXiv%27+and+year%3A2022+not+980%3Ahidden&action_search=Search&op1=a&m1=a&p1=&f1=&c=CERN+Document+Server&sf=&so=d&rm=&rg=10&sc=1&of=hb&wl=0 Year 2023: 037:'arXiv' and year:2023 not 980:hidden https://cds.cern.ch/search?ln=en&p=037%3A%27arXiv%27+and+year%3A2023+not+980%3Ahidden&action_search=Search&op1=a&m1=a&p1=&f1=&c=CERN+Document+Server&sf=&so=d&rm=&rg=10&sc=1&of=hb&wl=0

  Records harvested by INSPIRE through curators: 035:'oai:inspirehep.net' not 037:'arXiv' and year:YYYY

  Year 2021: 035:'oai:inspirehep.net' not 037:'arXiv' and year:2021 https://cds.cern.ch/search?ln=en&p=035%3A%27oai%3Ainspirehep.net%27+not+037%3A%27arXiv%27+and+year%3A2021&action_search=Search&op1=a&m1=a&p1=&f1=&c=CERN+Document+Server&sf=&so=d&rm=&rg=10&sc=1&of=hb&wl=0 Year 2022: 035:'oai:inspirehep.net' not 037:'arXiv' and year:2022 https://cds.cern.ch/search?ln=en&p=035%3A%27oai%3Ainspirehep.net%27+not+037%3A%27arXiv%27+and+year%3A2022&action_search=Search&op1=a&m1=a&p1=&f1=&c=CERN+Document+Server&sf=&so=d&rm=&rg=10&sc=1&of=hb&wl=0 Year 2023: 035:'oai:inspirehep.net' not 037:'arXiv' and year:2023 https://cds.cern.ch/search?ln=en&p=035%3A%27oai%3Ainspirehep.net%27+not+037%3A%27arXiv%27+and+year%3A2023&action_search=Search&op1=a&m1=a&p1=&f1=&c=CERN+Document+Server&sf=&so=d&rm=&rg=10&sc=1&of=hb&wl=0

• [x] Ask Paulina, how often we want to retrieve this data Follow up: Every year

---

Setup Airflow workflow:

• [x] Add tables migration
• [x] Write a DAG

---

DB

• [x] Connect Airflow workflow to DB tables
• [x] Connect Supserset BI to DB tables

---

Feedback regarding results

• [ ] Ask Salome, Paulina, Lydia for feedback regarding the results we are getting from CDS. Adjust if needed

---

Deploy

• [x] Deploy to QA
• [x] Deploy to PROD

[ErnestaP]

Ticket: https://cern.service-now.com/service-portal?id=ticket&table=u_request_fulfillment&n=RQF2659173 Considering: solution, that CDS will run a script on their side and we will upload the result in db