Ingest data from the TDC data sources into the portal

[Uchechukwu-Onye-Igbo]

As a user of the TDC portal, I want the data from the priority TDC data sources to live in the portal so that I can access and preview the data directly through the portal.

The sources to ingest are:

• JRC-IDEES (Already handled by @Mikanebu) 40% only, but will finish it
• EUROSTAT (by @luccasmmg)
• GFEI (by @luccasmmg)

Tasklist

• [x] Prepare ingestion script for JRC source
• [x] Prepare ingestion script for GFEI
• [x] Prepare ingestion script for EUROSTAT
• [x] Publish all 3 sources to our data portal
• [x] Setup Github CI/CD for running on regular basis

Acceptance Criteria

Given that I am interested in a data from any of the priority sources on the TDC portal,

• [x] I can preview and interact with the data directly in the portal
• [x] I can download the full data directly from the portal without having to be redirected out of the portal
• [x] Data will be updated on regular basis with keeping previous versions of the resource

[Verena205]

The other ones should be Eurostat (https://tdc-data-portal.vercel.app/search?data_provider="EUROSTAT") and GFEI data --> @khaeru will provide more information on this

[Verena205]

The GFEI is related to the #17
The dataset is yet to be included in the website

[khaeru]

• The original GFEI data is at https://doi.org/10.5281/zenodo.10148348.
• We have code at https://github.com/transport-data/gfei-2023 that converts these to SDMX.
• @PierpCazzola or his collaborator Leonardo Paoli will be updating the Zenodo record to include the converted SDMX files; hopefully in the next week or so.
• IMHO this data source should not be special-cased; rather we need a general-purpose capability to ‘ingest’ any record from Zenodo and similar permanent archives to a corresponding TDC CKAN record. This could involve:
  • Transforming Zenodo metadata to TDC metadata.
  • Displaying a clear link to the original Zenodo record.
  • Either mirroring the files, or linking to the files in their original location. (I am not sure what value is added by mirroring.)
  • Monitoring and refreshing the TDC CKAN record as the Zenodo record is updated.

[luccasmmg]

@khaeru im 99% that is not the case but just want to make it clear, this general purpose action where we could provide any random url from Zenodo and create ckan datasets based on that, would not require us to convert the data into SDMX format, since im assuming thats quite dependant on the shape of the file

[nicolas-becker]

Hi @luccasmmg & @Mikanebu , the SDMX version of the GFEI data was shared with us today.

Regarding your last comment, the general-purpose capability of ingesting data sources from Zenodo proposed by @khaeru (please correct me if I am wrong) is unrelated to it being SDMX formatted or not, as you noticed correctly. The SDMX conversion is rather related to the capability of providing a preview of the data on our platform and to further process/merge it with similarly formatted data sources. Therefor, you can use this SDMX data source as a use case to test the tabular preview of the TDC Formatted data (#171).

[Uchechukwu-Onye-Igbo]

Hello @nicolas-becker and @khaeru

You would recall we agreed to ingest three data sources: JRC-IDEES, EUROSTAT, and GFEI. Additionally, since the GFEI source has a version SDMX format, we will include it as a second resource - just like you said we would in the call yesterday.

cc @Mikanebu @luccasmmg

[nicolas-becker]

Hi @Uchechukwu-Onye-Igbo , yes this is what we agreed on. But if you add this sdmx version as a second resource, it would still be visualized in the dataset preview, right? This is why we have the following dropdown menu:

My comment was refering to the question of @luccasmmg , who is implementing the visualization of SDMX compliant data (or TDC Formatted data). The idea is that this dataset could be used by him to test this functionality.

[Uchechukwu-Onye-Igbo]

Hi @Uchechukwu-Onye-Igbo , yes this is what we agreed on. But if you add this sdmx version as a second resource, it would still be visualized in the dataset preview, right? This is why we have the following dropdown menu:

My comment was refering to the question of @luccasmmg , who is implementing the visualization of SDMX compliant data (or TDC Formatted data). The idea is that this dataset could be used by him to test this functionality.

Okay. @luccasmmg kindly take a look on this comment and revert to @nicolas-becker