Transform FGDC/CSDGM source to DCAT-US

btylerburton commented 8 months ago

User Story

In order to test another facet of the transformation step in our pipeline, datagovteam would like to transform an FGDC/CSDGM WAF source to DCAT-US and push that into CKAN.

This story depends on:

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

[ ] GIVEN a harvest source with an CSDGM schema, stored in a WAF file format WHEN I add that to our ETL Pipeline with the appropriate harvest source config THEN I expect the mdTranslator to transform the source into valid DCAT-US datasets AND I expect it to be able to pushed into a CKAN development instance without error.

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

No work planned outside the cloud.gov boundary.

Sketch

[ ] Configure new Harvest source pointing to a test WAF instance of CSDGM standard datasets
[ ] Extract that WAF using datagov-harvesting-logic WAF extract task
[ ] Transform datasets into DCAT-US using the MDTranslator application
[ ] Validate that the transformation conforms to DCAT-US standard
[ ] Push valid DCAT-US datasets into CKAN
[ ] Confirm that integration is successful and idempotent by running a test harvest source through the pipeline multiple times.

btylerburton commented 8 months ago

Combined with this ticket https://github.com/GSA/data.gov/issues/4565

btylerburton commented 8 months ago

Reopening in light of the fact that DCAT-US writer is not yet available.

rshewitt commented 1 week ago

confirmed with johnathan that fgdc reader is in good shape. "it’s one of the better ones".

GSA / data.gov