🍀 Document the FOUR Data Pipelines

[nickumia-reisys]

User Story

In order to inform existing and new harvesting processes and procedures, the Data.go Architect Team wants to document the FOUR pipelines that all harvesting travels through. These pipelines will either be (1) optimized in the current system or (2) fed into building a better new system from the start.

Acceptance Criteria

• [x] GIVEN research and analysis has been done surrounding the pipelines \ WHEN I look at this ticket \ THEN there is documentation about how the pipelines are structured with supporting details or insight into the complex intricacies

Background

• https://github.com/GSA/data.gov/issues/4411
• CKAN Extension/Plugin Documentation
• The CKAN Harvesting Interface

Security Considerations (required)

...

Sketch

• Work through the data lifecycle and pipeline structure for each of the following pipelines
  • [x] file json DCAT
    • Classes of concern: DataJsonHarvester, DatasetHarvesterBase, HarvesterBase
  • [x] file xml FDGC/ISO
    • Classes of concern: GeoDataGovDocHarvester, DocHarvester, GeoDataGovHarvester, SpatialHarvester, HarvesterBase
  • [x] file xml waf FDGC
    • Classes of concern: GeoDataGovWAFHarvester, WAFHarvester, GeoDataGovHarvester, SpatialHarvester, HarvesterBase
  • [x] api json ARCGIS
    • Classes of concern: ArcGISHarvester, SpatialHarvester, HarvesterBase
  • Note: Each pipeline consists of a source file interface (file, api, waf), a file type association (json, xml) and a schema specification (dcat, fdgc, arcgis). The idea behind specifying these is to highlight areas of abstraction to make the code more relevant and reusable. A proposed NEW pipeline consists of api json DCAT for large DCAT data.json files that are unwieldy when processed as a single entity.
  • api json DCAT
    • PROPOSED
• Data lifecycle consists of: (1) Source creation, (2) Source execution (job run) schedule, (3) Gather Stage, (4) Fetch Stage, (5) Import Stage
  • Note: the current harvesting methodology does the create/update/delete/etc at non-sequential times through the gather/fetch/import stages. Be sure to note the real sequence as it stands currently. If we decide to optimize or change something, that should be done as follow on work.

[nickumia-reisys]

Note: TWO harvesting pipelines have been deprecated (I believe both of these are FGDC/ISO, but not sure):

• api csw ??? ???
• api cms ???

[nickumia-reisys]

Comment is in history. I deleted it to make the ticket cleaner. See diagrams below for the most up to date information.

[btylerburton]

Link to MD Translator spike: https://github.com/GSA/data.gov/issues/4200

[nickumia-reisys]

DCAT Pipeline initial pass complete.


... Moving on to file xml FDGC/ISO

Just as a random note, our DCAT code is much more unorganized compared to the spatial upstream code...

[nickumia-reisys]

Single XML Pipeline initial pass complete.


... Moving on to file xml waf FDGC/ISO tomorrow

[nickumia-reisys]

WAF XML Pipeline initial pass complete.


... Moving on to api json ARCGIS next

[nickumia-reisys]

ArcGIS Pipeline initial pass complete.


... I'm done? 🎉

[nickumia-reisys]

The diagrams in the comments above represent the core of the harvesting optimization problem. What happens when.. What errors are not being captured... What assumptions are made that fail to be true.. The next step is reviewing the code, abstracting it into meaningful chunks, testing the functionality, preserving the best parts and fixing the broken parts. One of the corner stones of implementing a new version of this code deals with the following requirement:

1.2.3 Data.gov should be able to adapt to new data formats not originally accounted for in its design.

The controller diagram highlights some high-level abstractions for input/output definitions. However, for example, within the extract component, we want to be able to support multiple file types (and possibly new ones, i.e. rdf), so creating an abstraction for between "data download" and "data parsing/input" would allow us to hook in a new file type. Further discussion will ensue and we'll answer a lot of open questions. From there, it'll be easier to start implementing and executing on this work.