Identify & flesh out discrete functionality of the conveyor belt

[netsensei]

With this pull request in the works, we can now flesh out / isolate different functions in separate command classes.

Question: what should does tool do anyway? Answer: it's a "glue tool", which means it brings different Catmandu modules together so you don't have to write boilerplate bash or perl to set up a conveyor belt.

However, there are still discrete business requirements to be met. I can identify these variations of input / output. I can put them into 2 categories.

Push data to a datahub instance.

• Transport data from an API endpoint to a datahub instance.
  • Be able to differentiate API endpoint (Adlib instance, CollectiveAccess instance, etc.)
  • Be able to differentiate the exchange format (LIDO, DublinCore, MARCXML)
• Transport data from a local or remote database source to a datahub instance
  • Be able to differentiate DBMS's (MySQL, Mongo, etc.).
  • Be able to differentiate the exchange format (LIDO, DublinCore, MARCXML)
• Export data from a local flat file source to a datahub instance
  • Be able to differentiate the type of flat file (Adlib XML, CollectiveAccess XML,...)
  • Be able to differentiate the exchange format (LIDO, DublinCore, MARCXML)

Export data to a local flat file

• Export data from an API endpoint to a local flat file destination
  • Be able to differentiate API endpoint (Adlib instance, CollectiveAccess instance, etc.)
  • Be able to differentiate the exchange format (LIDO, DublinCore, MARCXML)
• Export data from a local flat file source to a local flat file destination (no push to datahub!)
  • Be able to differentiate the type of flat file (Adlib XML, CollectiveAccess XML,...)
  • Be able to differentiate the exchange format (LIDO, DublinCore, MARCXML)
• Export data from a local or remote database source to a local flat file destination
  • Be able to differentiate DBMS's (MySQL, Mongo, etc.).
  • Be able to differentiate the exchange format (LIDO, DublinCore, MARCXML)

We should be weary though "reinventing the wheel" here as Catmandu already does a lot of the heavy lifting out of the box.

[netsensei]

This statement still holds true:

We should be weary though "reinventing the wheel" here as Catmandu already does a lot of the heavy lifting out of the box.

It's extremely hard to build a generic tool. Let's ruthlessly guard the scope here. The tool should work within the context of the Flemish Art Context. So, only add things that can't be done with a vanilla Catmandu.

The main goal of the tool is to quickly set up and maintain robust ETL pipelines within an existing infrastructure. Only add new importer & exporter modules when new applications ar added to the infrastructure.

Use open formats and protocols as best as you can. Treat OAI-PMH as a first class citizen + avoid implementing custom API's of systems in separate modules as much as possible.