Support datalad-based generation and deployment scenario

[mih]

Having gone through https://github.com/datalad/datalad-catalog/issues/311 I want to make a suggestion for a (possibly already possible) workflow that would avoid the repeated specification of essential setup properties (like --catalog-dir), and align the handling better with datalad tooling and features.

I think the task of maintaining a catalog is rather similar to the task of maintaining a software package repository (see https://datalad-debian.readthedocs.io). In a datalad context we could have the following setup:

catalog_maint/ [ds]
    www/ [ds]
    superds/ [ds]
         (records/)
         subds/ [ds]
            (records/)

where

• each line with [ds] is a datalad dataset
• www is the target of --catalog-dir
• records/ is some place in a datalad (super)dataset with metadata records that correspond to (non-)Datalad datasets (or simply some that are not accessible)

So conceptually we have a constellation of 3 datasets:

• (A) a superdataset for the purpose of maintaining a catalog (this is where datalad run would be used in to turn metadata into catalog entries)
• (B) a subdataset to receive the HTML/JSON catalog -- this could be cloned/updated on a server to "deploy" a web-accessible version
• (C) a subdataset that represents the source for the catalog -- either a simple/single dataset with a bunch of pre-crafted records, or the superdataset of a humongous hierarchy

a maintenance workflow would be

• check (C) for updates, stop if there are none
• execute datalad run within (A) to take information from a defined state of (C) to update (B), save (A) and (B) accordingly with the update record
• push (B)
• update clone of (B) on a webserver

within the scope of (A), all configuration for that specific catalog could be captured and stored, avoiding any per-call necessity to redeclare constants

[jsheunis]

Sample structure: https://github.com/psychoinformatics-de/abcdj-catalog/tree/6ac29c8f0e0cb9a10eb094a7d9b7c95a3daa6520

Consider structuring this as:

• a command line call, with user interaction to capture paths / names of datasets, which then creates the complete structure
• a template repository on github

Notes:

• the template repository would lock the version of datalad-catalog with which the catalog is created (although this could be overwritten), while the command line option would use the version of datalad-catalog that is installed.