Move from subcommands to an extended command suite

[jsheunis]

Currently, catalog is the only entry point in the command suite, while the argument that immediately follows is interpreted as the subcommand, such as create, add, serve, etc.

Would it make sense to rather move towards extending the number of entry points in the command suite, and then do away with the subcommands?

Reasons for:

• This would make datalad-catalog more similar to other extensions such as metalad and gooey.
• Whenever a new subcommand needs to be added, this subcommand has to share the flag/argument space with all other subcommands. This is cumbersome to keep track of and sometimes requires what feels like unnecessary gymnastics.
• Adding a new entrypoint is quite seamless and does not have to affect existing entrypoints/arguments/flags at all

Reasons against:

• Command line calls might become a bit too verbose, e.g. datalad catalog-create ... or datalad catalog-set-super ... (or is this fine?)
• other reasons?

[mih]

As you stated: this is what other extensions (and datalad proper) are doing, for the reasons you gave. I see no reason why catalog should be different.

That being said, there is a long-standing desire for being able to do non-cumbersome subcommands https://github.com/datalad/datalad/issues/4539 But so far nobody considered it important enough to actually do it. For me personally the resulting nightmare of backward compatibility crutches makes this a rather unattractive direction to think about.

[jsheunis]

This effort would be a good opportunity to introduce Constraints from datalad-next

[jsheunis]

Current thinking of new commands + required/optional params + constraints:

datalad catalog-create:

Create a new catalog with or without metadata, and with or without a specified configuration file (defaults if not provided)

• "-c", "--catalog_dir": path-like; catalog directory; required
• "-m", "--metadata": path-like; metadata file; optional
• "-y", "--config-file": path-like; configuration file; optional, content must be json object
• "-f", "--force": required if catalog_dir already exists and if user wants to force in-place create command (i.e. replace parts of the catalog)

datalad catalog-add:

Add metadata to an existing catalog, with or without a specified dataset-level configuration file (defaults to the catalog-level config if not provided)

• "-c", "--catalog_dir": path-like, catalog directory; required + must already exist
• "-m", "--metadata": path-like, metadata file; required; must exist, content must be json lines
• "-y", "--config-file": path-like, configuration file; optional, content must be json object

datalad catalog-validate:

Validate metadata against the catalog schema. The schema version is determined from the catalog, if provided, otherwise from the latest supported version of the package installation.

• "-m", "--metadata": path-like, metadata file; required; must exist, content must be json lines
• "-c", "--catalog_dir": path-like, catalog directory; optional (must already exist if provided)
• TODO: if the catalog was created with a previous schema version, then to-be-added metadata will need to be validated against this previous schema version (of the catalog itself), not necessarily the latest schema version supported by the package installation (which is currently the case). Issue to be created.

datalad catalog-translate:

Translate a file with metalad-extracted metadata items from particular extractor structures into the catalog schema. The to-be-translated-to schema version is determined from the catalog, if provided, otherwise from the latest supported version of the package installation. Available translators (registered as entrypoints) will be filtered based on own criteria (extractor) to find the appropriate one.

• "-m", "--metadata": path-like, metadata file; required; must exist, content must be json lines
• "-c", "--catalog_dir": path-like, catalog directory; optional (must already exist if provided)

datalad catalog-serve:

Start a local http server for viewing/testing a local catalog.

• "-c", "--catalog_dir": path-like, catalog directory; required + must already exist

datalad catalog-remove:

Remove metadata of a specific dataset ID and VERSION from an existing catalog:

• "-c", "--catalog_dir": path-like, catalog directory; required + must already exist
• "-i", "--dataset_id": string; required
• "-v", "--dataset_version": string; required
• Notes:
  • Possibly requires interaction with CLI to confirm removal ("Are you sure?"), or otherwise --force flag?

datalad catalog-workflow:

Run a workflow of metadata extraction, translation, and catalog (entry) generation, given a DataLad dataset hierarchy and a specified subcommand:

• "subcommand": string; required; one of ["new", "update"]
• "-c", "--catalog_dir": path-like; catalog directory; required (must exist if subcommand == "update")
• "-d", "--dataset-path": path-like; dataset on which to run the workflow; required (must be a datalad dataset)
• "-s", "--subdataset-path": path-like; dataset on which to run the workflow; required if subcommand == "update"
• "-r", "--recursive": store_true (default False); optional; whether to recurse into subdatasets or not during workflow execution
• "-R", "--recursion-limit": int; optional;
• "-e", "--extractor": str; nargs="+"; optional (must be from the list of available extractor names)
• Notes:
  • Could possibly infer workflow type (new/update) based on (1) whether catalog already exists or not, (2) whether the (sub)dataset being processed already has content in the catalog or not

datalad catalog-set:

Utility for setting various properties of a catalog, based on the specified subcommand. Used to set the catalog home page, or to reset config at catalog- or dataset-level.

• "-c", "--catalog_dir": path-like, catalog directory; required + must already exist
• "subcommand": string; required; one of ["home", "config"]
• "-i", "--dataset_id": string; required if subcommand == "home" (used for setting home page id); optional if subcommand == "config" (used for resetting config at dataset-level)
• "-v", "--dataset_version": string; required if subcommand == "home" (used for setting home page version); optional if subcommand == "config" (used for resetting config at dataset-level)
• "-y", "--config-file": path-like, configuration file; required if subcommand == "config", used for setting home page version

datalad catalog-get:

Utility for getting various properties of a catalog, based on the specified subcommand.

• "-c", "--catalog_dir": path-like, catalog directory; required + must already exist
• "subcommand": string; required; one of ["home", "config", "metadata", "tree"]
• "-i", "--dataset_id": string; optional if subcommand == "config" (used for getting config at dataset-level); required if subcommand == "metadata" (used for getting metadata of specific dataset id+version)
• "-v", "--dataset_version": string; optional if subcommand == "config" (used for getting config at dataset-level); required if subcommand == "metadata" (used for getting metadata of specific dataset id+version)

[christian-monch]

I agree with all reasons for the introduction of an extended command suite. I also think that verbosity is not really a problem. Most invocations will probably be scripted anyway.

The command suite makes sense to me. I have only one comment on -y, --config-file. The text states it must be a JSON-object. In the API you would probably expect a dict (although other types would be valid JSON-objects), on the command line a YAML or JSON file.

[christian-monch]

This effort would be a good opportunity to introduce Constraints from datalad-next

I agree.

[jsheunis]

thanks @christian-monch!

[bpoldrack]

Agree with everything so far. Only minor comment:

-y, --config-file is a bit strange. Where's the y coming from? How about a "standard" file argument -F, --config-file (capitalized to not read as conventional "force")?

[jsheunis]

Thanks for catching, I had gotten used to it. I think it's some legacy thing where the file was originally a YAML file, hence y. But F feels like a good option.