Closed jsheunis closed 1 year ago
As you stated: this is what other extensions (and datalad proper) are doing, for the reasons you gave. I see no reason why catalog
should be different.
That being said, there is a long-standing desire for being able to do non-cumbersome subcommands https://github.com/datalad/datalad/issues/4539 But so far nobody considered it important enough to actually do it. For me personally the resulting nightmare of backward compatibility crutches makes this a rather unattractive direction to think about.
This effort would be a good opportunity to introduce Constraints
from datalad-next
Current thinking of new commands + required/optional params + constraints:
datalad catalog-create
:
Create a new catalog with or without metadata, and with or without a specified configuration file (defaults if not provided)
"-c", "--catalog_dir"
: path-like; catalog directory; required"-m", "--metadata"
: path-like; metadata file; optional"-y", "--config-file"
: path-like; configuration file; optional, content must be json object"-f", "--force"
: required if catalog_dir
already exists and if user wants to force in-place create command (i.e. replace parts of the catalog)datalad catalog-add
:
Add metadata to an existing catalog, with or without a specified dataset-level configuration file (defaults to the catalog-level config if not provided)
"-c", "--catalog_dir"
: path-like, catalog directory; required + must already exist"-m", "--metadata"
: path-like, metadata file; required; must exist, content must be json lines"-y", "--config-file"
: path-like, configuration file; optional, content must be json objectdatalad catalog-validate
:
Validate metadata against the catalog schema. The schema version is determined from the catalog, if provided, otherwise from the latest supported version of the package installation.
"-m", "--metadata"
: path-like, metadata file; required; must exist, content must be json lines"-c", "--catalog_dir"
: path-like, catalog directory; optional (must already exist if provided)datalad catalog-translate
:
Translate a file with metalad-extracted metadata items from particular extractor structures into the catalog schema. The to-be-translated-to schema version is determined from the catalog, if provided, otherwise from the latest supported version of the package installation. Available translators (registered as entrypoints) will be filtered based on own criteria (extractor) to find the appropriate one.
"-m", "--metadata"
: path-like, metadata file; required; must exist, content must be json lines"-c", "--catalog_dir"
: path-like, catalog directory; optional (must already exist if provided)datalad catalog-serve
:
Start a local http server for viewing/testing a local catalog.
"-c", "--catalog_dir"
: path-like, catalog directory; required + must already existdatalad catalog-remove
:
Remove metadata of a specific dataset ID and VERSION from an existing catalog:
"-c", "--catalog_dir"
: path-like, catalog directory; required + must already exist"-i", "--dataset_id"
: string; required"-v", "--dataset_version"
: string; required--force
flag?datalad catalog-workflow
:
Run a workflow of metadata extraction, translation, and catalog (entry) generation, given a DataLad dataset hierarchy and a specified subcommand:
"subcommand"
: string; required; one of ["new", "update"]"-c", "--catalog_dir"
: path-like; catalog directory; required (must exist if subcommand == "update"
)"-d", "--dataset-path"
: path-like; dataset on which to run the workflow; required (must be a datalad dataset)"-s", "--subdataset-path"
: path-like; dataset on which to run the workflow; required if subcommand == "update"
"-r", "--recursive"
: store_true
(default False
); optional; whether to recurse into subdatasets or not during workflow execution"-R", "--recursion-limit"
: int; optional; "-e", "--extractor"
: str; nargs="+"
; optional (must be from the list of available extractor names)datalad catalog-set
:
Utility for setting various properties of a catalog, based on the specified subcommand. Used to set the catalog home page, or to reset config at catalog- or dataset-level.
"-c", "--catalog_dir"
: path-like, catalog directory; required + must already exist"subcommand"
: string; required; one of ["home", "config"]"-i", "--dataset_id"
: string; required if subcommand == "home"
(used for setting home page id); optional if subcommand == "config"
(used for resetting config at dataset-level)"-v", "--dataset_version"
: string; required if subcommand == "home"
(used for setting home page version); optional if subcommand == "config"
(used for resetting config at dataset-level)"-y", "--config-file"
: path-like, configuration file; required if subcommand == "config"
, used for setting home page versiondatalad catalog-get
:
Utility for getting various properties of a catalog, based on the specified subcommand.
"-c", "--catalog_dir"
: path-like, catalog directory; required + must already exist"subcommand"
: string; required; one of ["home", "config", "metadata", "tree"]"-i", "--dataset_id"
: string; optional if subcommand == "config"
(used for getting config at dataset-level); required if subcommand == "metadata"
(used for getting metadata of specific dataset id+version)"-v", "--dataset_version"
: string; optional if subcommand == "config"
(used for getting config at dataset-level); required if subcommand == "metadata"
(used for getting metadata of specific dataset id+version)I agree with all reasons for the introduction of an extended command suite. I also think that verbosity is not really a problem. Most invocations will probably be scripted anyway.
The command suite makes sense to me. I have only one comment on -y
, --config-file
. The text states it must be a JSON-object. In the API you would probably expect a dict
(although other types would be valid JSON-objects), on the command line a YAML or JSON file.
This effort would be a good opportunity to introduce Constraints from
datalad-next
I agree.
thanks @christian-monch!
Agree with everything so far. Only minor comment:
-y, --config-file
is a bit strange. Where's the y
coming from? How about a "standard" file argument -F, --config-file
(capitalized to not read as conventional "force")?
Thanks for catching, I had gotten used to it. I think it's some legacy thing where the file was originally a YAML file, hence y
. But F feels like a good option.
Currently,
catalog
is the only entry point in the command suite, while the argument that immediately follows is interpreted as the subcommand, such ascreate
,add
,serve
, etc.Would it make sense to rather move towards extending the number of entry points in the command suite, and then do away with the subcommands?
Reasons for:
Reasons against:
datalad catalog-create ...
ordatalad catalog-set-super ...
(or is this fine?)