Refactor processing into cldfbench makecldf

[johenglisch]

Hm, I am trying to wrap my head around the command-line interface for pydictionaria. Because of the decentralised CLDFbench approach a lot of the functionality needs to be rethought:

• check: This is… barely maintained? Every time we ran this, we ignored like half the output because the behaviour of process changed when CLDF support was introduced.

• ls: Needs to redirected to read the info from the CLDFbenches.

• new: Replaced by cldfbench new --template dictionaria_submission

• process: Replaced by cldfbench makecldf

• release: Also done from within the CLDFbench?

• upload_media: How would this work in a CLDFbench world?

• add_comparison_meanings: will live on in some shape after our new Dictionaria × Concepticon marriage

Also, if the *-intern repo does not contain any actual data, it might be worth thinking about getting rid of the submission vs submission-internal split in the folder structure:

+- contributions.json   # submission metadata
+- datasets/            # data downloaded from Github/Zenodo
+- etc/                 # any submission data not part of the cldfbench (whatever that might be)
   +- submission-id-1/
   +- submission-id-2/
   +- …

Then, moving a submission from --internal to published could be done by just flicking a boolean, e.g.:

{
    "contributions": [
        {
            "sid": "tseltal",
            "number": 10,
            "doi": "10.5281/zenodo.3668865",
            "repo": "https://github.com/dictionaria/tseltal",
            "published": true
        },
        …
    ]
}

[xrotwang]

I'd just remove check and ls, replace new and process as you say. And then refactor the rest as cldfbench subcommands contributed by pydictionaria - via the the cldfbench.commands entry point.

[xrotwang]

In particular for the concepticon integration, a cldfbench subcommand will make accessing the Concepticon data simpler.

[IrenH]

I think all of the functionality that check had is now included in makecldf and then some, isn't it? cldf.log and examples.log are far superior to the original 'check' output. I am just not sure whether the media lookup has also been included in the cldf.log, but I think so...? (missing XYZ)

[johenglisch]

So after this, there will be no dictionaria command anymore and the pydictionaria code won't even know what a dictionaria-intern even is (because that will only be relevant to the webapp)? I kinda like the sound of that. (^^)

@IrenH Yes, iirc the processing code will just dump any unknown filenames into the cldf.log.

[xrotwang]

Yes, no more pydictionaria cli. And maybe - at least as far as loading the web app is concerned - dictionaria-intern could be replaced by a simple JSON file, that may even live in clld/dictionaria.

[IrenH]

with this whole new workflow are we still using private repositories for new submissions? each dictionary is in its own repository now, right? so we just set it from private to public once it is published? or how will this work? thanks!

[xrotwang]

Yes. Upon submission a new, cldfbench-ready, private repos will be created, and made public upon publication.

[IrenH]

that sounds good!