Make all recipeSpec details out of scope

[mvahowe]

We established in last week's meeting that, to properly handle additions, deletions and alterations of metadata, we would need to "come up with some sort of language". We also seem to have agreed that this would take 10 years (even though inventing Javascript only took 10 days, in C.) Taken together, those two statements mean that the details of recipeSpecs need to be out of scope. This PR shows what that looks like.

The relevant section of textTranslation.json is

  "recipeSpecs": [
    {
      "variantId": "p1",
      "processor": "https://thedigitalbiblelibrary.org/barjo/0.1",
      "algorithmFormat": "https://thedigitalbiblelibrary.org/dblscript/0.1/dblscript.schema.json",
      "algorithm": "https://thedigitalbiblelibrary.org/recipes/text_translation_booklists.dblscript.json",
      "data": "recipes/data/p1.json"
    },
    {
      "variantId": "p2",
      "processor": "https://thedigitalbiblelibrary.org/barjo/0.1",
      "algorithmFormat": "https://thedigitalbiblelibrary.org/dblscript/0.1/dblscript.schema.json",
      "algorithm": "recipes/scripts/something_generic.dblscript.json",
      "data": "recipes/data/p2.json"
    },
    {
      "variantId": "p3",
      "processor": "https://thedigitalbiblelibrary.org/barjo/0.1",
      "algorithmFormat": "https://thedigitalbiblelibrary.org/dblscript/0.1/dblscript.schema.json",
      "algorithm": "recipes/scripts/something_specific.dblscript.json",
      "data": "recipes/data/p3.json"
    },
    {
      "variantId": "somethingExotic",
      "processor": "https://opaque-url.org/dev/null"
    }
  ]

The minimum requirement is the name of the variant that is produced and the processor that produces it. The processor is identified by a URL, because this provides a unique id without a central registry. (I would hope that there would be useful information at the end of that URL but, personally, I can't imagine any toolchain running code from such a URL "blind".)

algorithmFormat optionally specifies the language used to describe a transformation (assuming that a processor can do more than one transformation). This could be a JSON or XML schema or something else. algorithm optionally specifies the actual transformation, either by referring to a URL or by linking to an ingredient within the burrito.

data optionally provides a path to an ingredient containing data to be passed to the algorithm, such as

• vendor-specific markup for modifying the source burrito metadata
• flags and other settings to affect processing by a generic algorithm

This approach seems to me to allow for a wide range of processing models, and for as much or as little transparency about that processing model as anyone wishes to express.

Obviously each idServer will decide which recipeSpecs to accept and/or process. I'll add config settings to the burrito store config files to control this.

If we go this way, DBL will begin work on a proprietary recipeSpec language that will replace the current DBL publications/licenses system. This will be the only way to create derived variants within DBL after the reboot in Q3 2021. DBL will work in collaboration with any other group that commits to implementing or using that language. Who knows, maybe it will become a de facto standard.

[jonathanrobie]

I would really find it helpful to know exactly what kinds of operations are needed, and for what. A simple, clear writeup of that would make the requirements clearer to me.

There are many existing solutions for problems close to what you describe. For instance, JSON Patch ([RFC 6902] (https://tools.ietf.org/html/rfc6902)) might be plausible.

http://jsonpatch.com/

JSON Diff is not a standard, but it may also be worth considering:

https://pypi.org/project/json-diff/

[mvahowe]

I don't think you understand the problem I'm trying to solve but, regardless, we need to decide whether processing is in scope or not. If not, it doesn't matter whether anyone understands the problems, or how anyone decides to address them.

[jonathanrobie]

I am quite certain that I do not understand the problem you are trying to solve. I would like to understand before deciding whether it is in scope or not. I would like to know if it is directly relevant to the problem Scripture Burrito solves. I would like to know if implementations will use this for interoperability. I would like to know what operations are required, for what kinds of use cases.

For me, those are the things I need to know to say whether it is in scope.

[mvahowe]

DBL at least is going to want a way to do this that includes variables, iteration/mapping and conditionals, and probably doesn't want to support multiple alternatives.

[jonathanrobie]

DBL at least is going to want a way to do this that includes variables, iteration/mapping and conditionals, and probably doesn't want to support multiple alternatives.

Could you please give a clear description of the problem to which this is a solution? And could you please explain whether this affects interoperability requirements for an application that provides burritos to DBL or consumes them from DBL, and how?

[mvahowe]

This PR makes all those questions out of scope. Last time we talked about this, that's what you wanted.

[mvahowe]

DBL is going to continue to process Scripture text that is uploaded, as it has for years. The current solution is proprietary. I'm happy for the new, enhanced solution to be proprietary too.

[mvahowe]

Last time we went around this you delivered your verdict:

I'm inclined to call YAGNI on this - You Ain't Gonna Need It. Including this feature will make it harder to finish our work.

So this is my response - no standard way to do any form of processing, and then I'll go solve the problem somewhere else. According to your quote above, this will make it easier to finish our work.