quantifyearth / shark

3 stars 1 forks source link

Provide overrides for front matter input specifications #50

Closed mdales closed 4 months ago

mdales commented 4 months ago

In the past we have had the input key in the front matter definition for a sharkdown file:

---
inputs:
- "/data/test/1.csv"
- "/data/other/data.txt"
---
...

In this PR I do two things mostly:

This means that now your map would look like:

---
inputs:
  input1: "/data/test/1.csv"
  input2: "/data/other/data.txt"
---
...

And then you can invoke it with:

$ dune exec -- md --input1=/maps/mwd24/somedata.csv --input2=~/test/data.txt ...

This allows easier use of shark as a command line exploratory system in addition to being a more CI based thing. There's an interesting question of how you specify these inputs in a CI world - this kind of dynamic binding is somewhat the antithesis of what shark is trying to achieve (setting everything in stone for reproducibility reasons), but I think pragmatically we need something like this to deal with there exploratory science before you get the world set in stone.

To help with the traceability, this change will insert a new shark-import block into the markdown itself, so you can see what happened in the output markdown. This means if you have the results you can see what the substitutions were at least.

Additional changes by this PR: