quantifyearth / shark

3 stars 1 forks source link

Add import blocks supporting git #22

Closed patricoferris closed 4 months ago

patricoferris commented 5 months ago

In step with the "how far can we get with paths starting with /data" this PR adds very basic Import blocks to Shark. Import blocks at the moment are essentially run blocks using git to pull a repository into /data using a basic Alpine image to do so. For now making them a distinct separate block that is translated into a build script is a little overkill, but I think in the long run it will make it easier for us to support other sources of data and make it nicer for users to simple write:

```shark-import
https://foo.git /data/foo
ftp://bar /data/bar
file:///home/alice/project /data/project
```

Or something like that rather than manually doing an Obuilder script. For now the format is simple, (URL, PATH) pairs with everything being assumed to be git. There could some role for Vurl here for extra metadata, but I could imagine coming up with am Import DSL for doing more interesting imports of data e.g. importing branches from a git repo and embedding them in the file system.

```shark-import
https://foo.git |> branches test* |> /data/foo
```

The tests now run a simple ocaml-test.md script that uses an import statement and then opam and dune to build things. The invocations of opam and dune use there less common "provide the file path" variants in order to activate the automatic dependency tracking. A small price to pay for what is otherwise a very nice setup I think!

patricoferris commented 5 months ago

The test here uses ocaml/opam so we need to implement support in OBuilder for finding the image's user and actually uid and gid

patricoferris commented 5 months ago

Right -- opam install mutates /home/opam/.opam/default which is implicitly needed by dune build because that is where dune is installed, but that is all implicit! Hmmm :S