[suggestion] image manifest

[vsoch]

I'm setting up a small registry (a table that renders a listing of repo2docker containers for a resource) and thought it would be a good idea for a container registry to serve a small manifest for itself (e.g., a manifest that includes the various tags, etc.) and then the main registry to serve a global manifest (e.g., the URL to the image manifests hosted at the individual repos. Before I roll my own manifest (and note that the global one is probably going to be something super simple, for example https://vsoch.github.io/repo2docker-containers/library.json) I wanted to bring it up for discussion here, because it could be provided as part of the continuous build folder! Thoughts or ideas?

[vsoch]

I'm thinking something simple like:

• use a combination of jekyll / scripts during CI to generate new manifests for the containers (likely the manifest can come from docker / similar, and the summary of manifests jekyll)
• push to github pages from the PR on successful deployment, e.g., https://circleci.com/docs/2.0/gh-bb-integration/#security
• then link to the "master" manifest above from the registry repository of choice

And then there would be some simple command line script that knows how to parse those various pieces - starting with the master repo manifest, then allowing the user to find (and inspect) containers in the registry, most likely just to get tags (but maybe other things too?)

[vsoch]

Still lots to do, but just wanted to show you some work in progress! With the continuous-build setup I can now run a build and deploy back to github pages to automatically generate...

• inspection of packages
• general list of tags
• image manifests

And I'm (slowly) figuring out how to break the yaml into functions (more like snippets) so that redundant code is reduced. I'll have a whole write up / examples for how this comes together, actually I'm traveling and haven't had much time to work on this (sad dinosaur!) In a nutshell, we are going to have a workflow that looks like:

• a registry of containers that a user can select from (note right now this is just a pretty table), but soon each linking back to
• a github repo where the version controlled code is, and the new versions are built with testing, and deployed to Docker Hub and
• Github pages that serves not only a static API of the manifest / inspection, but also a nice web interface for the user to explore the container (I haven't added this code yet, but soon!) When this page exists, it will be at the primary Github Pages url (e.g., https://vsoch.github.io/repo2docker-keras/) so then a user can link to it from the registry.

The following two use cases are supported from the above:

Developer

Build a Container

• A developer can just copy the circle configuration with his / her notebook and requirements.txt, deifne the environment variables, and get this little "serves it's own metadata" Github repo and container set up.

Provide a Registry

• A developer can also deploy their own "registry" - which is basically a github pages site with testing for these repos. A PR to this registry coincides with adding an entry for a container repo, and doing the checks for Github pages (and then merging to add to the table).
• A developer can also just contribute to another registry via a PR.

User

The user aspect is cooler! I'm going to write a little command line script that basically parses the json served by the registry to allow the user to discover containers. If a user needs to search software within a container, given that each individual repo api provides it, this is something that is possible (albeit not at huge scale). This means that a user could do something like:

repo2docker-registry search jupyter
...
repo2docker-registry list
...

The tool would also just allow the user to search the table to browse container repos, find one he/she likes, then use the forward tool to start an interactive notebook. E.g. something like:

bash start.sh singularity-repo2docker <var1> ...

And really, the API served by the registry repo could plug in to wherever it's needed. :) Anyhoo, I am likely to be slow with the remaining travel (airports are really stressful... !!!) but I hope to be settled back in and put all these things together sooner than later!

[vsoch]

And I'm still working on this very heavily (expect a lot of future failed builds as I get it wrong!) but here is an example of what I'm learning about the Circle config --> https://circleci.com/gh/vsoch/repo2docker-keras/111#config/containers/0

It's actually nothing about circle, but about using anchors / aliases to define snippets of things (for reuse). The biggest challenge so far is with respect to environment variable scope, but I've found that writing to $BASH_ENV and then sourcing does the trick.

[vsoch]

Another update, the web interface (skeleton / base) is mostly done - the tags, manifests, and inspections are done automatically by the CI. I have it written up but will still be a bit before I can make it nice and ready to go for mass consumption :) --> https://vsoch.github.io/repo2docker-keras/

This setup is slightly more advanced because it requires creating a Github Machine User (another account with an email / keys) but it's still all web interface stuffs. Anyhoo, more to come! :)