Expose image on export and publish

[craig-willis]

Problems addressed

• The registry requires authentication so it isn't possible for anyone to access the built image for a tale
• When publishing or exporting a tale, the image must be built locally with repo2docker, which results in a different image
• Third-party integrations (e.g., CORE2) can't fetch the image for archiving purposes

Approach

• https://github.com/whole-tale/deploy-dev/pull/52/ adds a new public read-only Docker registry at images.local.wholetale.org making it possible to pull any built image without authenticating. Since both registry and images share the same filesystem, any image can be pulled publicly.
• Adds the public image URL to the manifest on export and publish, handles import (assumes image is present in our registry)
• Updates logic for run-local.sh to build the image if no digest is present and otherwise pull.

To Test:

• Deploy with https://github.com/whole-tale/deploy-dev/pull/52/
• Create a tale without building the image
  • Export the tale then inspect and execute run-local.sh
  • Since image cannot be pulled, script should contain command to build locally with r2d
  • Optionally execute run-local.sh and confirm it works
• Run the tale in WT (i.e., build the image)
  • Export the tale and inspect and execute run-local.sh
  • There should be no r2d command and image should be pulled from images.local.wholetale.org
• Publish the tale to sandbox.zenodo.org
  • Inspect metadata.json, note image digest:

    {
        "@id": "images.local.wholetale.org/tale/b3fe3003ade99e0d3b07702430c16712@sha256:4672da847c7e21f39ddbfbcf5b44b8c55926f206640912f4bf5c7d7bd90f8a6d",
        "@type": "schema:SoftwareApplication",
        "schema:applicationCategory": "DockerImage"
    }

  • Delete and re-import the tale
  • Confirm that confirm the imageInfo digest matches but references registry.local.wholetale.org

Additional notes

• Commented out failing DV test
• Moved run-local template logic to new function because flake8 started complaining that stream() was too complex
• I'm not certain about using ManifestParser to convert from public to private registry on import. The idea is that an exported/published tale should reference images... but an internal/imported tale should always reference registry...
• This does nothing to address the archiving issue (https://github.com/whole-tale/whole-tale/issues/75), it only makes it possible to pull the image as long as we have it. As a separate task, we should consider backing up and migrating the registry on new deployments.
• This PR provides no solution to identifying images that are suitable for GC. We could tag images at certain points in the lifecycle (e.g., tags associated with versions, runs, public, or published tales), but needs more thought.
• There may be some concern about making all images public, but this seems low-risk since we do not include private code or data. For comparison, CO requires the image to be built locally for anything that hasn't been published.

[codecov[bot]]

Codecov Report

Merging #551 (8168d88) into master (c7d8afd) will increase coverage by 0.01%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #551      +/-   ##
==========================================
+ Coverage   92.81%   92.82%   +0.01%     
==========================================
  Files          60       60              
  Lines        4745     4767      +22     
==========================================
+ Hits         4404     4425      +21     
- Misses        341      342       +1     

Impacted Files	Coverage Δ
server/lib/exporters/bag.py	100.00% <100.00%> (ø)
server/lib/manifest.py	93.08% <100.00%> (+0.11%)	:arrow_up:
server/lib/manifest_parser.py	88.19% <100.00%> (+0.45%)	:arrow_up:
server/lib/dataverse/provider.py	95.96% <0.00%> (-0.41%)	:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[Xarthisius]

Works great, code changes LGTM! One issue I have is more a conceptual one: with this change you can get a different bag for the same versionId depending on what you did on the platform. Perhaps we should add imageInfo to the set of fields we use to detect if something "happened" to the Tale and create a new version for a generic export. However, that doesn't solve the issue when user selects specific version to export from the "version history" menu.