Add ability to easily access experiment results from object storage (COS/S3)

[lresende]

Currently, the build and experiment interfaces are sort of disconnected on the UI side and it would be good to start working on this integration. One idea would be to create an S3 file browser that would give you access to all the S3 instances configured in runtime metadata.

I was thinking about having something like a multiple S3 instance file browser

├─── S3 instance 1
├─── bucket 1
│   ├─── experiment
│   │    notebook.ipynb
│   │    notebook.html
│   │    data_output.csv
├─── bucket 2
│   ├─── experiment
│   │    notebook.ipynb
│   │    notebook.html
│   │    data_output.csv
├─── S3 instance 1
├─── bucket 1
│   ├─── experiment
│   │    notebook.ipynb
│   │    notebook.html
│   │    data_output.csv

Once you browse and file the file you want, it should give you the ability to open/visualize or copy the file locally where the original file is located

Some possible references:

• https://github.com/youngthejames/jupyterlab_filetree
• https://www.npmjs.com/package/multicontentsmanager
• https://github.com/jupyterlab/jupyterlab-git/issues/684
• https://github.com/viaduct-ai/hybridcontents
• https://github.com/jpmorganchase/jupyter-fs

[akchinSTC]

@ptitzler - This should be the S3 issue

[romeokienzler]

discussing this next week

[ptitzler]

Trying to generalize the Elyra specific needs this would probably translate to the following high level requirements. (Note that the intention of the requirements listed below is not to define a complete S3 "browser" solution, which would need to support browsing, uploading, and downloading of objects, in addition to bucket management operations. I am only trying to identify the minimum functionality required to make it possible for users to _access pipeline artifacts that are stored in a bucket on OS from the JupyterLab UI.)

• An S3-browser solution should provide value on it's own (e.g. be useful as as a general purpose S3 browser/download user interface, limited as it might be).
• An S3 browser should support connecting to multiple S3 instances.
• It should be possible to connect to an S3 instance interactively - the user enters connectivity information in the UI
• It should be possible to programmatically connect to an S3 instance. In the case of Elyra, the connectivity information is already stored in a runtime configuration, and a user probably shouldn't have to re-enter the credentials to use the browser.
  • It should be possible to browse the bucket list
  • It should be possible to browse the bucket content, which includes objects without a key name prefix (file.txt), with a key name prefix (/path/to/file.txt) and key name prefixes (aka "directories" - /path/to)
  • It should be possible to download individual objects from a bucket into the current work directory in JupyterLab, to make them accessible from the File Browser
  • It should be possible to set the cwd location to which the browser view opens. This would support a scenario where a user could automatically be taken to a location of interest. For example, after a pipeline run completes, a user might open a "link", which opens the browser to the location where the artifacts for the experiment run are stored.

[ptitzler]

UI observations for jupyter-fs v0.0.5 / jupyterlab_filetree v0.2.2, which I've connected to a minio instance:

• It should provide value on it's own: too many issues to be usable; see details below
• It should support connecting to multiple S3 instances: ok (one tab per s3 resource)
• It should be possible to connect to an S3 instance interactively: ok, although not very user friendly (configured via settings file)
• It should be possible to browse the bucket list: not supported
• It should be possible to browse the bucket content:
  • list objects without a key name prefix: ok
  • list empty "directory": fail/bug?
  • list "directory" with content: fail/bug?
• It should be possible to download individual objects ...:
  • trying to download any object results in Refusing to serve hidden file, via 404 Error

General observations:

• Upload is very slow
• Cannot create a "directory" with a custom name
• Cannot cd into newly created "directory"