Reference data storage API proposal

[jayrbolton]

This codebase could potentially be used for reference data storage, allowing us to keep a standardized, centralized place to upload & download RE import data, SDK app data, etc. I propose adding an endpoint that provides a JSON-RPC 2.0 api for uploading files:

Methods:

• upload_file - Upload a new file and get its ID. Returns a hex-encoded content hash of the file, which represents its ID. If the file already exists, then the user is given the file ID and this is a no-op. The user can provide some optional additional metadata which is stored in Minio's object metadata.
• download_file - Download a file from the store using its ID, which is its content hash.

The server can gzip files if they have not already been compressed. We can also have a Python client module that takes care of creating tarballs, gzip compression/extraction, proper low-memory streaming, and checking downloaded data against its content hash.

Optionally, we could have a policy to expire files if they haven't been accessed after a certain time period, such as 500 days.

Blake2b hashing should be used.

Users can upload directories if they pack them into a tarball, and the client can help with this. The server can detect if uploaded files are already gzipped using this

[xpe]

Blake2b hashing should be used.

Why do you recommend it? Speed of hashing?

Have you considered BLAKE3?

[jayrbolton]

Yeah, blake2b is a lot faster, I would use blake3, but the python package seems to be in development, and they warn you against using it in production without extra work. Blake2b is builtin to hashlib in py 3.6+ so very stable

[xpe]

A few contextual questions:

• What are the requirements?
• What existing file caching options have you considered? How well do they cover the requirements?

[jayrbolton]

Good questions..

• Requirements: ability for many users to access the same set of reference data in order to run programs such as importers, bioinformatics tools, etc. This is non-workspace data. My major use case is various importers for ArangoDB.
  • I'm realizing we may need auth for private research data. I can think of a couple use cases. I think we could implement this without much trouble using KBase auth. I will add to the spec above, but this feature could be added later
• Other options:
  • We could use a hosted service like GCS or S3. I really doubt KBase would go this route, and I have no lbl access to anything like this.
  • We could use the existing file cache server API. However, I think it uses concepts that aren't relevant here, and we could reuse a lot of the code for a different API without a lot of work.
  • We could use blobstore that Gavin wrote relatively recently. I'm not sure if it has extra API baggage we don't want, and I'm not sure if others don't want this store used for non-workspace data. But this is worth considering.

[jayrbolton]

In Slack, we tentatively decided to use the blobstore API for this: https://github.com/kbase/blobstore