jos or minio - requirements

[agstephens]

Requirements:

1. Level of abstraction: simplify:

   • [x] PUT: should not require user to specify the size of the file
   • putf exists to PUT the file, so this exists in minio.
   • [x] DELETE BUCKET: recursively in a single function

2. Cataloguing and metadata:

   • [x] Propose a simple intake-based approach
   • See: #30

3. Wildcard search:

   • [x] Could you put "*" or other greppy characters in the GET path? (Maybe this should be used via a local (Intake?) catalogue?)
   • There are ways to do this, combining mc find and mc cp, e.g.:

     
     $ mc find namestore/ag1 --regex ".*.txt"
     namestore/ag1/output-1.txt 
     namestore/ag1/random/a.txt
     namestore/ag1/random/b.txt

   ]$ mc find namestore/ag1 --regex ".*.txt" --exec "mc cp {} random/second/" ....uk/ag1/random/b.txt: 5 B / 5 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 29 B/s 0s

4. Need a command-line tool

   • [x] Investigate minio for this - does it meet the stuff above?
   • minio gives us most of what we need (https://docs.min.io/docs/minio-client-complete-guide.html#find)
   • we might add a few jos command-line options as extensions if needed

5. Could you stream it directly from object store into Fortran process?

   • [x] You can cat the contents of an object and pipe it where you like
   • mc cat namestore/ag1/random/a.txt | sed 's/a/b/g'