20210623: Stage 3 TRB approval OK - we are proceeding with Stage 4
20210715:
see #6
1) moving towards dual client for s3 upload (js + cli) - or working around the presigned key/file-piece + throttling issue
(browser based : html/js for small workloads - presigned dynamic creds)
(CLI based : aws cli for large workloads - use assigned temp creds) - cli will be packaged in a runnable app
Work out the impact on the UI experience with a split browser/app
Use Case flows
See #8 needs copy of this flow
Counsel loads portal site
Counsel tests bandwidth for s3 upload via portal
Counsel selects either browser based or cli/client based upload depending on mbps speed and/or volume of data
Counsel optionally zips entire fileset (Note throughput) - see https://github.com/obrienlabs/radar/issues/20
compressing 300GB to test on a macbook with an external HDD via TB (not SSD as I doubt most clients have 2-4TB SSD drives external yet) - initial numbers is 20MB/sec or 160mbps (which is a fragment of the theoretical max of a HDD just under 1Gbps - also bound by the 1G network) - therefore we would need over 2h to compress a 160GB threshold
Counsel initiates an s3 upload transaction via portal or cli/client
Counsel waits for upload to complete/retry/stream - gets submission receipt
this will take up to 10h for 160GB at 30mbps upload speed
System notifies MINU of upload
MINU loads internal/admin portal site
MINU selects transaction for download/deletion-from-s3
MINU tests bandwidth for S3 download via portal
MINU initiates an S3 download transaction via portal or cli/client to be stored TBD in ISED
MINU waits for download to complete/retry/stream - gets download receipt
Counsel selects either browser based or cli based upload depending on mbps speed and/or volume of data
MINU deletes transaction files from S3 via portal
20210623: Stage 3 TRB approval OK - we are proceeding with Stage 4
20210715:
see #6
1) moving towards dual client for s3 upload (js + cli) - or working around the presigned key/file-piece + throttling issue (browser based : html/js for small workloads - presigned dynamic creds) (CLI based : aws cli for large workloads - use assigned temp creds) - cli will be packaged in a runnable app Work out the impact on the UI experience with a split browser/app
Use Case flows
See #8 needs copy of this flow
Counsel loads portal site Counsel tests bandwidth for s3 upload via portal Counsel selects either browser based or cli/client based upload depending on mbps speed and/or volume of data Counsel optionally zips entire fileset (Note throughput) - see https://github.com/obrienlabs/radar/issues/20