Design revised upload for RDM where files are stored directly in S3 bucket

clansing commented 2 years ago

Notes from 10/13

Make sure to take the RDM visibility setting into account in the design
Keep metadata-only record in PCM. We will not set this automatically. If user checks this box, we will not create a bucket folder for the dataset.
We instead will create a fake file in RDM storage for each dataset that is not metadata-only. This is to prevent RDM from throwing validation exceptions if no files are found. This also allows the file visibility button to remain visible.
In the design document make sure to add a chart of the bucket path permissions for each dataset. See Matt's livewire example at https://github.com/livewire-data-platform/meta/wiki/Controlling-Access-to-Project-and-Dataset-Information-in-LDP
Need to make sure that the UI widget used for S3 browsing in the dataset landing page allows downloads.
Set another policy on the dataset path that downloads are not allowed unless the user has the msdlive member role in cognito.

clansing commented 2 years ago

Initial design document attached:

zguillen commented 2 years ago

This has a potential for collaboration with RDM developers or partners, see:

And potentially this:

MSD-LIVE / issues