NASA-PDS / s3-browser-cloudfront

Web view for files in S3 buckets
Other
1 stars 0 forks source link

Update S3 Browser to support configurable top-level directory structures #31

Open jordanpadams opened 6 months ago

jordanpadams commented 6 months ago

💡 Description

The S3 browser should be pointed at 2 buckets:

data/
  open-data-registry/  ----> this will probably need to be hard-coded
     small-bodies/  -----> this will probably need to be hard-coded
         gbo.ast.catalina.survey/ ---> https://pds-css-archive.s3.us-west-2.amazonaws.com

  small-bodies/ ---> pds-sbn-archive-prod bucket behind whatever that cloudfront https://cloudfront.net/s3/pds-sbn-archive-prod

  # This will eventually point to all the node buckets behind cloudfront, e.g.
  atmospheres/ ---> pds-atm-archive-prod - https://cloudfront.net/s3/pds-atm-archive-prod
  imaging/ ---> pds-img-archive-prod - https://cloudfront.net/s3/pds-img-archive-prod
  geosciences/ ---> pds-geo-archive-prod - https://cloudfront.net/s3/pds-geo-archive-prod 
  naif/ ---> pds-naif-archive-prod - https://cloudfront.net/s3/pds-naif-archive-prod
  etc

⚔️ Parent Epic / Related Tickets

No response

jordanpadams commented 6 months ago

@eddiesarevalo I would like to make sure we are ready for this to work in production now, versus having to come back to it over and over. Let me know how this goes, and we can go from there.

tloubrieu-jpl commented 6 months ago

@jordanpadams will update that ticket.

eddiesarevalo commented 6 months ago

@jordanpadams If we add some code to the s3 browser that can differentiate between a directory and a bucket we can have a map of this in JSON and generate this.

So the directories would be like this:

/odr
  /sbn
    /gbo.ast.catalina.survey/ ---> https://pds-css-archive.s3.us-west-2.amazonaws.com
  /atm
  /img
  /geo
  /naif
jordanpadams commented 6 months ago

@eddiesarevalo copy. whatever you think is best.

As a note on the directories, the odr directory is for Open Data Registry data, right now, it will only contain sbn data. The rest of the node directories I would like at the top level. For right now, it would look like this:

odr/
|--sbn/
|----gbo.ast.catalina.survey/ ---> https://pds-css-archive.s3.us-west-2.amazonaws.com
sbn/  ---> https://cloudfront.net/sbn-archive-bucket

# eventually there will be more nodes at this top level of the directory structure
viviant100 commented 5 months ago

created DSIO-5829 - test bucket in progress

viviant100 commented 5 months ago

@eddiesarevalo the archive bucket is created in test env. Please let me know if that works for you. Thanks.

jordanpadams commented 5 months ago

Status: Issue trying to create static website host on s3-browser-app-dev bucket. @jordanpadams to ping SAs and MCP folks to get this enabled

eddiesarevalo commented 4 months ago

SAs set up some sort of hosting using oac. As far as I can tell it runs. https://dwfmsppt3a750.cloudfront.net/catalina-sky-survey/sbn/gbo.ast.catalina.survey/data_calibrated/703/2020/20Apr02/

This cloudfront points to our dev s3 bucket without using static website hosting.

jordanpadams commented 4 months ago

Blocked by #68

tloubrieu-jpl commented 3 months ago

Unblocked by #68, but blocked by cloudfront distribution deployment in production.