search

hackoregon / 2019HackORDataScienceTemplate

Template to get the 2019 data science parts of a Hack Oregon project started :)
MIT License
2 stars 0 forks source link

Add guidelines to list S3 public bucket folder / file structure #4

Open karenng-civicsoftware opened 5 years ago

karenng-civicsoftware commented 5 years ago

There has been a request to find ways to list the directory structure of S3 files.

Right now we are unable to list the content of our public hacko-data-archive S3 bucket with the AWS CLI, most possibly due to permission issues Reference from stack overflow here @DingoEatingFuzz Is it possible for us to change the permission of the files & folders in that public S3 bucket? If permission is not the problem, how can we enable this?

[resolution] Karen will write a platform request to ask for a REST API to check what files are availabe in the S3 bucket

DingoEatingFuzz commented 5 years ago

I responded to this in the sagemaker PR: https://github.com/hackoregon/hackoregon-aws-infrastructure/pull/61#issuecomment-495030101

karenng-civicsoftware commented 5 years ago

@DingoEatingFuzz being able to list the data files will allow better collaboration between the different teams. The desired workflow is to do 

$ aws s3 ls s3://hacko-data-archive

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

I do not think this will be computationally intensive operation. Why wouldn't you want to allow access?

DingoEatingFuzz commented 5 years ago

It is not computationally intensive, and I don't mind if data scientists are given access tokens to use with the aws cli tool, the issue is around security.

Even though the S3 bucket is public, you'd have to know the URL you want in order to download anything. If I make the ListObjects operation available to the public, then anyone could crawl through the bucket and clone the whole thing.

I would like to eventually open this up, but not before doing an audit and making sure there is truly nothing in there that we would want exposed.

karenng-civicsoftware commented 5 years ago

resolution: @karenng-civicsoftware will gather a list of emails of the data scientists needing AWS accounts to list the files on the S3 buckets. Then share it with Michael to create the accounts.