chanzuckerberg / single-cell-data-portal

The data portal supporting the submission, exploration, and management of projects and datasets to cellxgene.
MIT License
63 stars 12 forks source link

bug(rdev): rdev bucket name upload submission conflicts #2554

Closed Bento007 closed 1 year ago

Bento007 commented 2 years ago

Describe the bug

In rdev environments, due to the shape of the IAM policy, we can only upload files to env-rdev-cellxgene-dataset-submissions/*. This could become a problem if we run functional tests on an rdev environment that include the submission lambda. It may also cause other rdev environments to try processing a submission that was not intended for their environment. There is no easy fix at the moment as it would require writing a separate policy for the rdev environment to allow a wild card. The new IAM policy resource would like something like this:

"arn:aws:s3:::env-rdev-cellxgene-dataset-submissions/*/${czi-cellxgene-dev.us.auth0.com/:sub}/*"

This would allow for the rdev environment name to be present. Some additional tweak to the submission lambda to parse out the rdev environment for the key name.

The post_s3_credentials will also need to be updated to parse the submission bucket name to only include the bucket name and add the rdev environment to the upload path.

Expected behavior

Datasets uploaded to the rdev submission bucket should only be process by that rdev environment.

metakuni commented 1 year ago

Closing since it's not an issue yet and as a P2 we likely won't get to this anytime soon. A new issue will be created when/if the need arises.