Earth-Information-System / fireatlas

https://earth-information-system.github.io/fireatlas/docs/

7 stars 3 forks source link

Setup OIDC and AssumeRoleWithWebIdentity on EIS SMCE Account for GH Actions Interaction #74

Closed ranchodeluxe closed 2 months ago

ranchodeluxe commented 4 months ago

Problem

We need to run tests but some of our data is too big to store in the /tests/data/ folder and needs to be on s3. We want to use the new EIS bucket s3://smce-eis-feds-output-conus but with SMCE and MFA this means we'll need to set up an OIDC provider and role for GH Actions

Action

@dfelikson can create users for @eorland and @mccabete in the EIS SMCE account (if they don't exist)? These users should have admin privileges if possible so they do some of the IAM and s3 tasks we'd need (maybe temporarily)
@ranchodeluxe can either get an account on the EIS SMCE with limited privs (or maybe good privs) and should help the above folks set up the OIDC provider and role
Make necessary changes to the GH test actions in @zebbecker PR so we can test it against the new role

jsignell commented 4 months ago

my vote is for new-fangled bucket.

zebbecker commented 4 months ago

Ideally I'd like this to be in the v3 release- are we too blocked by permissions stuff? It would be nice to have some automated tests protecting the main branch.

Then again, we can still run the tests manually (pytest --runslow) even while we are sorting out accesses for the test runner, so this doesn't need to block the release if we are just waiting on external permissions stuff to get sorted out.

ranchodeluxe commented 4 months ago

Ideally I'd like this to be in the v3 release- are we too blocked by permissions stuff? It would be nice to have some automated tests protecting the main branch.

Yeah, we're stuck waiting for Tess and Eli to get "approved" into the system and then I have to get access and then we have to figure out how to do all the things. So that's gonna be a few days. If you really want to wait then we can push back the release to mid next week

zebbecker commented 4 months ago

Ok, I will rebase my test branch and run the tests manually. Don't push back on my account!

mccabete commented 3 months ago

@ranchodeluxe I do have the permissions for this now. I will also be almost entirely OOO this week. If you have some instructions for the type of OIDC that will work with github actions, I can try to set up the role async.

ranchodeluxe commented 2 months ago

@zebbecker: I think maybe we should pair on this together b/c those yahoos at SMCE still haven't gotten my AWS account in order and I cannot do anything yet

zebbecker commented 2 months ago

@ranchodeluxe want to pair this afternoon? I'll be free in about 30 mins after our project meeting ends through end of day, and do have my AWS login working

ranchodeluxe commented 2 months ago

@ranchodeluxe want to pair this afternoon? I'll be free in about 30 mins after our project meeting ends through end of day, and do have my AWS login working

Let me set up an hour tomorrow 👍

ranchodeluxe commented 2 months ago

let's clarify a path for test data and we can create a new s3 bucket if we are not happy with the name (i am not 😄):

zebbecker commented 2 months ago

It turns out that we already have a public read only bucket called s3://eis-fire-public. I moved the "large" (its actually not that big) test data there.

The GitHub Actions runner in the pytest workflow now assumes the github-actions-oidc-role, which can be viewed and configured in the EIS SMCE AWS org.

@mccabete thank you for working through the permissions stuff and @ranchodeluxe thank you for your help on this!

Closing the issue as #144 does what we needed.