Testing AWS CI flow

[saravan2]

After :

1. PR : https://github.com/facebookexperimental/resctl-demo/pull/261 goes through
2. Adding secrets to https://github.com/facebookexperimental/resctl-demo/
3. New PR in https://github.com/facebookexperimental/resctl-demo/ to add IAM

We need to start testing AWS CI workflow

[santoshmahto7]

@saravan2 @davide125 : I don't have admin permission to add secrets to github CI. I would need that to test, so add me if it is worth for me to have admin power.

[davide125]

I've just added the secrets on our end.

[santoshmahto7]

I've just added the secrets on our end.

Perfect. Thanks

[davide125]

Merged and kicked off a run: https://github.com/facebookexperimental/resctl-demo/actions/runs/5257716435

[santoshmahto7]

@davide125 Deployment seems successful but we also need to test that lamda as per task. I feel this need to be done on your side. Let me know if I need to be part of this.

[saravan2]

@davide125

Step 3 : New PR in https://github.com/facebookexperimental/resctl-demo/ to add IAM

Is that necessary ?

[davide125]

The IAM stuff should be all set up now. I don't actually know how to test the lambda though...

[santoshmahto7]

@davide125 @saravan2 Once PR https://github.com/facebookexperimental/resctl-demo/pull/292 is merged, The lambda workflow can be tested in machine as ( tetsing is done in my local aws setup) :

Lambda workflow: Client triggers lambda -> lambda runs --> save result to s3 bucket --> Create github Issue with link of result stored in s3.

Testing steps:

1. In AWS lambda confirm lambda has role and policy setup as described in README

2. In AWS lambda, Create Lambda Function Url - this function URL will be used to trigger lambda as in next step

3. Configure the AWS System Manager-> parameter store to have Github credentials

   • Name: /iocost-bot/token
   • Value: github personal token

4. Trigger lambda as : ./resctl-bench --result < json or gzip benchmark result > upload --upload-url <Lambda Function URL> e.g
   ./bootstrap --result resctl-bench-result_2023_05_01-19_18_49_1.json.gz upload --upload-url https://ygvr6jnjckwamfao5xztg6idiu0ukjeb.lambda-url.eu-north-1.on.aws/

5. Check the s3 bucket if result has been uploaded there (Note: aws lambda should have write permission to s3 bucket)

6. Check if issue is created in iocost-benchmark projects like this : https://github.com/iocost-benchmark/iocost-benchmarks/issues/71

Note: The aws endpoints are hardcoded in lambda code so if aws endpoints differs then we need to update code. ping me for that.

[ewyler]

@davide125 has offered to test this this week

[ewyler]

Need to talk to Davide again about this

[davide125]

Configure the AWS System Manager-> parameter store to have Github credentials

Name: /iocost-bot/token Value: github personal token

We can't use personal tokens for this. Can we update the lambda to use OIDC instead?

[santoshmahto7]

@davide125 : I am looking on that how OIDC should be configured in AWS and hopefully will get it done soon.

[ewyler]

Continuing work on this to figure out OIDC

[santoshmahto7]

@davide125 : After reading AWS OIDC docs, I'm bit confused with how OIDC authentication should be used in our case.
IIUC AWS OIDC is used to authorise client application(e.g github) for accessing aws resources, but what we need here is other way around. i.e Authorise aws lambda to access client application resources(i.e github). I'm not sure if it is possible, I can try if it is possible. Any docs or instructions for this will be more helpful here. Thanks in advance.

[ewyler]

The problem is on the Github side and avoiding using a personal token. I don't see OIDC listed in the Github docs, just for Github Apps - I'm not sure what @davide125 was referring to. It seems the App token is what we'd want - https://docs.github.com/en/rest/overview/authenticating-to-the-rest-api?apiVersion=2022-11-28

[davide125]

Yeah app tokens should work for this. Thanks!

[ewyler]

You should be able to setup a GitHub App that uses oauth from looking at the docs. That should be workable for us.

On Wed, Nov 15, 2023, 2:21 PM Davide Cavalca @.***> wrote:

Yeah app tokens should work for this. Thanks!

— Reply to this email directly, view it on GitHub https://github.com/iocost-benchmark/iocost-benchmarks-ci/issues/16#issuecomment-1813120775, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQXC3JHLVTRTPMZM5MG46LYEUI2LAVCNFSM6AAAAAAYYX2BW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJTGEZDANZXGU . You are receiving this because you commented.Message ID: @.***>

[santoshmahto7]

@davide125 @htejun A new github app "iocost-issue-creator" (https://github.com/apps/iocost-issue-creater) is created and installed in iocost-benchmark organisation. l amda code is updated to use this app here : https://github.com/facebookexperimental/resctl-demo/pull/302
Lambda uses this app to file issue in iocost-benchmarks product with the submitted benchmarks.

Once merged we can test the lambda flow as mentioned in https://github.com/iocost-benchmark/iocost-benchmarks-ci/issues/16#issuecomment-1662422233 but Instead of storing person token in AWS system manager we can store following github app (iocost-issue-creator) information as :

Name: /iocost-bot/appid Value: xxxxx // Get this value from iocost-issue-creator github app page & Name : /iocost-bot/privatekey Value : xxxxxxx // // Get this value from iocost-issue-creator github app page

[davide125]

@santoshmahto7 can you add me/this org to the github app please? I can't find it, and I think I need access to the app page to pull the appid/privatekey to store on the AWS side. Thanks!

[davide125]

This is failing with an exception on the lambda side when uploading: log-events-viewer-result.csv

[santoshmahto7]

This is failing with an exception on the lambda side when uploading: log-events-viewer-result.csv

I will take a look if this is any configuration error. @davide125 Could you share the command you used to trigger lambda from benchmark result ?

[davide125]

./target/debug/resctl-bench --result /tmp/resctl-bench-result_2024_03_15-04_01_20.json.gz upload --upload-url https://yxvf2x7zfkqga6sknkxgczwonq0yvkrm.lambda-url.us-east-1.on.aws, with the binary built from a fresh checkout off the main branch.

[santoshmahto7]

./target/debug/resctl-bench --result /tmp/resctl-bench-result_2024_03_15-04_01_20.json.gz upload --upload-url https://yxvf2x7zfkqga6sknkxgczwonq0yvkrm.lambda-url.us-east-1.on.aws, with the binary built from a fresh checkout off the main branch.

I fixed one issue with github app authentication since it was moved to organisation https://github.com/facebookexperimental/resctl-demo/pull/316

You are running the command correctly but you can also verify following point as:

• s3 bucket name is iocost-submit
• s3 bucket reason is us-east-1
• AWS all payment is done.
• Latest lambda code is uploaded in aws lambda

[davide125]

Bucket name is wrong, put up https://github.com/facebookexperimental/resctl-demo/pull/317 to fix it

[davide125]

Ok upload works with that applied but the URL in the issue is wrong: https://github.com/iocost-benchmark/iocost-benchmarks/issues/88

[davide125]

Fix for that should be https://github.com/facebookexperimental/resctl-demo/pull/318

[davide125]

Also merged https://github.com/iocost-benchmark/iocost-benchmarks-ci/pull/20 as that code was still referencing invalid buckets

[davide125]

The action that was supposed to rebuild iocost-benchmarks-ci got disabled due to lack of activity, manually enabled it and re-ran it to get a new build with #20 in

[obbardc]

@davide125 Do you have any updates about this?

[davide125]

Looks like the merge CI is broken because the resctl-demo artifacts expired: https://github.com/iocost-benchmark/iocost-benchmarks/actions/runs/10067289408/job/27830450070

[obbardc]

I've reenabled the scheduled jobs (and assed a job to make sure they don't expire) and rebuilt the demo artifacts at https://github.com/iocost-benchmark/iocost-benchmarks-ci/actions/runs/10071274177

[obbardc]

The merge job is getting further, but still failing due to installation of dependency problems: https://github.com/iocost-benchmark/iocost-benchmarks/actions/runs/10071298822/job/27841230084

I think it makes sense to have a separate scheduled CI test to test the merge CI. I'll open a separate ticket about that.

[obbardc]

I fixed the workflow in https://github.com/iocost-benchmark/iocost-benchmarks/pull/90 there are still issues with the workflow (captured in separate tickets), but I think this unblocks the bucket testing for this ticket? Can you please look again?

[obbardc]

OK the lambda function is currently https://yxvf2x7zfkqga6sknkxgczwonq0yvkrm.lambda-url.us-east-1.on.aws.

But submitting a result ends up with an Internal server error response from the Lambda function.

The (limited) log snippet from AWS include:

thread 'main' panicked at resctl-bench/src/lambda.rs:51:81: 
called `Option::unwrap()` on a `None` value 

The Lambda function should be updated to not ... just panic ... but instead throw the error back to the user :-)