Integrate the rotation with Google Calendar

[pslawless]

Add the rotation automatically to the Performance Team Calendar (or a perf triage calendar, if that would be better).

[mcomella]

The invite was not successfully added this week: the logs only said

Cached user credentials not found

[mcomella]

The Action triggers on push now so it'll be easier to debug (i.e. any push to main). On the most recent push, we see:

ERROR: unable to locate Google Cloud Project secrets when adding google calendar reminder: [Errno 2] No such file or directory: '.google-cloud-project-secrets.json'

<snip - the generated rotation for this week>

DEBUG: environment variables present?
IN_AUTOMATION... UNAVAILABLE
PERF_TRIAGE_BOT_CACHED_USER_SECRETS... UNAVAILABLE
Cached user credentials not found

[mcomella]

The task to add the calendar reminder failed this week with:

2022-08-22T10:24:58.5299532Z Traceback (most recent call last):
2022-08-22T10:24:58.5308995Z   File "/home/runner/work/perf-triage/perf-triage/rotation.py", line 312, in <module>
2022-08-22T10:24:58.5324398Z     main()
2022-08-22T10:24:58.5327726Z   File "/home/runner/work/perf-triage/perf-triage/rotation.py", line 296, in main
2022-08-22T10:24:58.5329423Z     add_gcal_reminder(args.production, rotations[next_week], generated_next_week)
2022-08-22T10:24:58.5331478Z   File "/home/runner/work/perf-triage/perf-triage/rotation.py", line 196, in add_gcal_reminder
2022-08-22T10:24:58.5333242Z     credentials = gcal.auth_as_user()
2022-08-22T10:24:58.5335023Z   File "/home/runner/work/perf-triage/perf-triage/gcal.py", line 123, in auth_as_user
2022-08-22T10:24:58.5336831Z     creds.refresh(Request())
2022-08-22T10:24:58.5338284Z   File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/google/oauth2/credentials.py", line 302, in refresh
2022-08-22T10:24:58.5339835Z     ) = reauth.refresh_grant(
2022-08-22T10:24:58.5341641Z   File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/google/oauth2/reauth.py", line 347, in refresh_grant
2022-08-22T10:24:58.5343309Z     _client._handle_error_response(response_data)
2022-08-22T10:24:58.5345257Z   File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
2022-08-22T10:24:58.5346942Z     raise exceptions.RefreshError(error_details, response_data)
2022-08-22T10:24:58.5349097Z google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})

Two things:

1. it caused the generate rotation task to fail and strangley interwove the error traceback with our print statements. (from davehunt) Perhaps we should separate generation from the calendar reminder task to prevent failing both
2. Since the credentials were unable to refresh, I'm concerned that we won't be able to use this approach to adding calendar reminders. I'm guessing it couldn't refresh the credentials because they're too old (i.e. and require manually intervention). If we can't use this approach, the reminder will either need to not happen via google calendar or it will need a user to intervene (maybe a local script + cron job? Or a button on the perf-triage website to add a reminder)

[mcomella]

To understand why the error occurred, I found the source where the error message, ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'}), is constructed by the client: invalid_grant refers to the error type and "Bad Request" is the description of the error from the server. The constant that defines invalid_grant is named _REAUTH_NEEDED_ERROR.

It sounds like these credentials need user intervention to be refreshed (i.e. I'll log into the service account, get new credentials, and update GitHub Actions) though I'm not sure what "Bad Request" might refer to. I'm concerned the credentials won't be able to be refreshed indefinitely, especially since we don't update the refresh token after each run. It appears I've been using credentials for my own account rather than the service account so I've been unable to test whether or not the refreshing is working.

I'll make the error handling more robust while I'm there.

[mcomella]

I should watch the calendar reminders for maybe 2 months to ensure they continue to work (i.e. the credentials are able to be refreshed) so I'll leave this issue open to remember to do that.

[mcomella]

The server sent the reminder successfully! 🥳 I'll still keep an eye on it for a while.

[mcomella]

The reminder failed with:

  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_data)
google.auth.exceptions.RefreshError: ('invalid_grant: Token has been expired or revoked.', {'error': 'invalid_grant', 'error_description': 'Token has been expired or revoked.'})

I think the issue is that we are unable to persist the refreshed credentials from week-to-week. The CI environment isn't a good place to persist state, however, so I don't know where else we can run this.

[julienw]

It seems to me that this is working pretty well these days. Do you know if this issue has been resolved @davehunt?

[davehunt]

I took over running this manually from Mike. I think we should keep this open until it's fully automated, or close it if we think that's not feasible. I haven't spent any time looking into it as it only takes a minute per week. It's a risk though if I forget or am on PTO.

[julienw]

I see, then the work would be to automate persisting state to the github repository, like you're doing manually?

[davehunt]

The work would be running the command as a GitHub Action (or similar). I believe the issue Mike faced was related to authentication and credentials.