CI with github actions

[eviljeff]

Given the recent changes at travis-ci we want to investigate if github actions is a viable alternative

[diox]

We'll need to check with Mozilla Github ops (#github-admin:mozilla.org on Matrix), as it's disabled org-wide at the moment. There were security concerns initially, which I think can be addressed with some configuration, but more importantly some cost questions (because of the plan Mozilla is under for Github)

[eviljeff]

re: cost, https://github.com/pricing (and other pages) state "Free for public repositories" - that may be something that conflicts with our (legacy) org settings on github though.

[eviljeff]

progress with https://github.com/eviljeff/olympia/blob/actions-investigation-flat/.github/workflows/test.yml 6 of 9 jobs passing. The following tests are still failing:

reviewers-and-zadmin - test fails

FAILED src/olympia/reviewers/tests/test_utils.py::TestReviewHelperSigning::test_nominated_to_public_recommended FAILED src/olympia/reviewers/tests/test_utils.py::TestReviewHelperSigning::test_nomination_to_public

amo-lib-locales-and-signing - errors

_ TestMonitor.test_signer __ ____ TestSigning.test_callsigning ____ __ TestSigning.test_call_signing_on_file_in_guarded_file_path __

main - test fails

FAILED src/olympia/git/tests/test_utils.py::test_temporary_worktree - Asserti... ~FAILED src/olympia/git/tests/test_utils.py::test_get_mime_type_for_blob[entry19-foo.ts-text-text/javascript]~

the 2 fails and 3 errors appear to be autograph related. The remaining 1 main test fail cause is unclear.

[eviljeff]

progress with https://github.com/eviljeff/olympia/blob/actions-investigation-flat/.github/workflows/test.yml - a single failing test remains:

• FAILED src/olympia/git/tests/test_utils.py::test_temporary_worktree

=================================== FAILURES ===================================
___________________________ test_temporary_worktree ____________________________
[gw1] linux -- Python 3.8.6 /home/runner/work/olympia/olympia/.tox/main/bin/python3
src/olympia/git/tests/test_utils.py:93: in test_temporary_worktree
    assert output.startswith(repo.git_repository.path)
E   AssertionError: assert False
E    +  where False = <built-in method startswith of str object at 0x7fd365f3bd40>('/tmp/pytest-of-runner/pytest-0/popen-gw1/test_temporary_worktree0/storage/git-storage/1/1/1/addon/.git/')
E    +    where <built-in method startswith of str object at 0x7fd365f3bd40> = '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_temporary_worktree0/storage/git-storage/1/1/1/addon  7b746ff [master]\n'.startswith
E    +    and   '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_temporary_worktree0/storage/git-storage/1/1/1/addon/.git/' = pygit2.Repository('/tmp/pytest-of-runner/pytest-0/popen-gw1/test_temporary_worktree0/storage/git-storage/1/1/1/addon/.git/').path
E    +      where pygit2.Repository('/tmp/pytest-of-runner/pytest-0/popen-gw1/test_temporary_worktree0/storage/git-storage/1/1/1/addon/.git/') = <olympia.git.utils.AddonGitRepository object at 0x7fd368cc4d60>.git_repository
        output     = ('/tmp/pytest-of-runner/pytest-0/popen-gw1/test_temporary_worktree0/storage/git-storage/1/1/1/addon  '
 '7b746ff [master]\n')
        repo       = <olympia.git.utils.AddonGitRepository object at 0x7fd368cc4d60>
        settings   = <pytest_django.fixtures.SettingsWrapper object at 0x7fd367355fa0>
------------------------------ Captured log call -------------------------------
INFO     z.git_storage:utils.py:245 Initialized git repository "/tmp/pytest-of-runner/pytest-0/popen-gw1/test_temporary_worktree0/storage/git-storage/1/1/1/addon"

[eviljeff]

https://github.com/eviljeff/olympia/blob/actions-investigation-flat/.github/workflows/test.yml

Apart from the last remaining failing test, there are no other blockers on CI side.

Re: security concerns - it's possible to limit actions to just github provided ones, or by wildcard, so I don't know if that would assuage some of the concerns.

Annoyances/deficiencies vs. travis:

• you can't (currently) restart single jobs, only the entire thing, which could get annoying when we have intermittent failures

Improvements:

• We should add caching to improve performance (there's a github provided cache action);
• It's possible we could gain some performance, or complexity, improvements by having elasticsearch/autograph as services rather than setting them up ourselves.
• We might be able to get details of the failing tests output as annotations to be shown on the run summary page (they're all just Process completed with exit code 1. currently which is a bit pointless)
• I'm not sure we strictly need tox any longer with the ability to add conditionals in the workflow directly. It'd be a non-trivial task to convert the setup but the end configuration would be more flat and probably more easily manageable (though we would end up with a much larger workflow file)
• Notification to Slack support?

[willdurand]

We switched to circle-ci so we probably don't need this issue anymore? at least, not until circle-ci goes crazy?

[eviljeff]

Afaik the position on github actions (which may have shifted while I was off) was that it was the desirable end solution, but we didn't want to jump into it while the security and org-operational aspects were still up in the air. Switching to Circle-ci in the mean-time was forced on us by travis-ci.