oliverchang
commented
3 years ago We do want a fully e2e test for github (and any other CI we support in the future).
Perhaps what we should do instead is:
- Have a stable "cifuzz" tag that we update.
- Only run e2e tests on master/main rather than on individual PRs. We can add automation to automatically update the "cifuzz" tag if everything passes.
Ideally an end-to-end test would use a github workflow to test cifuzz (as I do in github.com/jonathanmetzman/cifuzz-external-example). But I don't think it's possible to dynamically set the branch i.e. I don't think we can create a workflow to test the action in this repo (as I tried here: https://github.com/google/oss-fuzz/pull/6106/files#diff-fcc54602a11ad75897d96c3daa23ecb2f02ea8a36194eda331bad0a6f6c66c73R23). Another option is to have a script that changes the branch in another repo to do the test. This what I've manually been doing in the above repo. A final option is to have a normal python test that invokes some kind of a official script we create for running cifuzz without github actions. I think we need a script like this anyway (for non-github users). The downside of this approach is it won't be fully e2e for github users, if we change something in the action.yml file, the test might not be able to catch it.