[PROPOSAL] Allow to run crystal circleci tests for contributors

miry commented 7 months ago

Currently it is required core developers, who has access to crystal-lang/crystal to run tests and dist jobs. It is not effective with more contributors.

circleci allows to trigger pipelines with custom paramaters. Example:

export CIRCLECI_API_KEY=<generate from circleci> : Create token in https://app.circleci.com/settings/user/tokens
export CRYSTAL_BRANCH="release/1.10" : against what crystal branch execute distribution-scripts
export FORK_OWNER=miry : PR authors nick
export FORK_BRANCH=<branch> : branch of PR in <user>/distribution-scripts

curl --request POST \
     --url "https://circleci.com/api/v2/project/github/${FORK_OWNER}/crystal/pipeline" \
     --header "Circle-Token: ${CIRCLECI_API_KEY}" \
     --header "content-type: application/json" \
     --data '{"branch":"'$CRYSTAL_BRANCH'","parameters":{"run_distribution_scripts_tests":true,"distribution-scripts-repo":"https://github.com/'$FORK_OWNER'/distribution-scripts.git","distribution-scripts-version":"'$FORK_BRANCH'"}}'

sample circleci workflow

TODO:

[x] https://github.com/crystal-lang/distribution-scripts/pull/273
[ ] https://github.com/crystal-lang/distribution-scripts/pull/288
[ ] https://github.com/crystal-lang/distribution-scripts/pull/267
[ ] Migrate build linux package
[ ] Migrate build docker package

References

straight-shoota commented 7 months ago

I had wondered about this as well. However, so far only members have worked on the building setup so this wasn't an issue. Having an easy path for external contributors would certainly be nice, but it should also be put in perspective. We should not waste a considerable amount of effort to set this up if it's barely used afterwards.

It works - though not as smooth - to have someone with push rights assist external contributors. Or we could give the necessary access rights to trusted contributors so they can work on their own.

Anyway, some notes on the topic:

The circleci runners have access to deployment secrets. We need to ensure they cannot be leaked to external contributors making pull requests to the repository.
Instead of tweaking around the current setup's limitations, we could also consider more substantial changes. A deficiency also for internal contributors is that changes to distribution-scripts require a commit on crystal to test them. This could perhaps be automated, but maybe changing the basic setup would make this easier. For example we could swap the relationship between repos, having CI defined in distribution-scripts and pull in the content from crystal (instead of the other way around). This would make it easier to contribute to distribution-scripts. It would still be triggered from releases on crystal (but also other triggers internal to distribution-scripts).

miry commented 7 months ago

@straight-shoota thank you for the clarification

TLDR: I have built prototype and would require few tweaks in both repos (https://github.com/crystal-lang/distribution-scripts/pull/273) and (https://github.com/crystal-lang/crystal/compare/master...miry:crystal:test-darwin). It is still require a script to trigger job and docs how to start testing.

It looks like this: CircleCI workflow

Security

Because all circle ci runners would be run on behalf of Contributors, it would garantie that secrets from crystal org are not shared or leaked. I see also plus compute resources would be distributed among users.

Migrate tests to distribution-scripts

It would be much better. I think it is anyway done partialy on circleci config for builds like darwin and linux - I found in scripts downloading of full crystal repo. UPDATE: I am still think #273 would be good for this scenario as well.

Let me know if you ok with my prototype or I should check second posobility instead.

straight-shoota commented 7 months ago

Because all circle ci runners would be run on behalf of Contributors, it would garantie that secrets from crystal org are not shared or leaked.

I suppose we would want CI to run on PRs (i.e. in crystal-org), though. Not just on contributor's branches. But I expect we could manage to keep secrets limited to the release environment.

straight-shoota commented 7 months ago

Moving release CI here seems like a good idea. It's probably gonna take some time, though.

But we can simply start by setting up the CI workflows in this repo. Doesn't need to be the full thing and it can grow gradually. Once we know it works, we can swap release builds in.

Another important aspect though is that we've had the idea to migrate the release workflow to GitHub Actions. The main reason for this being aarch64-linux builds on our own runners (tests are already running there). The CircleCI OpenSource plan doesn't seem to allow custom runners, unfortunately.

There has already been some progress on this in #211, but I'm not sure where exactly we're standing with that. That PR is definitely quite huge and contains lots of individual changes. We can probably take a lot of things from there, though.

miry commented 5 months ago

Introduced 2 PRs to migrate some CI functionalities from CircleCI to GithubActions:

crystal-lang / distribution-scripts