bitsbeats / drone-tree-config

Drone helper for mono repositories.
Apache License 2.0
103 stars 24 forks source link

Add ability to leverage an in memory cache #36

Closed mach6 closed 3 years ago

mach6 commented 3 years ago

When drone-tree-config returns a "multi-machine" pipeline to drone, drone can in-turn hit drone-tree-config again for each stage of the "multi-machine" pipeline..

This change set gives drone-tree-config the ability to respond to such "duplicate" requests via an in memory cache.

As a result, the provider API is not leveraged every time and the likelihood of being rate limited is reduced


Example output of the new test;

=== RUN   TestCache
time="2021-01-19T13:37:30-08:00" level=info msg="2e5559eb-b50c-4144-b177-bdccd6fdbcd1 foosinn/dronetest started"
time="2021-01-19T13:37:30-08:00" level=info msg="2e5559eb-b50c-4144-b177-bdccd6fdbcd1 match: foosinn/dronetest"
time="2021-01-19T13:37:30-08:00" level=debug msg="drone-tree-config environment" after=8ecad91991d5da985a2a8dd97cc19029dc1c2899 before= branch= ref= slug=foosinn/dronetest trigger=@cron
time="2021-01-19T13:37:30-08:00" level=warning msg="2e5559eb-b50c-4144-b177-bdccd6fdbcd1 @cron, rebuilding all"
time="2021-01-19T13:37:30-08:00" level=warning msg="recursively scanning for config files with max depth 2"
time="2021-01-19T13:37:30-08:00" level=debug msg="Repositories.CompareCommits 200: http://127.0.0.1:45879/api/v3/repos/foosinn/dronetest/contents/?ref=8ecad91991d5da985a2a8dd97cc19029dc1c2899"
time="2021-01-19T13:37:30-08:00" level=debug msg="2e5559eb-b50c-4144-b177-bdccd6fdbcd1 checking foosinn/dronetest .drone.yml"
time="2021-01-19T13:37:30-08:00" level=debug msg="Repositories.CompareCommits 200: http://127.0.0.1:45879/api/v3/repos/foosinn/dronetest/contents/.drone.yml?ref=8ecad91991d5da985a2a8dd97cc19029dc1c2899"
time="2021-01-19T13:37:30-08:00" level=info msg="2e5559eb-b50c-4144-b177-bdccd6fdbcd1 found foosinn/dronetest .drone.yml"
time="2021-01-19T13:37:30-08:00" level=debug msg="Repositories.CompareCommits 200: http://127.0.0.1:45879/api/v3/repos/foosinn/dronetest/contents/afolder?ref=8ecad91991d5da985a2a8dd97cc19029dc1c2899"
time="2021-01-19T13:37:30-08:00" level=debug msg="2e5559eb-b50c-4144-b177-bdccd6fdbcd1 checking foosinn/dronetest afolder/.drone.yml"
time="2021-01-19T13:37:30-08:00" level=debug msg="Repositories.CompareCommits 200: http://127.0.0.1:45879/api/v3/repos/foosinn/dronetest/contents/afolder/.drone.yml?ref=8ecad91991d5da985a2a8dd97cc19029dc1c2899"
time="2021-01-19T13:37:30-08:00" level=info msg="2e5559eb-b50c-4144-b177-bdccd6fdbcd1 found foosinn/dronetest afolder/.drone.yml"
time="2021-01-19T13:37:30-08:00" level=debug msg="Repositories.CompareCommits 200: http://127.0.0.1:45879/api/v3/repos/foosinn/dronetest/contents/afolder/abfolder?ref=8ecad91991d5da985a2a8dd97cc19029dc1c2899"
time="2021-01-19T13:37:30-08:00" level=info msg="2e5559eb-b50c-4144-b177-bdccd6fdbcd1 config-cache added entry for {slug:foosinn/dronetest ref: before: after:8ecad91991d5da985a2a8dd97cc19029dc1c2899 event: trigger:@cron author:}"
time="2021-01-19T13:37:30-08:00" level=info msg="2e5559eb-b50c-4144-b177-bdccd6fdbcd1 finished"
time="2021-01-19T13:37:30-08:00" level=info msg="d145c939-80c7-4883-813b-fca40b1c9f7a config-cache found entry for {slug:foosinn/dronetest ref: before: after:8ecad91991d5da985a2a8dd97cc19029dc1c2899 event: trigger:@cron author:}"
time="2021-01-19T13:37:30-08:00" level=info msg="0a662f3e-9177-4e2f-8f02-bb0854a4f9a3 foosinn/dronetest started"
time="2021-01-19T13:37:30-08:00" level=info msg="0a662f3e-9177-4e2f-8f02-bb0854a4f9a3 match: foosinn/dronetest"
time="2021-01-19T13:37:30-08:00" level=debug msg="drone-tree-config environment" after=8ecad91991d5da985a2a8dd97cc19029dc1c2899 before= branch= ref= slug=foosinn/dronetest trigger=@cron
time="2021-01-19T13:37:30-08:00" level=info msg="0a662f3e-9177-4e2f-8f02-bb0854a4f9a3 config-cache found entry for {slug:foosinn/dronetest ref: before: after:8ecad91991d5da985a2a8dd97cc19029dc1c2899 event: trigger:@cron author:}"
time="2021-01-19T13:37:30-08:00" level=info msg="0a662f3e-9177-4e2f-8f02-bb0854a4f9a3 finished"
time="2021-01-19T13:37:30-08:00" level=info msg="d145c939-80c7-4883-813b-fca40b1c9f7a config-cache found entry for {slug:foosinn/dronetest ref: before: after:8ecad91991d5da985a2a8dd97cc19029dc1c2899 event: trigger:@cron author:}"
time="2021-01-19T13:37:30-08:00" level=info msg="config-cache expired entry for {slug:foosinn/dronetest ref: before: after:8ecad91991d5da985a2a8dd97cc19029dc1c2899 event: trigger:@cron author:}"
--- PASS: TestCache (0.00s)
PASS
mach6 commented 3 years ago

@foosinn JFYI - I have been running with these changes for ~6 months -- very large mono repo that does ~250 pipelines per day.. Once combined with the consider-file approach, I haven't seen any rate limit hits ;)

mach6 commented 3 years ago

@foosinn Any feedback on this one? Cheers.

foosinn commented 3 years ago

Hey @mach6, sadly i've got some critical deadlines to hit mid february. After that i'll have some time to take care about the repos. Sorry for the delay, but i at least want to test it before merging an releasing. Thanks a lot for your awesome contributions anyways!

mach6 commented 3 years ago

@foosinn Hope all is well. Just a gentle nudge here -- have you been able to look at this?

foosinn commented 3 years ago

Merged an released. Thanks so much!