This optimizes the database calls within the pizza oven.
Instead of iterating and inserting each author and commit, this uses methods to bulk insert those records:
Each author is added to a transaction. When that transaction is executed and if there is a conflict, the transaction for that record "does nothing". This takes 1 call to the database.
The commit authors are re-queried in order to build a map of inserted authors emails and IDs. These IDs are used when bulk inserting into the commits table.
Each commit is added to a transaction. When the transaction is executed, each commit is inserted.
On some preliminary testing using the freecodecamp/freecodecamp repo (which as about ~30,000 commits), each with a fresh local database (so network latency should be negligible):
Using main:
āÆ time pizza bake -e http://localhost:8080 https://github.com/freecodecamp/freecodecamp --wait
Resp body:
pizza bake -e http://localhost:8080 --wait 0.01s user 0.02s system 0% cpu 1:17.42 total
Using this branch:
āÆ time pizza bake -e http://localhost:8080 https://github.com/freecodecamp/freecodecamp --wait
Resp body:
pizza bake -e http://localhost:8080 --wait 0.01s user 0.02s system 0% cpu 3.322 total
Description
This optimizes the database calls within the pizza oven.
Instead of iterating and inserting each author and commit, this uses methods to bulk insert those records:
On some preliminary testing using the freecodecamp/freecodecamp repo (which as about ~30,000 commits), each with a fresh local database (so network latency should be negligible):
Using
main:Using this branch:
which ends up being about a 24x faster.
What type of PR is this? (check all applicable)
Related Tickets & Documents
N/a - related to doing some scaling and deployments after integrating with some of our internal services.
Mobile & Desktop Screenshots/Recordings
N/a
Added tests?
Added to documentation?
[optional] Are there any post-deployment tasks we need to perform?
[optional] What gif best describes this PR or how it makes you feel?