librariesio / libraries.io

:books: The Open Source Discovery Service
https://libraries.io
GNU Affero General Public License v3.0
1.12k stars 202 forks source link

Add Indexes to dependency and repository_dependency on created_at date #3142

Closed mikeyoung85 closed 1 year ago

mikeyoung85 commented 1 year ago

This PR is to help out the Projects Dependencies Link daily job which has been failing to complete for a while now. The main reason this job fails is that it is impossible to query the dependencies table because of its large size and it is querying an unindexed column. The PR adds an index for created_at and scopes the values down to remove the time portion and index it as a Date so that values are grouped up a bit in the index and updates the jobs to query by Date and instead of DateTime.

This migration is something that should be run manually outside of our normal deployment since it will take some time for the database to add the indexes.