search

openeduhub / metalookup

Provide metadata about domains w.r.t accessibility, licencing, adds, etc.
GNU General Public License v3.0
5 stars 0 forks source link

Background cache warmup #113

Closed MRuecklCC closed 2 years ago

MRuecklCC commented 2 years ago

This PR finishes the transition to away from the MetadataBase class towards the Extractor interface by replacing the remaining occurrences of MetadataBase inheritance with the respective functionality.

Those changes should not affect how the actual extractors work, just how their code is organized. As one can see e.g. from the DirectMatch extractor, the logic is pretty simple once consolidated in a single function and with all the other variants (controlled via enums etc) removed.

This PR also introduces an endpoint for cache warmup. This endpoint takes a list of urls and then fires requests on it's own api from a subprocess. This will gradually populate the cache of the application without rendering it non-responsive.

MRuecklCC commented 2 years ago

@RMeissnerCC another pretty substantial PR. Again, the commits should be somewhat independent, i.e. reviewing individual commits may be reasonable. It would also be possible to split it into multiple PRs.