search

seqeralabs / staticreg

A tool to generate a static website from an OCI registry containing all the images and tags the provided user has access to.
Apache License 2.0
0 stars 0 forks source link

asynchronous registry fetching #16

Closed fntlnz closed 2 months ago

fntlnz commented 2 months ago

This PR introduces a mechanism that wraps the registry implementation to keep a cache of images and images tags in order to avoid asking the registry over and over.

This was a problem when using the server mode in registries with hundreds images and hundreds of tags where effectively to render the home page we need to also ask the registry for tags. That resulted in thousands of requests to the registries that were repeated over and over.

With this strategy, requests are done only after a set refresh interval --refresh-interval while calling serve to tell staticreg how often to refresh its internal cache from the registry. This mechanism sits on top of the mechanism that caches the HTML of the generated web pages allowing the user to prioritize if having fresh data is more important than being fast (and making less requests) and vice versa.

fntlnz commented 2 months ago

Thanks for the review @alberto-miranda - I'll try to document more.

I'm keeping this on hold because there are still a few performance issues I need to tackle.

fntlnz commented 2 months ago

This is ready. Now the internal cache is structured in a way that we don't need to do many calls to tag lists to render the index so it's scalable even on registries with thousands of repos. Since we changed our way of using it from the initial idea I also removed the generate command, it didn't make sense to port all these changes there.