search

pilosus / pip-license-checker

Check license types for third-party dependencies: permissive, copyleft, proprietory, etc.
https://blog.pilosus.org/posts/2021/09/07/pip-license-checker/
Other
67 stars 3 forks source link

Caching for external API requests #111

Closed pilosus closed 1 year ago

pilosus commented 1 year ago

Rate-limiting is a good thing (#101 and #109). But it still doesn't solve the problem fully when requesting meta for the same packages over and over again. There's no need check license for the same package of the same version on PyPI, as they cannot be overriden anyway, only yanked.

So, caching would be a good solution to the problem.

CLI option like --request-cache MILLISECONDS with some reasonable defaults (e.g. 1 hour) would be even better.

pilosus commented 1 year ago

For a tool running as one-time the benefits of caching are possible only if:

  1. The cache is stored persistently across runs (e.g. in external database)
  2. The tool is transformed into a server
pilosus commented 1 year ago

I think a solution with external DB is not something we would like to build any time soon. Let’s close the issue and reopen when necessary