dipdup-io / dipdup

Modular framework for creating selective indexers and featureful backends for dapps
https://dipdup.io
MIT License
97 stars 52 forks source link

Cache latest release from GitHub for version check in xdg_shared_dir #1095 #1113

Closed MrRoudyk closed 1 month ago

MrRoudyk commented 1 month ago

Optimize Version Check with Caching

Summary: Enhances version checking by adding a caching mechanism, reducing dependency on real-time GitHub API calls.

Changes:

Notes:

Closes #1095

bruhhgnik commented 1 month ago

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

I have experience with Python development and ML models in general.

How I plan on tackling this issue

Start by implementing functions to cache the latest version in a JSON file with a 24-hour TTL. I would modify the existing version check to retrieve the version from cacche first, only querying GitHub if the cache is missing or expired. I would use the appdirs library to manage the storage location for cross-platform compatibility.

Kaminar-i commented 1 month ago

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

hello i would love to work on this issue

How I plan on tackling this issue

please kindly assign

estherbreath commented 1 month ago

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

I am a full-stack developer, and would leverage on my background to;

  1. Implementing front-end optimizations like client-side caching and UI improvements
  2. Optimizing back-end services with server-side caching and API enhancements
  3. Utilizing DevOps practices for infrastructure scaling and monitoring
  4. Ensuring thorough testing and quality assurance across all layers

How I plan on tackling this issue

I will take these few steps by way of summary to work on this issue within the possible limited time.

Add caching functions Store cached version with a 24-hour TTL Optimize Version Checking by modifying the check version to; ◦ Retrieve version from cache ◦ Query GitHub API only if cache is missing or expired Investigate and resolve the problem with aiohttp and fire_and_forget

MrRoudyk commented 1 month ago

@droserasprout Thank you for the feedback! I have made all the changes you suggested. Please take a look at the updated code.

droserasprout commented 1 month ago

Great, thanks!

akintewe commented 1 month ago

I am applying to this issue via OnlyDust platform.

My background and how it can be leveraged

I'm a CS major with a lot of Python experience, and I've worked on a lot of projects that involved caching systems, performance optimisations, and API integrations. My expertise in software engineering best practices and principles allows me to tackle challenging issues like this one with effectiveness and maintainability.

How I plan on tackling this issue

I would first analyse the current version checking system and determine the best places to include the cache before implementing caching for the latest release version check. I would create a caching system with read and write functions and a 24-hour TTL, storing the version information in a platform-appropriate location using the appdirs library. The _check_version method would be changed to only call the GitHub API when absolutely necessary, instead of always attempting to retrieve the version from cache. I would see to it that the process's error handling and logging are done correctly. Testing would include misses, expirations, and cache hits, among other scenarios. Lastly, if the caching implementation is impacted by any possible issues with aiohttp and fire_and_forget, I would solve them.