pyronear / pyro-platform

Detection & monitoring platform of wildfires
https://platform.pyronear.org/
Apache License 2.0
10 stars 11 forks source link

Adding caching decorator to cache and reuse heavy data #26

Closed VincGargasson closed 3 years ago

VincGargasson commented 3 years ago

GOAL OF THIS PR

Storing in background heaviest data in a cache function and being able to reuse it within 60 sec.

Function used : @cache from flask-caching

How it works? :

  1. Defining the configuration of the cache function at the beginning of the script main.py
  2. Integrating the cache.memoize() decorator between the callback and function we aim
  3. The config used here store the data into temporary file in app/.cache
frgfm commented 3 years ago

@VincGargasson I think you need to add your cache-data-dir to the .gitignore ;)

VincGargasson commented 3 years ago

Thanks for the PR! The .gitignore needs to be updated, as well as the requirements.txt. The rest seems fine to me, could you post some references on how you used flask_caching so that I can take a closer look please?

@frgfm I'll change this right away. The way I used it is as follow:

  1. I set up the cache function at the begining of main.py and used filesystem config so the data are stored into a directory of the project easily (we can use redit database but it's more complex).
  2. I spoted function that collect some potential heavy data and put cache.memoize() function between callback and function
  3. The function keep in memory the data for 60sec so that if the function is called within this timeframe with the same args, it gives back the ouput immediately.

As you can see, it's quite simple and I have discovered it last week on several forums. Please let me know if you still have some questions.

frgfm commented 3 years ago

@VincGargasson Nice! Thanks for the explanation! Although, regarding the PR, you still need to remove the byte content files from the PR (you put the folder in .gitignore, but you need to remove those that were staged :upside_down_face: )

frgfm commented 3 years ago

Let me know if you need help with that @VincGargasson :ok_hand: