search

neonwatty / meme_search

Index your memes by their content and text, making them easily retrievable for your meme warfare pleasures. Find funny fast.
https://memesearch.co/
Apache License 2.0
288 stars 9 forks source link

Enhancement Docker - Use Supercronic to automatically run create.py #8

Closed thijsvanloef closed 1 month ago

thijsvanloef commented 2 months ago

Indexing currently requires the user to run the following command manually:

docker exec meme_search python meme_search/utilities/create.py

If supercronic would be installed, you would be able to use a cron expression to run it at a set interval.

Let me know if this is something you want me to include, I can open a PR :)

neonwatty commented 2 months ago

great idea!

a few thoughts:

  1. at present the entire directory is indexed with this command
  2. in terms of design - should we cron or have another process / container monitor the index directory for additions / subtractions?

not sure at the moment on 2, but i need to crank out 1 in any case before proceeding in either direction. i'll be able to do that tonight.

what do you think about 2?

thijsvanloef commented 2 months ago
  1. I noticed that it indexed the entire input directory while testing the cron functionality, that does complicate it somewhat.
  2. I explored 2 options:
    1. cron
    2. inotifywait

Cron will allow you to define a cron shedule and opens the ability to add more cron tasks in the future. inotifywait checks for actual changes to the directory before running the create.py,l saving resources, and is probably the more user friendly option.

During testing I would much prefer the inotifywait for this specific process.

neonwatty commented 2 months ago

All right!

Just pushed these changes (among others) to address 1:

A new "refresh index" button to update the index when images are added or removed from the data/input image directory, affecting only the newly added or removed images. See the gif in the updated README.md.

Pressing this new button just calls the same command

python meme_search/utilities/create.py

but now this command doesn't completely re-create the index, but only

This gives us something in the direction of 2 for now as well - as you can now manually refresh your index!

As for a cron for 2 - inotifywait would be for the docker version right?

More tests have been added to /tests directory as well - haven't pushed them all to actions yet!

neonwatty commented 1 month ago

Closing for now given the manual refresh index button update.