disinfoRG / ZeroScraper

Web scraper made by 0archive.
https://0archive.tw
MIT License
10 stars 2 forks source link

Manage PID with db #53

Closed pm5 closed 4 years ago

pm5 commented 4 years ago

Since each cronjob is run in separated Docker containers, making sure that there is at most one process running at any time with PID file lock does not work (the lock file is not shared). This PR moves PID lock to a db variable table in order to share the lock state among cronjobs. Also the pid lock is changed to use python context manager, which eliminates some edge cases where the lock is not released.

pm5 commented 4 years ago

This will fix #42.