Since the crawler will be broken into a separate app from the web server, and it will be run from cron, it needs to have the ability to detect if an instance of the crawler is already running.
pid file in the user|temp dir would work. Maybe create a script that runs that instantiates the crawler instead of putting it in the crawler directly. crawler/init.py
Since the crawler will be broken into a separate app from the web server, and it will be run from cron, it needs to have the ability to detect if an instance of the crawler is already running.
pid file in the user|temp dir would work. Maybe create a script that runs that instantiates the crawler instead of putting it in the crawler directly.
crawler/init.py