Multiple instances of auto-archiver and Proxmox / Azure

bellingcat / auto-archiver

Automatically archive links to videos, images, and social media content from Google Sheets (and more).

https://pypi.org/project/auto-archiver/

MIT License

578 stars 60 forks source link

Multiple instances of auto-archiver and Proxmox / Azure #37

Closed djhmateer closed 2 years ago

djhmateer commented 2 years ago

I'm hosting 3 instances of the auto-archiver on 3 separate VM's. I've allocated 4GB of RAM to each and the systems work well.

Is anyone running multiple instances on a single VM, and found any issues with simultaneous calls to ffmpeg / Gecko drivers. That is what concerns me the most.

Running python in a virtual env eg pipenv run python auto_archive.py should segregate that side I guess.

djhmateer commented 2 years ago

Reading what the project has done with auto_auto_archive.py I've taken a similar approach by calling the archiver sequentially ie

pipenv run python auto_archive.py --config config-1.yaml
pipenv run python auto_archive.py --config config-2.yaml

More detail code here which I may PR into here, or perhaps a README.

https://github.com/djhmateer/auto-archiver/blob/main/infra/cron.sh