db0 / fedi-safety

A script that goes through a lemmy images in storage and tries to prevent illegal or unethical content
GNU Affero General Public License v3.0
63 stars 6 forks source link

Add support for realtime-scanning multiple buckets round-robin #16

Open ruudschilders opened 3 months ago

ruudschilders commented 3 months ago

I have 7 instances of which I want to scan the buckets, and the mastodon ones need 3 separate locations scanned (because it also stores other stuff in subfolders) That would require over 10 processes running, each process requires a bit over 1GB GPU RAM, and I have only 8GB GPU RAM available.

Therefore it would be useful if 1 process could round-robin the realtime scanning for multiple configs.

Currently I have a folder for each, with its own .env file and lemmy_safety.db Maybe it could round-robin over all folders and read the .env's and store in the right db? Or you could have different env and DB in the same folder with their own suffix?