search

db0 / fedi-safety

A script that goes through a lemmy images in storage and tries to prevent illegal or unethical content
GNU Affero General Public License v3.0
63 stars 6 forks source link

Multiple local file locations? #8

Closed poVoq closed 3 months ago

poVoq commented 10 months ago

I was thinking of using this to also scan other public user uploads on the same server (XMPP, Wiki etc.), but it seems this only supports a single location?

Would it be hard to add this?

db0 commented 10 months ago

It wouldn't be particularly hard to scan more than one location. Are you running it in the same server, or via ssh?

poVoq commented 10 months ago

All local, same server.

Edit: But multiple locations via the same SSH connection might be nice too.

poVoq commented 10 months ago

Thinking about this a bit more: What would be the overhead of simply starting the process multiple times pointing to different config files for different storage locations?

I am mostly concerned about limited VRAM.

db0 commented 10 months ago

to run the script twice? If you do it parallel, it would double the VRAM used

poVoq commented 10 months ago

Sure, but how much of that is duplicated overhead and how much is simply a result of scanning other images?

I guess keeping the AI model in vram is what takes most and thus that part would be pure duplicated overhead?

And what would be the difference between running the script once with 10 threads, vs. running it twice with 5 threads each?

db0 commented 10 months ago

And what would be the difference between running the script once with 10 threads, vs. running it twice with 5 threads each?

double the VRAM without any benefit in speed

poVoq commented 10 months ago

I see. Well in that case my original feature request is still valid 😊