Good point! I don't have time to implement it today or tomorrow, but I will put it on my todo list for next week.
I'll just put some notes here for myself.
Create a dataset by searching for NSFW and SFW content and store the clean body of the responses. 1000 samples in each class is probably a good start.
Train a bernoulli naive bayes classifier on the text with tf-idf vectors as features. Use rust-phf to create a perfect hashmap from term => termid at compile.
Filtering logic
When turned off, don't do anything.
When strict, filter anything where model prediction probability is over a given threshold (don't know what the threshold should be yet)
When moderate, run model on user query. Filter all webpages where $P[NSFW | webpage] > P[NSFW | query]$. A higher $P[NSFW | query]$ will therefore result in less filtering. Maybe we need to add an offset to the $P[NSFW | query]$.
Good point! I don't have time to implement it today or tomorrow, but I will put it on my todo list for next week. I'll just put some notes here for myself.
Create a dataset by searching for NSFW and SFW content and store the clean body of the responses. 1000 samples in each class is probably a good start.
Train a bernoulli naive bayes classifier on the text with tf-idf vectors as features. Use rust-phf to create a perfect hashmap from term => termid at compile.
Filtering logic