Addressing issues regarding this system

[DonaldTsang]

It would be easy to break such a system and cause mis-tagging by using these libraries as a demonstration on the weakness of using NN for automated image tagging.

• https://github.com/tensorflow/cleverhans
• https://github.com/bethgelab/foolbox
• https://github.com/advboxes/AdvBox

Here are some of my proposals to make it more resilient:

1. image augmentation to prevent overfitting
   • https://github.com/aleju/imgaug
   • https://github.com/albumentations-team/albumentations
   • https://github.com/mdbloice/Augmentor
2. usage of multiple models for the same task https://arxiv.org/abs/1809.00065 and maybe add Inception or others to the system
   • https://github.com/BertMoons/Comparing-CNN-Architectures
   • https://www.semanticscholar.org/paper/6045cccf284bf2d0ca407b4146ada29c1df038ad
   • https://github.com/CeLuigi/models-comparison.pytorch
   • https://www.groundai.com/project/learning-transferable-architectures-for-scalable-image-recognition
3. de-noising the image using something like
   • https://github.com/yu4u/noise2noise
   • https://github.com/GuoShi28/CBDNet
   • https://github.com/wbhu/DnCNN-tensorflow

AND THEN THERE IS THIS (claiming that most mitigation strategy fails) https://github.com/anishathalye/obfuscated-gradients

I have talked about something similar in https://github.com/halcy/DeepDanbooruActivationMaps/issues/3

[DonaldTsang]

Some useful information regarding the semantic segmentation of images https://github.com/mrgloom/awesome-semantic-segmentation Weird problems that will arise from using the repos within verbatim:

1. How do we deal with tag synonyms and tag subsets? Do we create a system of which segmented regions can have multiple tags?
2. What about character tags vs facial/clothing component tags? How do we correlate them together into a logical manner? hierarchies?
3. What about segmented regions that are too small? Would it get picked up by DD 1.0 but not DD+SS system?
4. How many layers do we need maximum? 32? (since that is the maximum amount of tags per image in general?) 64/128/256?

[DonaldTsang]

Some ideas in how to implement a Semantic Segmentation dataset/model "ShoujoSegment"

1. The initial dataset phase
   • Gather a list of images with strong heatmap confidence
   • Use Recaptha's 3x3 5F-3T-1U test to refine the borders (remember to augment and noise them)
   • Collect results from volunteers and address weighting and credibility issues
2. The Semantic Segmentation training phase
   • Create the system model (or better yet multiple models)
   • Use the collected data to train the system
   • Optimize the system speed and accuracy wise regarding ensembles
3. The data refinement phase
   • Increase the scope of images used
   • Use Recaptha's 3x3 5F-3T-1U test to refine the borders (remember to augment and noise them)
   • Use volunteer's results to refine the Semantic Segmentation
4. Others things that can be done outside of this loop
   • Create micro-models (that is a simplified version of the main model) for mobile systems
   • Apply this system into a new social media network for community contributions
   • Use the "ShoujoSegment" system to refine DeepDanbooru and vice versa

This concept would be applied as the "Humans in the Loop"or "Active Learning" system. A good example would be:

• https://github.com/marc-gorriz/CEAL-Medical-Image-Segmentation and https://arxiv.org/abs/1711.09168
• https://github.com/sinhasam/vaal and https://arxiv.org/abs/1904.00370
• https://github.com/janelia-flyem/gala
• https://github.com/nihalsid/ViewAL and https://arxiv.org/abs/1911.11789
• https://github.com/SarderLab/H-AI-L and https://arxiv.org/abs/1812.07509
• https://github.com/neuropoly/deep-active-learning
• https://github.com/Sheshansh/SuggestiveAnnotation and https://arxiv.org/abs/1706.04737

If there are crowdsourced Semantic Segmentation this can help http://ilpubs.stanford.edu:8090/1161/1/main.pdf and http://ceur-ws.org/Vol-2173/paper10.pdf

[DonaldTsang]

I am just going to put this here, for those who wants to go from label to table.

• https://people.csail.mit.edu/mrub/ObjectDiscovery/
• https://www.researchgate.net/publication/325052999_Object_Discovery_and_Cosegmentation_Based_on_Dense_Correspondences
• https://www.researchgate.net/publication/320350075_Unsupervised_Image_Co-segmentation_via_Guidance_of_Simple_Images