Compute more stats on laion5b

[robvanvolt]

• Distribution of word length: Number of words (0, 1, 2 ..., 50?) and corresponding % of captions, if not possible, then number of characters (0, 1, ..., 500?) and corresponding % of captions for that number of characters
• Top entities in dataset (oxford dictionary words: https://raw.githubusercontent.com/dwyl/english-words/master/words_alpha.txt) (*)
• Exact dataset size (I think it is already in the doc or overleaf)
• Metrics for Language (top 5 languages, n and %)
• NSFW metrics (n and %)
• Watermark metrics (n and %)
• Word count statistics: Compare linguistic diversity by computing the number of unique unigrams (words), bigrams, and trigrams occurring at least 10 times
• Linguistic statistics
  • part-of-speech (POS) tagging with "en_core_web_trf" model from SpaCy to tag captions (result should be n (%) of Common nouns, propper nouns, adjectives and verbs)
  • Toplist (top 10 for example) of common nouns, proper nouns, adjectives and verbs (*)

(*) are related, if the POS is not possible, then just the oxford word top list

Optional, but not necessary as we provide our own NSFW tagging:

• Filter and check how many image-text pairs remain by https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words

[robvanvolt]

1. For all authors...

   • [x] Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
   • [x] Did you describe the limitations of your work?
   • [x] Did you discuss any potential negative societal impacts of your work?
   • [x] Have you read the ethics review guidelines and ensured that your paper conforms to them? Ethics Guideline

2. If you are including theoretical results...

   • [ ] Did you state the full set of assumptions of all theoretical results?
   • [ ] Did you include complete proofs of all theoretical results?

3. If you ran experiments (e.g. for benchmarks)...

   • [ ] Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
   • [ ] Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
   • [ ] Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
   • [ ] Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

   • [ ] If your work uses existing assets, did you cite the creators?
   • [ ] Did you mention the license of the assets?
   • [ ] Did you include any new assets either in the supplemental material or as a URL?
   • [ ] Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
   • [x] Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

5. If you used crowdsourcing or conducted research with human subjects... (not applicable)

   • [x] Did you include the full text of instructions given to participants and screenshots, if applicable?
   • [x] Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
   • [x] Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

[Zasder3]

• URL Domain frequency (% of total and top-10)

[rom1504]

URL Domain frequency (% of total and top-10)

already did that, it's a very long tail, how can we extract information from that?

[Zasder3]

If that's the result that's really all we have to say, worth just stating what % some top-k is for future users.

[rom1504]

from cade "a scan for racial slurs and sexual keywords"

[rom1504]

ok summary

what we already have:

• Exact dataset size (I think it is already in the doc or overleaf) : in blogpost https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/
• Metrics for Language (top 5 languages, n and %) : in blogpost https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/
• NSFW metrics (n and %) : in blogpost https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/
• Watermark metrics (n and %) : in blogpost https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/
• URL Domain frequency (% of total and top-10) -> I have it however it's low information http://3080.rom1504.fr/cah/laion5B_domain_stats/

what is left:

• Word count statistics: Compare linguistic diversity by computing the number of unique unigrams (words), bigrams, and trigrams occurring at least 10 times
• part-of-speech (POS) tagging with "en_core_web_trf" model from SpaCy to tag captions (result should be n (%) of Common nouns, propper nouns, adjectives and verbs)
• Toplist (top 10 for example) of common nouns, proper nouns, adjectives and verbs are related, if the POS is not possible, then just the oxford word top list
• a scan for racial slurs and sexual keywords : Filter and check how many image-text pairs remain by https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words

my opinion:

• nsfw words : can do it but I have low hope it will work compared to a classifier
• top words : can do and then we can intersect with any word list, however I don't believe it'll be super useful
• POS : too costly, not sure what's the goal
• unigram/bigram/trigram : why not

what I can compute on top:

• distribution by imagenet classes
• distribution by other taxonomies