Create dataset african_minds_publisher

[albertvillanova]

• uid: african_minds_publisher
• type: primary
• description:
  • name: African Minds publisher
  • description: African Minds is an open access, not-for-profit, publisher of scholarly books. African Minds publishes predominantly in the social sciences and its authors are typically African academics or those with a close affinity with the continent. African Minds offers innovative approaches to those frustrated by a lack of support from traditional publishers. At African Minds the emphasis is less on the commercial viability of publications than on fostering access, openness and debate in the pursuit of growing and deepening the African knowledge base.
  • homepage: https://www.africanminds.co.za/
  • validated: True
• languages:
  • language_names:
  • English
  • language_comments: a handful of books in French and Portuguese too
  • language_locations:
  • Africa
  • validated: False
• custodian:
  • name: African Minds
  • in_catalogue:
  • type: A nonprofit/NGO (other)
  • location: South Africa
  • contact_name: general information
  • contact_email: info@africanminds.co.za
  • contact_submitter: False
  • additional: https://en.wikipedia.org/wiki/African_Minds
  • validated: False
• availability:
  • procurement:
  • for_download: Yes - it has a direct download link or links
  • download_url: https://www.africanminds.co.za/books/
  • download_email:
  • licensing:
  • has_licenses: Yes
  • license_text:
  • license_properties:
    • open license
  • license_list:
    • cc-by-4.0: Creative Commons Attribution 4.0 International
  • pii:
  • has_pii: Yes - text author name only
  • generic_pii_likely:
  • generic_pii_list:
  • numeric_pii_likely:
  • numeric_pii_list:
  • sensitive_pii_likely:
  • sensitive_pii_list:
  • no_pii_justification_class: general knowledge not written by or referring to private persons
  • no_pii_justification_text:
  • validated: False
• source_category:
  • category_type: collection
  • category_web:
  • category_media: books/book publisher
  • validated: False
• media:
  • category:
  • text
  • text_format:
  • .PDF
  • audiovisual_format:
  • image_format:
  • database_format:
  • text_is_transcribed: No
  • instance_type: book
  • instance_count: 100<n<1K
  • instance_size: n>10,000
  • validated: False
• fname: african_minds_publisher.json

[albertvillanova]

Commented by @StellaAthena: https://github.com/bigscience-workshop/data_tooling/issues/57#issuecomment-971148752

There only appears to be approximately 120 books on this site. While it’s a potentially excellent resource for many applications, it would be infeasible to build a dataset of hundreds of gigabytes out of components this small.

Based on my experience working on the Pile, I would strongly recommend putting on hold any data source that does not have over 5 GB of text, and only accepting ones with less than 10 GB if they’re special. If the goal is to comprise one quarter of the training data for the multilingual model (a number I have in my head but don’t know where it came from), we need at least 250 GB of text. It’s going to be much less work to find 25 10 GB sources than it will be to find 250 1 GB sources.