Create dataset pmc_article

[albertvillanova]

• uid: pmc_article
• type: processed
• description:
  • name: PMC Article
  • description: PubMed Central (PMC) is a free archive of biomedical and life sciences journal literature at the U.S. National Institutes of Health's National Library of Medicine (NIH/NLM). It is a repository for journal literature deposited by participating publishers, as well as for author manuscripts that have been submitted in compliance with the NIH Public Access Policy and similar policies of other research funding agencies.
  • homepage: https://www.ncbi.nlm.nih.gov/pmc/
  • validated: True
• languages:
  • language_names:
  • English
  • language_comments:
  • language_locations:
  • World-Wide
  • validated: False
• custodian:
  • name: NCBI
  • in_catalogue:
  • type: A government organization
  • location: United States of America
  • contact_name: PubMed Central (PMC) Customer Service
  • contact_email: pubmedcentral@ncbi.nlm.nih.gov
  • contact_submitter: False
  • additional: https://www.ncbi.nlm.nih.gov/home/about/contact/
  • validated: False
• availability:
  • procurement:
  • for_download: Yes - it has a direct download link or links
  • download_url: https://www.ncbi.nlm.nih.gov/pmc/tools/textmining/
  • download_email:
  • licensing:
  • has_licenses: Yes
  • license_text: Articles in PMC usually include an explicit copyright statement. To see the details for any given record, see the “Copyright and License information” link on an article record. In the absence of a copyright statement, users should assume that standard copyright protection applies, unless the article contains an explicit statement to the contrary. In case of doubt, contact the journal publisher to verify the copyright status of an article.
  • license_properties:
    • multiple licenses
  • license_list:
    • other: Other license
  • pii:
  • has_pii: Yes - text author name only
  • generic_pii_likely:
  • generic_pii_list:
  • numeric_pii_likely:
  • numeric_pii_list:
  • sensitive_pii_likely:
  • sensitive_pii_list:
  • no_pii_justification_class: general knowledge not written by or referring to private persons
  • no_pii_justification_text:
  • validated: False
• processed_from_primary:
  • from_primary: Taken from primary source
  • primary_availability: Yes - they are fully available
  • primary_license: Yes - the dataset has the same license as the source material
  • primary_types:
  • scientific articles/journal
  • validated: False
  • from_primary_entries:
• media:
  • category:
  • text
  • text_format:
  • other
  • .JSON
  • audiovisual_format:
  • image_format:
  • database_format:
  • .JSON
  • text_is_transcribed: No
  • instance_type: article
  • instance_count: 1M<n<1B
  • instance_size: n>10,000
  • validated: False
• fname: pmc_article.json

[albertvillanova]

The PMC Article datasets are:

• [x] PMC Open Access Subset
  • DONE:
  • tmp: community dataset: https://huggingface.co/datasets/albertvillanova/pmc_open_access
  • definitive: community dataset in pmc org: https://huggingface.co/datasets/pmc/open_access
• [ ] Author Manuscript Dataset
• [ ] Historical OCR Dataset
• [ ] LitArch Open Access Subset

[albertvillanova]

Maybe we should create a Hub organization for all these datasets: pmc or pubmed_central.

[albertvillanova]

• Org created: https://huggingface.co/pmc
• Repo created: https://huggingface.co/datasets/pmc/open_access

[yjernite]

what's the difference between this and https://github.com/bigscience-workshop/data_tooling/issues/74 ?

[albertvillanova]

@yjernite:

• This one gets its data from the NCBI website directly:
  • https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_bulk/oa_comm/txt/
  • https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_bulk/oa_noncomm/txt/
  • https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_bulk/oa_other/txt/

• 74 gets its data from the dump that the people from The Pile did:

  • https://the-eye.eu/public/AI/pile_preliminary_components/PMC_extracts.tar.gz
  • Please, note that this server is currently down because of the recent natural disaster in the central US
  • The Pile has created a mirror server: https://mystic.the-eye.eu/public/AI/pile_preliminary_components/PMC_extracts.tar.gz
  • But this needs being updated in our canonical dataset

[lvwerra]

This is done, posted in #74: here.

Done: https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_en_pmc

Sample:

{
'text':  "==== Front\nPLoS BiolPLoS BiolpbioplosbiolPLoS Biology1544-91731545-7885Public Library of Science San Francisco, USA 10.1371/journal.pbio.0000005Research ArticleGenetics/Genomics/Gene TherapyInfectious DiseasesMicrobiologyPlasmodiumThe Transcriptome of the Intraerythrocytic Developmental Cycle of Plasmodium falciparum\n P. falciparum IDC TranscriptomeBozdech Zbynek ..."
'meta': "{'pmid': 12929205}"
}

[albertvillanova]

Thanks @lvwerra.