Remove publisher information for styles in (most) articles

adam3smith commented 2 years ago

Continuing discussion from here: https://github.com/citation-style-language/styles/pull/5868#issuecomment-1024478190

Should likely be combined with https://github.com/citation-style-language/styles/issues/2495

Any ideas how to organize this welcome. Initial ideas:

Start by creating data that is "problematic" in an easily identifiable way (e.g. journal articles with PUBLISHER as publisher and running this through a citeproc to list all affected styles
Identify the (very few imo) styles where we don't want to remove the behavior. I think this might be limited to ABNT and GOST for article publishers, e.g.
See if we can find a couple of common macros that we can bulk change using sed or the like and run those scripts
Put the rest in a google sheet, divide & conquer

denismaier commented 2 years ago

Ok, I've prepared a script that does this (via Pandoc):

import os, sys, subprocess

# get testfile
testfile = sys.argv[1]

styles = []

# get list of styles
for x in os.listdir():
    #if len(styles) == 100: break # for testing purposes only test 100 styles
    if x.endswith(".csl"): styles.append(x)

# containers for ok and problematic styles
ok_styles = []
problematic_styles = []

#check each style
for style in styles:
    basename = style.rsplit('.', 1)[0]
    print(f'checking {basename}', end=' --> ')
    # run pandoc
    pandocCommand = ["pandoc", testfile, "-t", "plain", "--citeproc", "--csl", f"{style}"]
    s = subprocess.check_output(pandocCommand, stdin=None, stderr=None, shell=False, universal_newlines=False)
    s = s.decode('utf-8')
    # check if "Publisher" is in output
    if "Publisher" in s:
        result = "Problematic"
        problematic_styles.append(style)
    else: 
        result = "Ok"
        ok_styles.append(style)
    print(result)

# output results
num_of_ok_styles = len(ok_styles)
num_of_problematic_styles = len(problematic_styles)

print(f'{num_of_problematic_styles} problematic styles found')
print(f'{num_of_ok_styles} ok styles found')

# write to files
ok_styles_file = "ok_styles.txt"
problematic_styles_file = "problematic_styles.txt"

with open(ok_styles_file, 'w', encoding='utf-8') as f:
    f.writelines('\n'.join(ok_styles))
    print(f'Ok styles saved to {ok_styles_file}')

with open(problematic_styles_file, 'w', encoding='utf-8') as f:
    f.writelines('\n'.join(problematic_styles))
    print(f'Problematic styles saved to {problematic_styles_file}')

This can be run on a testfile, e.g. "publisher-test.md":

---
references:
- type: article-journal
  id: article-journal
  author:
  - family: Doe
    given: John
  issued:
    date-parts:
    - - 2022
  publisher: 'Publisher'
  title: 'Article title'
  container-title: 'Journal title'
  volume: 1
  issue: 1
  page: 1-20
  DOI: 10.12345/12345
  language: en
...

Test [@article-journal]

Running python styles-test.py publisher-test.md shows that 380 are affected and 2093 are ok.

Here's the list of affected styles:

acta-orthopaedica.csl advances-in-alzheimers-disease.csl african-zoology.csl aging.csl agora.csl al-jamiah-journal-of-islamic-studies.csl alcohol-and-alcoholism.csl alkoholizmus-a-drogove-zavislosti.csl alternatif-politika.csl ambio.csl american-anthropological-association.csl american-association-for-cancer-research.csl american-journal-of-plant-sciences.csl american-nuclear-society.csl american-society-for-pharmacology-and-experimental-therapeutics.csl american-society-of-civil-engineers.csl american-statistical-association.csl annalen-des-naturhistorischen-museums-in-wien.csl annals-of-work-exposures-and-health.csl antipode.csl antiquity.csl apa-5th-edition.csl applied-spectroscopy.csl aquatic-invasions.csl arbok-hins-islenzka-fornleifafelags.csl archeologicke-rozhledy.csl archeologies-et-sciences-de-lantiquite.csl arts-university-bournemouth.csl asian-studies-review.csl associacao-brasileira-de-normas-tecnicas-ufpr.csl australian-veterinary-journal.csl avian-pathology.csl babes-bolyai-university-faculty-of-orthodox-theology.csl baghdad-science-journal.csl bibliotheque-universitaire-de-medecine-vancouver.csl bibtex.csl biochemical-journal.csl biological-reviews.csl bioresources.csl body-and-society.csl british-journal-of-anaesthesia.csl british-journal-of-surgery.csl bursa-uludag-universitesi-sosyal-bilimler-enstitusu-metinici-atif.csl campus-adventiste-du-saleve-faculte-adventiste-de-theologie.csl canadian-geotechnical-journal.csl canadian-journal-of-dietetic-practice-and-research.csl canadian-journal-of-earth-sciences.csl canadian-journal-of-economics.csl canadian-journal-of-fisheries-and-aquatic-sciences.csl canadian-journal-of-soil-science.csl cellular-and-molecular-bioengineering.csl centaurus.csl chicago-author-date-16th-edition.csl chinese-gb7714-1987-numeric.csl chinese-gb7714-2005-author-date.csl clay-minerals.csl clays-and-clay-minerals.csl clinical-nuclear-medicine.csl clinical-physiology-and-functional-imaging.csl college-montmorency.csl comision-economica-para-america-latina-y-el-caribe.csl comparative-population-studies.csl conservation-biology.csl cranfield-university-numeric.csl cultural-geographies.csl current-gene-therapy.csl deutsche-gesellschaft-fur-psychologie.csl development-policy-review.csl diatom-research.csl din-1505-2-alphanumeric.csl din-1505-2-numeric-alphabetical.csl din-1505-2-numeric.csl din-1505-2.csl drugs-of-today.csl earthquake-spectra.csl economic-commission-for-latin-america-and-the-caribbean.csl ecoscience.csl egretta.csl elementa.csl emerald-harvard.csl ens-de-lyon-centre-d-ingenierie-documentaire.csl environment-and-urbanization.csl epidemiology-and-infection.csl escuela-nacional-de-antropologia-e-historia-full-note.csl estuaries-and-coasts.csl estudios-de-cultura-maya.csl estudios-de-fonetica-experimental.csl estudios-hispanicos.csl ethnomusicology.csl europace.csl european-journal-for-philosophy-of-religion.csl european-journal-of-ultrasound.csl european-respiratory-journal.csl evolution.csl evolutionary-anthropology.csl evolutionary-ecology-research.csl exercer.csl fachhochschule-sudwestfalen.csl facolta-teologica-dell-italia-settentrionale-milano.csl feminist-economics.csl forensic-anthropology.csl forensic-science-review.csl freie-hochschule-stuttgart.csl frontiers-in-optics.csl gallia-prehistoire.csl generic-style-rules-for-linguistics.csl geochemical-perspectives-letters.csl geochronometria.csl geophysical-journal-international.csl gesellschaft-fur-popularmusikforschung.csl gigascience.csl gost-r-7-0-5-2008-numeric.csl hand.csl harvard-cranfield-university.csl harvard-institut-fur-praxisforschung-de.csl harvard-manchester-metropolitan-university.csl harvard-melbourne-polytechnic.csl harvard-oxford-brookes-university-faculty-of-health-and-life-sciences.csl harvard-swinburne-university-of-technology.csl harvard-university-of-abertay-dundee.csl harvard-university-of-greenwich.csl harvard-university-of-the-west-of-england.csl harvard-university-of-wolverhampton.csl harvard-university-of-worcester.csl health-economics-policy-and-law.csl heart-rhythm.csl henoch.csl historia-scribere.csl historical-social-research.csl history-and-theory.csl history-of-the-human-sciences.csl hochschule-bonn-rhein-sieg.csl human-reproduction.csl human-resource-management-journal.csl human-wildlife-interactions.csl humboldt-state-university-environmental-resources-engineering.csl hydrobiologia.csl hydrological-sciences-journal.csl ices-journal-of-marine-science.csl im-gesprach.csl indian-journal-of-medical-research.csl infomin.csl institut-national-de-la-recherche-scientifique-sciences-sociales.csl instituto-de-investigaciones-sobre-la-universidad-y-la-educacion-moderno.csl instituto-superior-de-teologia-de-las-islas-canarias.csl inter-ro.csl interactive-cardiovascular-and-thoracic-surgery.csl international-development-policy.csl international-energy-agency-organisation-for-economic-co-operation-and-development.csl international-journal-of-climatology.csl international-journal-of-epidemiology.csl international-journal-of-geriatric-psychiatry.csl international-journal-of-lexicography.csl international-journal-of-occupational-medicine-and-environmental-health.csl international-organization.csl international-review-of-the-red-cross.csl inventaire-general-du-patrimoine-culturel-iso-690-full-note-with-ibid.csl inventaire-general-du-patrimoine-culturel-iso-690-full-note.csl inventaire-general-du-patrimoine-culturel-iso-690-note.csl invisu.csl iran-manual-of-style.csl iso690-full-note-cs.csl iso690-full-note-with-ibid-ro.csl iso690-note-fr.csl journal-and-proceedings-of-the-royal-society-of-new-south-wales.csl journal-de-la-societe-des-americanistes.csl journal-for-the-study-of-the-new-testament.csl journal-for-veterinary-medicine-biotechnology-and-biosafety.csl journal-fur-kunstgeschichte.csl journal-of-advertising-research.csl journal-of-agricultural-and-resource-economics.csl journal-of-applied-animal-science.csl journal-of-biomedical-materials-research-part-a.csl journal-of-bone-and-mineral-research.csl journal-of-breast-cancer.csl journal-of-business-logistics.csl journal-of-cardiothoracic-and-vascular-anesthesia.csl journal-of-clinical-and-translational-science.csl journal-of-combinatorics.csl journal-of-evolutionary-biology.csl journal-of-field-ornithology.csl journal-of-food-protection.csl journal-of-health-care-for-the-poor-and-underserved.csl journal-of-hearing-science.csl journal-of-infectious-diseases.csl journal-of-instrumentation.csl journal-of-interactive-marketing.csl journal-of-investigative-dermatology.csl journal-of-linguistics.csl journal-of-medical-internet-research.csl journal-of-new-zealand-grasslands.csl journal-of-nutrition.csl journal-of-petrology.csl journal-of-physical-therapy-science.csl journal-of-political-ideologies.csl journal-of-product-innovation-management.csl journal-of-psychiatry-and-neuroscience.csl journal-of-retailing.csl journal-of-rheumatology.csl journal-of-studies-on-alcohol-and-drugs.csl journal-of-the-air-and-waste-management-association.csl journal-of-the-american-college-of-surgeons.csl journal-of-the-korean-society-of-civil-engineers.csl journal-of-the-royal-statistical-society.csl journal-of-the-science-of-food-and-agriculture.csl journal-of-thermal-spray-technology.csl journal-of-thrombosis-and-haemostasis.csl journal-of-tropical-ecology.csl journal-of-urban-and-environmental-engineering.csl journal-of-value-inquiry.csl journal-of-wildlife-diseases.csl journal-of-zoo-and-wildlife-medicine.csl journal-of-zoo-biology.csl knee-surgery-sports-traumatology-arthroscopy.csl lancaster-university-harvard.csl language.csl latin-american-perspectives.csl lettres-et-sciences-humaines-fr.csl london-metropolitan-university-harvard.csl management-international.csl management-of-biological-invasions.csl mastozoologia-neotropical.csl medicinskiy-akademicheskiy-zhurnal.csl mercator-institut-fur-sprachforderung-und-deutsch-als-zweitsprache.csl metropol-verlag.csl metropolia-university-of-applied-sciences-harvard.csl mimbar-hukum.csl mind-and-language.csl mineralogical-magazine.csl mis-quarterly.csl modern-humanities-research-association-author-date.csl mondes-en-developpement.csl music-theory-spectrum.csl national-library-of-medicine-grant-proposals.csl national-natural-science-foundation-of-china.csl national-university-of-singapore-department-of-geography-harvard.csl natures-sciences-societes.csl neurology.csl new-zealand-plant-protection.csl nordic-pulp-and-paper-research-journal.csl northeastern-naturalist.csl nouvelles-perspectives-en-sciences-sociales.csl nutrition-research-reviews.csl obafemi-awolowo-university-faculty-of-technology.csl occupational-medicine.csl oncotarget.csl organization.csl oryx.csl owbarth-verlag.csl oxford-art-journal.csl oxford-university-press-humsoc.csl padagogische-hochschule-heidelberg.csl padagogische-hochschule-vorarlberg.csl palaeontologia-electronica.csl pediatric-anesthesia.csl pedosphere.csl pest-management-science.csl phycological-research.csl phytopathologia-mediterranea.csl plant-species-biology.csl podzemna-voda.csl political-studies.csl pour-reussir-note.csl presses-universitaires-de-strasbourg-note.csl processing-and-application-of-ceramics.csl psychological-medicine.csl public-health-nutrition.csl radiographics.csl radiology.csl refugee-survey-quarterly.csl religion-in-the-roman-empire.csl representation.csl reviews-of-modern-physics-with-titles.csl revista-chilena-de-derecho-y-tecnologia.csl revista-cubana-de-meteorologia.csl revista-de-filologia-espanola.csl revista-noesis.csl revue-europeenne-des-migrations-internationales.csl rhodora.csl romanian-humanities.csl rose-school.csl rossiiskii-fiziologicheskii-zhurnal-imeni-i-m-sechenova.csl sage-harvard.csl scandinavian-journal-of-work-environment-and-health.csl science-china-earth-sciences.csl science-china-life-sciences.csl seminaire-saint-sulpice-ecole-theologie.csl society-for-historical-archaeology.csl society-of-biblical-literature-author-date.csl sorbonne-student-law-review.csl spanish-legal.csl spectroscopy-letters.csl spie-journals.csl springer-humanities-author-date.csl springer-humanities-brackets.csl springer-imis-series-migrationsgesellschaften.csl springer-vancouver-author-date.csl springer-vancouver-brackets.csl st-patricks-college.csl statistika-statistics-and-economy-journal.csl strategic-design-research-journal.csl strategic-entrepreneurship-journal.csl strategic-management-journal.csl studii-teologice.csl style-manual-australian-government-note.csl sunway-college-johor-bahru.csl tatup-zeitschrift-fur-technikfolgenabschatzung-in-theorie-und-praxis.csl taylor-and-francis-chicago-author-date.csl teologia-catalunya.csl the-american-journal-of-bioethics.csl the-american-journal-of-pathology.csl the-american-midland-naturalist.csl the-astrophysical-journal.csl the-biological-bulletin.csl the-bone-and-joint-journal.csl the-canadian-geographer.csl the-cancer-journal.csl the-holocene.csl the-international-journal-of-psychoanalysis.csl the-journal-of-laryngology-and-otology.csl the-journal-of-pain.csl the-journal-of-parasitology.csl the-journal-of-the-torrey-botanical-society.csl the-open-university-harvard.csl the-open-university-s390.csl the-quarterly-journal-of-economics.csl the-scandinavian-journal-of-clinical-and-laboratory-investigation.csl thrombosis-and-haemostasis.csl tissue-engineering.csl transversalites.csl tropical-animal-health-and-production.csl tsaqafah.csl ucl-institute-of-education-harvard.csl ucl-university-college-harvard.csl ucl-university-college-vancouver.csl ugeskrift-for-laeger.csl ultrasound-in-medicine-and-biology.csl uludag-universitesi-sosyal-bilimler-enstitusu-author-date.csl unesco-international-institute-for-educational-planning.csl unified-style-sheet-for-linguistics-de-gruyter-literature.csl unified-style-sheet-for-linguistics.csl united-nations-conference-on-trade-and-development.csl united-nations-development-programme-icca-legal-review.csl universidad-de-leon-harvard.csl universidad-evangelica-del-paraguay.csl universidade-estadual-do-oeste-do-parana-programa-institucional-de-bolsas-de-iniciacao-cientifica.csl universita-pontificia-salesiana-it.csl universita-pontificia-salesiana.csl universitat-basel-iberoromanistik.csl universitat-bremen-institut-fur-politikwissenschaft.csl universitat-heidelberg-historisches-seminar.csl universitat-mannheim-germanistische-linguistik.csl universite-catholique-de-louvain-histoire.csl universite-de-bordeaux-ecole-doctorale-de-droit.csl universite-de-lausanne-histoire.csl universite-du-quebec-a-montreal-etudes-litteraires-et-semiologie.csl university-of-aleppo-faculty-of-medicine.csl university-of-bradford-harvard.csl university-of-helsinki-faculty-of-theology.csl university-of-south-australia-harvard-2011.csl university-of-south-australia-harvard-2013.csl university-of-york-harvard-archaeology.csl university-of-york-harvard-environment.csl university-of-york-mla.csl vancouver-fr-ca.csl veterinary-radiology-and-ultrasound.csl victoria-university-harvard.csl vienna-legal.csl vita-latina-auteurs-anciens.csl water-environment-research.csl water-sa.csl wikipedia-templates.csl wirtschaftsuniversitat-wien-wirtschaftspadagogik.csl world-congress-on-engineering-asset-management.csl world-mycotoxin-journal.csl world-politcs.csl zdravniski-vestnik.csl zeitschrift-fur-fantastikforschung.csl zeitschrift-fur-padagogik.csl zwitscher-maschine.csl

denismaier commented 2 years ago

Here's a skeleton for a batch updater we can use once we figure out what actually needs to be done. The script will accept to arguments, a file containing styles to process, and a whitelist of styles that should not be updated even if they occur in the other file. Both files should contain one style per line.

import os, argparse

# parse arguments
parser = argparse.ArgumentParser()
parser.add_argument("-f", "--file", type=str, help="file containing a list of styles to update", default="styles.txt")
parser.add_argument("-w", "--whitelist", help="whitelist of style", type=str, default="whitelist.txt")
args = parser.parse_args()

# files to process
to_process_file = args.file
whitelist_file = args.whitelist

to_process = []
whitelist = []

# read files
if os.path.exists(to_process_file):
    with open(to_process_file, encoding='utf8') as f:
        to_process = [line.rstrip('\n') for line in f]

if os.path.exists(whitelist_file):
    with open(whitelist_file, encoding='utf8') as f:
        whitelist = [line.rstrip('\n') for line in f]

# filter out styles in whitelist from to_process list
to_process = [style for style in to_process if style not in whitelist]

# Update styles

for style in to_process:
    print(f"Updating: {style}", end=": ")
    print("I still need to figure out what to do here")

As for the actual updating, we can maybe do this with xslt or python itself, but I'll still have to figure out how exactly. So if someone is experienced with lxml....

denismaier commented 2 years ago

Or maybe xmldiff could help us here... https://pypi.org/project/xmldiff/

POBrien333 commented 1 year ago

knee-surgery-sports-traumatology-arthroscopy.csl was dealth with in https://github.com/citation-style-language/styles/pull/6441

citation-style-language / styles

Remove publisher information for styles in (most) articles #5872