Country share feature

[agentilb]

https://repo.scoap3.org/admin/countriesshare/

This script should:

• For a set of articles for a specific time range (publication dates).
• Take all the authors for the articles, and check their countries.
• In case of double/triple affiliation:
  • If one affiliation is CERN, the country chosen is CERN
  • If one affiliation is one HEP institute: KEK, SLAC, DESY or FERMILAB, the country chosen is the country of the HEP institution.
  • If no CERN affiliation or HEP institute, the country with the biggest GDP is chosen.
• The result should be an csv file with the list of all articles, with the repartition by country (1 line per article, 1 column per country)

Attached:

• List of 2020 GDP by country: API_NY.GDP.PCAP.PP.CD_DS2_en_csv_v2_4150885.csv

• File that I extracted from the repo using this feature, but it doesn't seem to work, since I get only 700 lines (where I should get some 15k lines): countries_share-3.csv

Would it be possible to see if we can make this feature work? We need data by the end of August for the years 2020 and 2021.

[drjova]

@agentilb the current implementation has many issues, i believe we will have to build it from scratch. At the end do you need the report to be a downloadable csv file? Do we have to store anything in the database?

[agentilb]

@drjova we need to be able to do the calculation, based on the criteria described above, it doesn't need to be stored in the database.

[drjova]

As we discuss we have to prepare a script

[Neodia]

Write a python script runnable in command line. The script parameters are :

• from_date
• to_date Both dates are in ISO-8601 format.

The script will then :

• Query ES with the given dates in the imprint dates.
• From the json response, we need :
  • DOI
  • recid
  • journal
  • creation_date
  • primary_category
  • authors
• For each journal :
  • Count number of authors
  • Arrange the authors by their countries
  • The countries should be defined by following the process described on top (in the issue description)
  • Count them and write the percentage to the CSV based on their country
• Export the end result to CSV