Implementation of IssnSubjectCodes for Crossref-2024 with download method

[panos-span]

Title: Implementation of IssnSubjectCodes for Crossref-2024, introducing a new method for data_source called download.

Description: This pull request introduces the IssnSubjectCodes module for Crossref-2024 into the Alexandria3k project. The module processes ISSN to ASJC (All Science Journal Classification) subject codes, supporting queries over its virtual table and enabling the population of an SQLite database with its data. This is a great work-around in order to get the subjects for the works table since crossref doesn't support them anymore.

Key features of this implementation include:

• Definition of the IssnSubjectCodes class which creates an object to handle ISSN to ASJC meta-data.
• A method to populate an SQLite database with ISSN and subject codes data, with an initial step to create the required CSV from API calls.
• Mock tests for validating the CSV creation and database population processes.
• Created the download command for data_source.py and __main__.py

Changes:

• Added issn_subject_codes.py to src/alexandria3k/data_sources/ with the implementation of the IssnSubjectCodes class.
• Added SQL script get-issns.sql to src/alexandria3k/sql/ for retrieving ISSNs from the works table.
• Added a test CSV issn_subjects.csv to tests/data/ for unit tests.
• Added unit tests in test_issn_subject_codes.py under tests/data_sources/ to validate the module's functionality.

Commit Details:

• src/alexandria3k/data_sources/issn_subject_codes.py:

  • Introduced IssnSubjectCodes class.
  • Added method to create CSV from API calls and create a data folder where the script is executed.
  • Added populate method override to handle CSV creation and data insertion into the database.

• src/alexandria3k/sql/get-issns.sql:

  • Added SQL script to select distinct ISSNs from the works table.

• tests/data/issn_subjects.csv:

  • Added sample CSV data for unit testing.

• tests/data_sources/test_issn_subject_codes.py:

  • Added unit tests to validate the IssnSubjectCodes module's CSV creation and database population.

Library Used: This implementation utilizes the pybliometrics library to interact with the Elsevier API. The ASJC codes are retrieved through this API. Users will need an API key and an institutional token to access the Elsevier API. The config file pybliometrics.cfg must be inside the ./config folder of the user.

Testing: Unit tests have been added to verify:

• CSV creation with correct ISSN and subject code entries.
• Correct population of the SQLite database with the generated CSV data.

Notes: Ensure that the pybliometrics library is correctly initialized and that the required API keys are configured.

Checklist:

• [x] Code adheres to the project's style guidelines.
• [x] Documentation and comments are included as needed.
• [x] Unit tests are added and passing.

[dspinellis]

This is impressive; well done! Please:

• [ ] Fix the CI failure
• [ ] Add some indication of the time required to download the data
• [ ] Consider splitting the process into the formation of the data set (with a new action download verb and directory for its plugins) and the traditional population step.

[panos-span]

The API lets you make 9 requests per second, with a limit of 25000 requests per week, when having an institutional token, which needs to be requested by the Elsevier API support team. A user can add many API keys, not just one.

This data source should only be applied for custom examples, when a user has already in mind the range of the ISSN subjects that might be needed. I added the details in the start of the class.

[panos-span]

• [x] Add some indication of the time required to download the data
• [x] Consider splitting the process into the formation of the data set (with a new action download verb and directory for its plugins) and the traditional population step.

I implemented the latest recommended changes. I just don't know where to upload a test config file for pybliometrics so the CI will pass.

Once this will be addresed, i will squash my commits.

[dspinellis]

I implemented the latest recommended changes. I just don't know where to upload a test config file for pybliometrics so the CI will pass.

Once this will be addresed, i will squash my commits.

The CI script can copy the config file from tests/data to $HOME. However, note that the config file shall not include any secret (e.g. API) keys. If secrets are needed to test it, then you should mock the response.

[panos-span]

alexandria3k/data_sources/issn_subject_codes.py:89:4: R0913: Too many arguments (6/5) (too-many-arguments)

Should i give less options to the user? What should i do?

Otherwise everything else can be handled immediately.

[dspinellis]

alexandria3k/data_sources/issn_subject_codes.py:89:4: R0913: Too many arguments (6/5) (too-many-arguments)

Should i give less options to the user? What should i do?

• Add a new subcommand download which looks in the new download_sources for plugins. It can have as a positional argument the output data path and as an optional arguments an SQL query to yield the keys to download.
• Use tempfile.NamedTemporaryFile to obtain a temporary file path where the data will be stored. Delete this when the database gets populated.

[panos-span]

Everything should work as intended now!

[dspinellis]

Note that a data source may (as in your case) or may not (as in all others currently) support a download subcommand. In the second case a suitable error message should be shown.

[panos-span]

• [x] Added requested options for user for download command
• [x] Added necessary handling of workflow operation of download with populate
• [x] Removed unnecessary libraries
• [x] Created progress bar for ISSN downloading
• [x] Fixed white space git diff issue

If everything is looking good, i will squash again the commits into one, and this time try not to mess with the whitespaces.

[panos-span]

Now, we can accept for the populate method as input the input_path , as well as for the download method. In addition to that, we also check if the populate method signature can take the extra parameter so there wont be any occurring issues. Lastly, the file created is no longer temporary (the path is provided by the user of course).

[panos-span]

• [x] Added in data_source.py a default download method
• [x] Adjusted data_source parameter usage

[panos-span]

Still work to do I'm afraid. I hope this doesn't discourage you.

No problem! Every suggestion is welcome and appreciated.

[panos-span]

All done!