The-Academic-Observatory / academic-observatory-workflows

Telescopes, Workflows and Data Services for the Academic Observatory
https://academic-observatory-workflows.readthedocs.io
Apache License 2.0
16 stars 1 forks source link

Fix/Pubmed Telescope: Merge upserts and deletes #181

Closed alexmassen-hane closed 1 year ago

alexmassen-hane commented 1 year ago

A proposed fix for the multiprocessing step in the Pubmed telescope for writing out the merged upserts and deletes. The function save_merged_upserts_and_deletes now returns the path to the file that it writes to. The task will now only log the path if the spawned process in the process pool has completed, using future.result().

Also added logging to the merge_upserts_and_deletes function so that we know it is processing each updatefile properly.

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 85.71% and project coverage change: -0.04% :warning:

Comparison is base (c0395d9) 95.22% compared to head (c60c0f5) 95.18%. Report is 1 commits behind head on develop.

:exclamation: Current head c60c0f5 differs from pull request most recent head 802169c. Consider uploading reports for the commit 802169c to get more accurate results

Additional details and impacted files ```diff @@ Coverage Diff @@ ## develop #181 +/- ## =========================================== - Coverage 95.22% 95.18% -0.04% =========================================== Files 20 20 Lines 5191 5197 +6 Branches 718 719 +1 =========================================== + Hits 4943 4947 +4 Misses 161 161 - Partials 87 89 +2 ``` | [Files Changed](https://app.codecov.io/gh/The-Academic-Observatory/academic-observatory-workflows/pull/181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The-Academic-Observatory) | Coverage Δ | | |---|---|---| | [...bservatory\_workflows/workflows/pubmed\_telescope.py](https://app.codecov.io/gh/The-Academic-Observatory/academic-observatory-workflows/pull/181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The-Academic-Observatory#diff-YWNhZGVtaWNfb2JzZXJ2YXRvcnlfd29ya2Zsb3dzL3dvcmtmbG93cy9wdWJtZWRfdGVsZXNjb3BlLnB5) | `92.53% <85.71%> (-0.10%)` | :arrow_down: | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/The-Academic-Observatory/academic-observatory-workflows/pull/181/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The-Academic-Observatory)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.