search

aphp / eds-scikit

eds-scikit is a Python library providing tools to process and analyse OMOP data
https://aphp.github.io/eds-scikit
BSD 3-Clause "New" or "Revised" License
35 stars 5 forks source link

small IO and biology fixes #33

Closed Thomzoy closed 1 year ago

Thomzoy commented 1 year ago

3 small fixes:

Saving tables locally

In spark client or cluster mode, saving a table as parquet won't work because of permission error: executors and driver aren't the same user.

How its solved:

Tables are first collected, and then saved locally by the driver only.

Incorrect timestamp error

When collecting tables, pyarrow throws an error when stumbling upon incorrect timestamps (smaller than pd.Timestamp.min or bigger than pd.Timestamp.max).

How it's solved

A filtering is done in HiveData, which remplace incorrect timestamps

Biology configuration file

When creating a configuration file via create_config_from_stats, one row per code AND care site is created, along with a line aggregating all care sites. We only want to keep this row (identified by df.care_site_short_name == "ALL")

How it's solved

Filtering, when necessary, on the care_site_short_name column

codecov-commenter commented 1 year ago

Codecov Report

Base: 91.89% // Head: 91.92% // Increases project coverage by +0.02% :tada:

Coverage data is based on head (bd30bca) compared to base (e29f4a5). Patch coverage: 91.30% of modified lines in pull request are covered.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #33 +/- ## ========================================== + Coverage 91.89% 91.92% +0.02% ========================================== Files 66 67 +1 Lines 2123 2143 +20 ========================================== + Hits 1951 1970 +19 - Misses 172 173 +1 ``` | [Impacted Files](https://codecov.io/gh/aphp/eds-scikit/pull/33?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aphp) | Coverage Δ | | |---|---|---| | [eds\_scikit/io/i2b2\_mapping.py](https://codecov.io/gh/aphp/eds-scikit/pull/33?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aphp#diff-ZWRzX3NjaWtpdC9pby9pMmIyX21hcHBpbmcucHk=) | `12.65% <0.00%> (ø)` | | | [eds\_scikit/utils/custom\_implem/custom\_implem.py](https://codecov.io/gh/aphp/eds-scikit/pull/33?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aphp#diff-ZWRzX3NjaWtpdC91dGlscy9jdXN0b21faW1wbGVtL2N1c3RvbV9pbXBsZW0ucHk=) | `87.50% <87.50%> (ø)` | | | [eds\_scikit/event/consultations.py](https://codecov.io/gh/aphp/eds-scikit/pull/33?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aphp#diff-ZWRzX3NjaWtpdC9ldmVudC9jb25zdWx0YXRpb25zLnB5) | `98.36% <100.00%> (+0.08%)` | :arrow_up: | | [eds\_scikit/io/data\_quality.py](https://codecov.io/gh/aphp/eds-scikit/pull/33?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aphp#diff-ZWRzX3NjaWtpdC9pby9kYXRhX3F1YWxpdHkucHk=) | `100.00% <100.00%> (ø)` | | | [eds\_scikit/io/hive.py](https://codecov.io/gh/aphp/eds-scikit/pull/33?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aphp#diff-ZWRzX3NjaWtpdC9pby9oaXZlLnB5) | `100.00% <100.00%> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aphp). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aphp)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.