agrc / palletjack

A library for updating AGOL data from various external sources
https://agrc.github.io/palletjack/palletjack/
MIT License
12 stars 0 forks source link

fix: empty shapes, better memory usage #71

Closed jacobdadams closed 11 months ago

jacobdadams commented 11 months ago

Will bump a minor version.

codecov[bot] commented 11 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (7913200) 93.30% compared to head (1e686a3) 93.32%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #71 +/- ## ========================================== + Coverage 93.30% 93.32% +0.01% ========================================== Files 7 7 Lines 1061 1063 +2 Branches 143 144 +1 ========================================== + Hits 990 992 +2 Misses 62 62 Partials 9 9 ``` | [Files](https://app.codecov.io/gh/agrc/palletjack/pull/71?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=agrc) | Coverage Δ | | |---|---|---| | [src/palletjack/extract.py](https://app.codecov.io/gh/agrc/palletjack/pull/71?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=agrc#diff-c3JjL3BhbGxldGphY2svZXh0cmFjdC5weQ==) | `92.14% <100.00%> (ø)` | | | [src/palletjack/utils.py](https://app.codecov.io/gh/agrc/palletjack/pull/71?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=agrc#diff-c3JjL3BhbGxldGphY2svdXRpbHMucHk=) | `95.07% <100.00%> (+0.02%)` | :arrow_up: |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

jacobdadams commented 11 months ago

can you explain where the better memory usage comes from? I don't see anything that stands out right away.

in L618-628, rather than storing each dataframe to a list and then concating them all at the end, I concat as I go. Depending on how the numpy arrays are managed under the hood, it may not actually save a ton.