Closed arturponinski closed 1 year ago
Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | ||
---|---|---|---|---|---|
petl/io/json.py | 46 | 48 | 95.83% | ||
<!-- | Total: | 95 | 97 | 97.94% | --> |
Totals | |
---|---|
Change from base Build 2692922856: | 0.04% |
Covered Lines: | 12713 |
Relevant Lines: | 13956 |
This PR has the objective of improving the support of generators in fromdicts. The current implementation uses itertools.tee which according to docs and production deployments uses large amounts of memory, leading to out of memory kills of processes. This PR aims to keep the improved support of generators by using a filecache, similar to sorting, to allow multiple iterations.
Closes https://github.com/petl-developers/petl/issues/618
Changes Moved _iterchunk from sorts to petl.util.base. Imported to sorts as _iterchunk for BC
Changes
Checklist
Use this checklist for assuring the quality of pull requests that include new code and or make changes to existing code.
tox
/pytest