Fix generator support in fromdicts - use file cache instead of iterto…

[arturponinski]

…ols.tee

This PR has the objective of improving the support of generators in fromdicts. The current implementation uses itertools.tee which according to docs and production deployments uses large amounts of memory, leading to out of memory kills of processes. This PR aims to keep the improved support of generators by using a filecache, similar to sorting, to allow multiple iterations.

Closes https://github.com/petl-developers/petl/issues/618

Changes

1. Refactored DictsGeneratorView in petl.io.json to use file cache
2. Moved _iterchunk from sorts to petl.util.base. Imported to sorts as _iterchunk for BC

Checklist

Use this checklist for assuring the quality of pull requests that include new code and or make changes to existing code.

• [ ] Source Code rules apply:
  • [x] Includes unit tests
  • [ ] New functions have docstrings with examples that can be run with doctest
  • [ ] New functions are included in API docs
  • [ ] Docstrings include notes for any changes to API or behaviour
  • [ ] All changes are documented in docs/changes.rst
• [ ] Versioning and history tracking rules apply:
  • [x] Using atomic commits when possible
  • [x] Commits are reversible when possible
  • [x] There is no incomplete changes in the pull request
  • [x] There is no accidental garbage added in source code
• [x] Testing rules apply:
  • [x] Tested locally using tox / pytest
  • [x] Automated testing passes (see CI)
  • [x] Unit test coverage has not decreased (see Coveralls)
• [ ] State of these changes is:
  • [ ] Just a proof of concept
  • [ ] Work in progress / Further changes needed
  • [x] Ready to review
  • [x] Ready to merge

[coveralls]

Pull Request Test Coverage Report for Build 2693186888

• 78 of 79 (98.73%) changed or added relevant lines in 4 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.03%) to 91.085%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
petl/io/json.py	34	35	97.14%
<!--	Total:	78	79	98.73%	-->

Totals
Change from base Build 2692922856:	0.03%
Covered Lines:	12689
Relevant Lines:	13931

---

💛 - Coveralls

[arturponinski]

@juarezr I've updated the docstring for fromdicts, please re-review this. Once you accept I'd like to ship it :tada:

[juarezr]

@arturponinski,

Nice solution. If you need any help, please don't be afraid of contacting me.

[juarezr]

@arturponinski,

Nice solution. If you need any help, please don't be afraid of contacting me.

@arturponinski,

I didn't understand if you consider this PR ready to merge. I don't have any objections if you want to merge and ship it.

[arturponinski]

@juarezr this took a while. I had asked for some more reviews and discussion about this solution and ended up not entirely happy with it. Dumping the whole generator to the file before yielding any data was considered as a faulty solution. Done some more trials and I've finally created a solution that I am happy with: https://github.com/petl-developers/petl/pull/626

[juarezr]

@arturponinski,

Can we close this as #626 superseded it and it was merged?

[arturponinski]

Definitely to be closed, this one is obsolete now