Fix fromdictsgenerator separate function

[arturponinski]

This PR has the objective of improving the support of generators in fromdicts. THe current implementation uses itertools.tee which according to docs and production deployments uses large amounts of memory, leading to out of memory kills of processes. This PR aims to revert the functionality and introduce a dedicated fromdictsgenerator function, which uses filecache, similar to sorting.

Changes

1. Added new fromdictsgenerator to petl.io.json
2. Reverted the detection of generators in fromdicts introduced in 1.7.5 and added a warning about passing the generator to fromdicts
3. Added info about the function to docs
4. Moved _iterchunk from sorts to petl.util.base. Imported to sorts as _iterchunk for BC

Checklist

Use this checklist for assuring the quality of pull requests that include new code and or make changes to existing code.

• [ ] Source Code rules apply:
  • [x] Includes unit tests
  • [x] New functions have docstrings with examples that can be run with doctest
  • [x] New functions are included in API docs
  • [ ] Docstrings include notes for any changes to API or behaviour
  • [ ] All changes are documented in docs/changes.rst
• [ ] Versioning and history tracking rules apply:
  • [x] Using atomic commits when possible
  • [x] Commits are reversible when possible
  • [x] There is no incomplete changes in the pull request
  • [x] There is no accidental garbage added in source code
• [ ] Testing rules apply:
  • [x] Tested locally using tox / pytest
  • [x] Automated testing passes (see CI)
  • [x] Unit test coverage has not decreased (see Coveralls)
• [ ] State of these changes is:
  • [ ] Just a proof of concept
  • [ ] Work in progress / Further changes needed
  • [x] Ready to review
  • [x] Ready to merge

[coveralls]

Pull Request Test Coverage Report for Build 2659590742

• 87 of 89 (97.75%) changed or added relevant lines in 5 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.03%) to 91.081%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
petl/io/json.py	40	42	95.24%
<!--	Total:	87	89	97.75%	-->

Totals
Change from base Build 2458237953:	0.03%
Covered Lines:	12693
Relevant Lines:	13936

---

💛 - Coveralls

[bmaggard]

As we are not bumping releases, should this be handled transparently in fromdicts() as it currently is, so as not to break API?

[juarezr]

As we are not bumping releases, should this be handled transparently in fromdicts() as it currently is, so as not to break API?

Feedback from production deployments always looks like an excellent way to improve.

@arturponinski,

As you have explored this issue closer:

• Do you see any impact in keeping the API as suggested by @bmaggard?
  • Or a flag would be required for handling both cases?
• Do you think the change introduced in v1.7.5 was an undetected regression?
  • If that's the case, how could we reduce the impact of the upgrade on the pre/pos version users?
  • Have any idea?

[arturponinski]

@bmaggard @juarezr the more I think about it, the more I lean to your suggestion to keep the support in the fromdicts.

1. Regarding the flag, adding it would make it not used by anyone in my opinion.
2. It seems as a regression, yes. This issue made us revert the petl update to the previous version we used
3. I think using temporary files should not be a major issue now, especially that sorting also uses files for cache. Perhaps updating the docs, stating that files are used for cache when generators are passed is worth considering.

Due to above, I've created another PR which keeps the BC and uses the file cache: https://github.com/petl-developers/petl/pull/625

[arturponinski]

Superseded by https://github.com/petl-developers/petl/pull/625