Right now, when records are fetched from the server, only the bare minimum is fetched, with the rest coming on-demand as you access fields of the record. This works in general, however if the user wants to download entire records (+ children), the result is a very large number of small requests.
This PR improves this. There are three main parts:
Being able to say include=['**'], which will result in the entire record being downloaded, including children
To handle fetching children, the children of multiple parent records are fetched together. For example, singlepoint records from the trajectories of multiple optimization records are batched together.
A time-based chunking iterator to help prevent very large requests from choking the server. This also includes a thread pool for background fetching
This requires a few server side changes, so unfortunately this cannot be tested against the running production servers. I am planning on a release in early April, though.
Changelog description
Improve fetching speed by allowing for including more of records
Status
[X] Implement new includes handling and child fetching
Description
Right now, when records are fetched from the server, only the bare minimum is fetched, with the rest coming on-demand as you access fields of the record. This works in general, however if the user wants to download entire records (+ children), the result is a very large number of small requests.
This PR improves this. There are three main parts:
Being able to say
include=['**'], which will result in the entire record being downloaded, including childrenTo handle fetching children, the children of multiple parent records are fetched together. For example, singlepoint records from the trajectories of multiple optimization records are batched together.
A time-based chunking iterator to help prevent very large requests from choking the server. This also includes a thread pool for background fetching
This requires a few server side changes, so unfortunately this cannot be tested against the running production servers. I am planning on a release in early April, though.
Changelog description
Improve fetching speed by allowing for including more of records
Status