Dependent resources loop over all data items in a batch sent from the connected parent. We can easily parallelize this loop.
detect if parallelize flag is enabled on a transformer when generating loop over items
if so use decorated version of the function (via dlt.defer)
in essence such decorator wraps transformer function in callables and many of those can be executed at the same time
Source name
rest_api
Describe the data you'd like to see
Dependent resources loop over all data items in a batch sent from the connected parent. We can easily parallelize this loop.
parallelizeflag is enabled on a transformer when generating loop over itemsAre you a dlt user?
Yes, I'm already a dlt user.
Do you ready to contribute this extension?
Yes, I'm ready.
dlt destination
No response
Additional information
No response