Parent / Children Tasks

[walmat]

Is your feature request related to a problem? Please describe. Currently, when multiple tasks are running on the same site, the same variants of the same product & site will monitor the same endpoints and are sending 3 requests / task to monitor data. This seems pretty taxing on proxies if you ask me, and seems a little wasteful.

Describe the solution you'd like Maybe we create a structure, where tasks monitoring the same site for the same product are split into 1 parent task and the rest children tasks of that parent task. This way, only the parent task would monitor the site for the product, and when found, propagate the data to the children tasks.

Describe alternatives you've considered Dunno yet! Just thought of this.

[pr1sm]

I like this! Another alternative to using parent/child tasks is to build a cache of product data we've captured as we're making requests. When we request a site's product data or even variant data for a specific product, we can check our cache first before making the request. This would remove the need for a "parent" task, but would allow us to solve the problem of making too many requests.

The cache itself would have to stored in a file, or in-memory on the main thread, then task-runners would make a request for the data through some function (similar to the request function they are making now) If there is a cache hit, we return the data. If there is cache miss, we make the request using all the same parameters as the request function (specifically the given cookie jar) and update the cache.

The cache would have to have a pretty short "hot" time so we can still get recent data -- I'm thinking we use the monitorDelay. This would mean the the first task to miss the cache would keep it hot on any request it makes, while any other requests from other tasks would use the cached data.