Maximum call stack size exceeded on waitAll(long_list_of_simple_tasks).

damianrr commented 4 years ago

I've a long list of simple tasks (30k of them) and I'd like to be able to wait for all of them and it fails with a Maximum call stack size exceeded error

Steps to reproduce

If you run this program:

const { range, map } = require("ramda");
const { task, of, waitAll } = require("folktale/concurrency/task");                                                                                                                
const tasks = waitAll(map(of, range(1, 30000)));
tasks.run().listen({ onRejected: console.error, onResolved: console.log });

Expected behaviour

It should return a list of consecutive numbers from 1 to 29999

Observed behaviour

Yet, I get this error:

.../src/node_modules/folktale/adt/union/union.js:86 
    function makeInstance() {
                         ^                                                                                      

RangeError: Maximum call stack size exceeded
    at makeInstance (.../src/node_modules/folktale/adt/union/union.js:86:26)
    at new Deferred (.../src/node_modules/folktale/concurrency/future/_deferred.js:84:26) 
    at Task.run (.../src/node_modules/folktale/concurrency/task/_task.js:404:22)
    at Task._computation (.../src/node_modules/folktale/concurrency/task/_task.js:341:36)
    at Task.run (.../src/node_modules/folktale/concurrency/task/_task.js:445:28)
    at Task._computation (.../src/node_modules/folktale/concurrency/task/_task.js:93:32)
    at Task.run (.../src/node_modules/folktale/concurrency/task/_task.js:445:28)
    at Task._computation (.../src/node_modules/folktale/concurrency/task/_task.js:341:36)
    at Task.run (.../src/node_modules/folktale/concurrency/task/_task.js:445:28)
    at Task._computation (.../src/node_modules/folktale/concurrency/task/_task.js:93:32)

Environment

(Describe the environment where the problem happens. This usually includes:

OS: Arch Linux
JavaScript: Node v13.13.0
Folktale version: 2.3.2 )

damianrr commented 4 years ago

Any workaround/idea that could help me circumvent this issue will also be appreciated!

Thanks!

robotlolita commented 4 years ago

This happens because all of your tasks are synchronous, and JS VMs don't deal well with recursion. waitAll ends up building a stack of tasks, and when they're evaluated synchronous it ends up filling up the bounded stack (having VMs implement tail recursion would really help here).

The work-around involves pretty much managing the stack so it doesn't fill up. So, for example, you can make all of the tasks asynchronous, in which case all of them will start on an empty stack when they run:

const { range, map } = require("ramda");
const { task, of, waitAll } = require("folktale/concurrency/task");
const asyncOf  = (x) => task(resolver => setImmediate(() => resolver.resolve(x)));
const tasks = waitAll(map(asyncOf, range(1, 30000)));
tasks.run().listen({ onRejected: console.error, onResolved: console.log });

But this has performance implications, and it's not always as straight-forward to do. In the general usage, Tasks mostly contain asynchronous computations, where this isn't as much of a problem, but these edge-cases can be pretty frustrating. Ideally Folktale would solve this (e.g.: running tasks in a stack-safe manner, at the expense of some performance), but I haven't had the time to work on it lately, so I'm not sure when/if this will be fixed.

damianrr commented 4 years ago

Hey @robotlolita thanks for getting back to me and taking the time to give a very useful response. I don't mind them being async, what I actually need to do is a "fire and forget" type of thing, my only problem is that I tried running your snippet and it gave me the exact same error. Could you take a look at it again?

origamitower / folktale