Requestor array factories

[diachedelic]

Often I find it necessary to generate a requestor array using the result of a prior requestor. I love that parseq is able to do this, but my approach is hard to follow and thus error prone.

function download_file(url) {
    return download_file_requestor(callback) {
        ...
    }
}

parseq.sequence([
    fetch_urls(),
    download_files_requestor(callback, urls) {
        try {
            return parseq.parallel(
                urls.map(download_file)
            )(callback); // It's easy to miss this invocation!
        } catch (exception) {
            return callback(undefined, exception);
        }
    }
]);

It occurred to me that parseq could accept functions in place of requestor arrays. These would take the value and return a requestor array.

parseq.sequence([
    fetch_urls(),
    parseq.parallel(
        function make_required_requestors(urls) {
            return urls.map(download_file);
        }
    )
]);

Is this a good idea?

[douglascrockford]

It is easy enough to fill an array with a function. I would rather keep Parseq simple.

[bunglegrind]

Often I find it necessary to generate a requestor array using the result of a prior requestor. I love that parseq is able to do this, but my approach is hard to follow and thus error prone. Is this a good idea?

You can wrap parseq with some helper functions, maintaining the core intact.

For instance, in your example:

function apply_parallel(
    requestor_factory,
    optional_requestor_factory,
    time_limit,
    time_option,
    throttle
) {
    return function (callback, value) {
        try {
            return parseq.parallel(
                value.map(requestor_factory),
                (
                    typeof optional_requestor_factory === "function"
                    ? value.map(optional_requestor_factory)
                    : []
                ),
                time_limit,
                time_option,
                throttle
            )(callback);
        } catch (e) {
            return callback(undefined, e);
        }
    };
}

[diachedelic]

That is surely better than complicating parseq, and the solution can be generalised further!

The lazy factory lets you delay the creation of a requestor until its input value is known. This can be useful when the arguments passed to a requestor factory are not known ahead of time.

function lazy(requestor_factory, initial_value) {
    return function lazy_requestor(callback, value) {
        try {
            return requestor_factory(value)(callback, initial_value);
        } catch (exception) {
            return callback(undefined, exception);
        }
    };
}

It helps avoid both treacherous boilerplate and excessive indirection.

parseq.sequence([
    fetch_urls(),
    lazy(function (urls) {
        return parseq.parallel(urls.map(download_file));
    })
])

[bunglegrind]

That's nice! btw, I suggest you to take a look a this repository: https://github.com/jlrwi/requestors