Suggestion needed - Wrapping AsyncGenerator

[Chuckytuh]

Sorry for opening an issue with what is basically a question/ feedback for API ergonomics...

Has any of you stumbled upon a situation where you'd want to wrap an AsynGenerator in a Result somehow where the ultimate goal is to typify the possible errors and effectively not have it throwing exceptions?

Let's say that the AsyncGenerator is wrapping a REST API Call and yielding results over time:

type SomeData = {};

type SomeApiResult = {
    content: SomeData[];
    nextToken?: string;
};

const getData = async (nextToken?: string): Promise<SomeApiResult> => /*...*/;

async function* getAllData(): AsyncGenerator<SomeData> {
    let loadMore = true;
    let prevNextToken: string | undefined = undefined;
    while (loadMore) {
        const result = await getData(prevNextToken);
        for (const data of result.content) {
            yield data;
        }
        loadMore = result.nextToken !== prevNextToken;
        prevNextToken = result.nextToken;
    }
}

One way I see it, is to have the generator return Results and as soon as an exception is thrown/promise rejection occurs, yield the error. This breaks the iteration as soon as the error occurs however, the side effect is that now the consumer has to unwrap each received value from the result and this breaks the norms of common generators.

async function* getAllData(): AsyncGenerator<Result<SomeData, SomeError>>;

[paduc]

Hi @Chuckytuh, thanks for opening this discussion.

I don’t use generators personally but here is my interpretation: the Result type needs to be unwrapped where the value is required. In your case, each time the generator yields a result, it can either be a value or an error, right? So it seems logical to me that the generator yields a Result and that the consumer unwraps that result and handles both cases (value or error).

Do you any other way to handle the error case ?

[Chuckytuh]

Thanks @paduc for engaging on the discussion!

Yes, that makes sense and was my first impression as well, sadly, this breaks the norm which means that other APIs, such as Readable.from won't work out of the box because it expects async generators to throw an exception ir order to close the readable.

It seems to be one of those cases where we can't get both worlds to live happily. If we want our AsyncGenerators to have errors modelled into the return type (through Result) we break interoperability with the rest of the ecosystem or, most probably, I'm overlooking something that is obvious :)

[paduc]

If the consumer expects an exception for the normal case, then it’s totally fine to throw an exception to comply.

´Result’ are useful to have typed errors, meaning the consumer can determine which kind of error is thrown. In your case, the expected exception is not really an error but rather a signal for the consumer to stop. I don’t think it’s appropriate to wrap that exception in a ´Result’. If your generator does have other types of errors (ex: connection lost), then you can use the ´Result’ type to handle those.

[supermacro]

Going to leave this issue open as I don't have much experience with Generators. Hopefully others in the community can chime in and provide more info!

[supermacro]

I'm adding a bounty to this if it ever gets resolved; $100 USD

[konradekk]

That case would be better solved by streams (see e.g. RxJS solutions). What you roughly do here is: maintain a stream reading iterables that you flatten (with mapping) into the stream. You could convert to async iteration from the streaming with some small effort (see e.g. this implementation on StackOverflow).

Iʼd second @paduc here (providing him with bounty!) and avoid Result completely here.

[supermacro]

This is solved via safeTry is it not? https://github.com/supermacro/neverthrow/?tab=readme-ov-file#safetry