VirtusLab / besom

Besom - a Pulumi SDK for Scala. Also, incidentally, a broom made of twigs tied round a stick. Brooms and besoms are used for protection, to ward off evil spirits, and cleansing of ritual spaces.
https://virtuslab.github.io/besom/
Apache License 2.0
122 stars 7 forks source link

Add parSequence and parTraverse combinators to Result and Output #438

Closed lbialy closed 5 months ago

lbialy commented 6 months ago

Current implementation of sequence/traverse functions is sequential which can very easily lead to accidental serialisation of operations. Given that resource constructors can be painfully slow in infra it should be possible for the user to decide whether their side effects have to be executed in given order or in parallel. We could just make sequence and traverse parallel by default (I think that's how Output.allOf works in pulumi-java and All/Output.all combinators in other SDKs work given that all of them are strict!) but that could be misleading, surprising and not really helpful for people used to working with lazy monads in Scala.

pawelprazak commented 5 months ago

Sounds good. For the record, you are right, Java uses CompletableFuture.allOf like this

public static <T> CompletableFuture<List<T>> allOf(Collection<CompletableFuture<T>> futures) {
        return CompletableFuture
                .allOf(futures.toArray(new CompletableFuture[futures.size()]))
                .thenApply(unused -> futures.stream()
                        .filter(ignoreNullValues())
                        .map(CompletableFuture::join) // join() is not blocking here, by the time this function is called, the future is guaranteed to be complete
                        .collect(Collectors.toList())
                );
    }

This will wait for all of the futures without any ordering. So it can be parallel in theory.

lbialy commented 5 months ago

In java it is parallel by default because CompletableFuture is not lazy :)

Implemented and done.

pawelprazak commented 5 months ago

I remember there was some nuance regarding supply vs. supplyAsync but I don't remember the details any more ;)