search

numaproj / numaflow-python

Numaflow Python SDK
Apache License 2.0
52 stars 18 forks source link

Parallelize the update script #142

Closed ayildirim21 closed 6 months ago

ayildirim21 commented 6 months ago

Summary

Currently when we use the update script to automate the docker build/push process, it is run sequentially for each example image. Thus it takes 10-15 minutes for the command to finish executing and for the images to be pushed to quay.io.

Use Cases

After an SDK change is merged, as the update script is used to update the example images.

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

ayildirim21 commented 6 months ago

Possible solution is figuring out how to use local references, and then using matrix strategy to run each example images in parallel in a Github Actions workflow, similar to what is done in the Java and Go SDKs.