numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs
https://numaflow.numaproj.io/
Apache License 2.0
1.03k stars 103 forks source link

Add ability for examples in SDKs to be built without needing to clone enitre repo #1658

Open ayildirim21 opened 4 months ago

ayildirim21 commented 4 months ago

Summary

Currently, in order to facilitate the CI of our SDKs efficiently, we use local references in the dependency management files of the examples. If a user just copies the examples folder without cloning the entire repo and tries to build a docker image, this would lead to an error. We should explore different ways that we can continue to have efficient CI while providing a good experience to new users that do not necessarily want to contribute to the SDK. Another point of discussion as mentioned by @kohlisid is:

Whether we want to expose the Dockerfiles and dependency management files to the user or not. If they care about the code files only and the Docker/Dependency is just for us, then they probably can be omitted from the structure altogether and we have those template in our script/internal folder to use whenever we need it for the CI.

This issue is meant to track the broader discussion of what we want the Numaflow user experience to be like, while also making it as easy as possible for developers to contribute, as Numaflow continues to grow and be adopted.

Use Cases

When a user is exploring the examples, or just onboarding to the Numaflow platform.


Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

ayildirim21 commented 4 months ago

As per the discussion: https://intuit-teams.slack.com/archives/C02P1NKMBHQ/p1712945480906909

ayildirim21 commented 4 months ago

Another thought is that we should probably move the image-push target outside the examples, as it is used by the CI and not necessarily needed by users that just want to build locally and that probably do not have access to quay.io.