numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs
https://numaflow.numaproj.io/
Apache License 2.0
1.1k stars 112 forks source link

Surface SDK in-compatibility issue to users through logs #1504

Closed KeranYang closed 6 months ago

KeranYang commented 8 months ago

Summary

SDK in-compatibility is one of the most common issues that users encounter when they are working on their test pipelines. A very common answer that we provide on the numaflow-support channel is “upgrade your numaflow version to v1.x.x” or “use a newer sdk version v0.x.x ”. It would be great if we can surface such information through logs. “Numaflow platform is running on version a.b.c while the UDF is running on numaflow-go version x.y.z, they are not compatible, please update numaflow-go to version l.m.n”.


Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

kohlisid commented 6 months ago

Just to clarify, In this issue we are trying to identify the Numaflow and SDK versions and then match whether they are compatible right? And then provide the end user this information

KeranYang commented 6 months ago

@kohlisid Yes, hopefully by having this, users can resolve it all by themselves without numa support involved.

ayildirim21 commented 6 months ago

meeting notes:

# Go SDK
# server info file contains:
version: x.x.x
minimum numaflow version: >= x2.x2.x2 # if this key-value doesnt exists then old SDK version, please update your SDK log

------------------------

# numaflow
version: z.z.z
# maintain this information for all languages in a matrix, yaml file (root dir?), parse yaml into a dict in Go code
minimum Go SDK version: y.y.y
if numaflow version 'latest' skip numaflow check as you already have latest changes

# check before grpc connection is established
if x.x.x < y.y.y  || if z.z.z < x2.x2.x2 # compatibility issue 
if x.x.x >= y.y.y && z.z.z >= x2.x2.x2 # compatible