numaproj / numaflow-go

Numaflow Golang SDK
Apache License 2.0
46 stars 11 forks source link

feat: use same server info for all map modes #137

Closed kohlisid closed 3 months ago

kohlisid commented 3 months ago

Fixes https://github.com/numaproj/numaflow/issues/1838

kohlisid commented 3 months ago

Map Stream with new map mode metadata

no annotations in spec

    - name: flatmap
      limits:
        readBatchSize: 1
      udf:
        container:
          image: quay.io/kohlisid/numaflow-go/map-flatmap-stream:stable

Map stream server with mapstream.sock

 kubectl logs simple-pl-go-flatmap-0-w5nqq -f
2024/07/16 21:28:04 starting the gRPC server with unix domain socket... /var/run/numaflow/mapstream.sock

Server info file - /var/run/numaflow/mapper-server-info

kubectl logs simple-pl-go-flatmap-0-w5nqq numa -f | grep mapper
2024/07/16 21:27:42 Server info file /var/run/numaflow/mapper-server-info is not ready...

 kubectl logs simple-pl-go-flatmap-0-w5nqq numa -f | grep Info  
2024/07/16 21:28:05 ServerInfo: &{uds go 1.2.0-rc4 v0.7.0-rc2 map[MAP_MODE:stream-map]}

kubectl logs simple-pl-go-flatmap-0-w5nqq numa -f | grep enabled
2024-07-16T21:28:05.294120902Z  INFO    numaflow.MapUDF-processor       udf/map_udf.go:150 
Map mode enabled: Stream Map    {"pipeline": "simple-pl-go", "vertex": "flatmap", "protocol": "uds-grpc-map-udf"}
kohlisid commented 3 months ago

Batch Map with new map mode metadata

no annotations in spec

    - name: go-split
      scale:
        min: 1
      udf:
        container:
          image: quay.io/kohlisid/numaflow-go/batch-map-flatmap:stable1
          imagePullPolicy: Always

Batch Map stream server with batchmap.sock

kubectl logs flatmap-go-split-0-0xcsa -f
2024/07/17 18:50:23 starting the gRPC server with unix domain socket... /var/run/numaflow/batchmap.sock

Server info file - /var/run/numaflow/mapper-server-info

kubectl logs flatmap-go-split-0-0xcsa numa -f | grep mapper 
2024/07/17 18:50:08 Server info file /var/run/numaflow/mapper-server-info is not ready...

 kubectl logs flatmap-go-split-0-0xcsa numa -f | grep ServerInfo:
2024/07/17 18:50:24 ServerInfo: &{uds go 1.2.0-rc4 v0.8.0 map[MAP_MODE:batch-map]}

 kubectl logs flatmap-go-split-0-0xcsa numa -f | grep enabled    
{"level":"info","ts":"2024-07-17T18:50:24.456998519Z","logger":"numaflow.MapUDF-processor","caller":"udf/map_udf.go:173","msg":"Map mode enabled: Batch Map","pipeline":"flatmap","vertex":"go-split","protocol":"uds-grpc-map-udf"}
kohlisid commented 3 months ago

Unary Map with new map mode metadata

no annotations in spec

    - name: go-split
      scale:
        min: 1
      udf:
        container:
          image: quay.io/kohlisid/numaflow-go/map-flatmap:stable
          imagePullPolicy: Always

Unary Map stream server with map.sock

kubectl logs flatmap-go-split-0-mvcdg -f
2024/07/17 18:57:42 starting the gRPC server with unix domain socket... /var/run/numaflow/map.sock

Server info file - /var/run/numaflow/mapper-server-info

kubectl logs flatmap-go-split-0-mvcdg -f numa | grep mapper
2024/07/17 18:57:27 Server info file /var/run/numaflow/mapper-server-info is not ready...

 kubectl logs flatmap-go-split-0-mvcdg -f numa | grep ServerInfo
2024/07/17 18:57:43 ServerInfo: &{uds go 1.2.0-rc4 v0.7.0-rc2 map[MAP_MODE:unary-map]}

 kubectl logs flatmap-go-split-0-mvcdg -f numa | grep enabled   
{"level":"info","ts":"2024-07-17T18:57:43.308422104Z","logger":"numaflow.MapUDF-processor","caller":"udf/map_udf.go:196","msg":"Map mode enabled: Unary Map","pipeline":"flatmap","vertex":"go-split","protocol":"uds-grpc-map-udf"}
kohlisid commented 3 months ago

Unary Map with no metadata (and new numaflow)

no annotations in spec

    - name: go-split
      scale:
        min: 1
      udf:
        container:
          image: quay.io/numaio/numaflow-go/map-flatmap:stable
          imagePullPolicy: Always

Unary Map stream server with map.sock

 kubectl logs flatmap-go-split-0-elw3l -f
2024/07/17 19:01:19 starting the gRPC server with unix domain socket... /var/run/numaflow/map.sock

Server info file - /var/run/numaflow/mapper-server-info

 kubectl logs flatmap-go-split-0-elw3l -f numa | grep mapper
2024/07/17 19:01:06 Server info file /var/run/numaflow/mapper-server-info is not ready...

 kubectl logs flatmap-go-split-0-elw3l -f numa | grep ServerInfo
2024/07/17 19:01:19 ServerInfo: &{uds go 1.2.0-rc4 v0.7.0-rc2 map[]}

 kubectl logs flatmap-go-split-0-elw3l -f numa | grep enabled   
{"level":"info","ts":"2024-07-17T19:01:19.426399748Z","logger":"numaflow.MapUDF-processor","caller":"udf/map_udf.go:196","msg":"Map mode enabled: Unary Map","pipeline":"flatmap","vertex":"go-split","protocol":"uds-grpc-map-udf"}