feature: gRPC

[jy4096]

python notes: run poetry shell before starting the development gPRC notes:

1. First is the proto file: The proto file is identical to the one in numaflow-go https://github.com/numaproj/numaflow-go/blob/main/pkg/apis/proto/function/v1/udfunction.proto. I copied and pasted it. (Copy the https://github.com/numaproj/numaflow-go/blob/main/pkg/apis/proto folder and paste at the repo root, rename it to protos)

2. Step two: To generate the python proto code: from code repo root, run the following command.

   python -m grpc_tools.protoc -I./protos/function/v1/ --python_out=pynumaflow/function/ --grpc_python_out=pynumaflow/function/  protos/function/v1/udfunction.proto

   move the generated file to the generated folder NOTE: need to manually add from pynumaflow.function.generated import udfunction_pb2 as udfunction__pb2 in the generated udfunction_grpc_pb2 file. from pynumaflow.function.generated is needed to import udfunction_pb2 as mentioned in https://groups.google.com/g/grpc-io/c/hpPnAGY0ksQ. server.py also imports from pynumaflow.function.generated import udfunction_pb2 as udfunction__pb2 If we find a proper fix we can replace the current from.. approach...

3. Need to manually add a udfunction_pb2.pyi file for the data types defined in the proto file.

4. Next is to add new datatypes in _dtypes.go (not sure if this is the correct file to put new data types)

5. Implement the server.py

6. Add an udfunction example: forward message

7. Add tests

I refactor the code so that the function is using gRPC and sink is still using http. Also need help with the code style/structure/design pattern etc.. not confident in those things

Thank you!

ref: python gRPC testing pkg https://grpc.github.io/grpc/python/grpc_testing.html python gRPC unit test example.. https://blog.masuqat.net/2018/04/16/grpc-python-unittest/ async graceful shutdown example https://github.com/lidizheng/grpc/blob/master/examples/python/helloworld/async_greeter_server_with_graceful_shutdown.py

[codecov[bot]]

Codecov Report

Merging #4 (40a2d79) into main (5de24c4) will increase coverage by 3.88%. The diff coverage is 85.83%.

@@            Coverage Diff             @@
##             main       #4      +/-   ##
==========================================
+ Coverage   85.43%   89.31%   +3.88%     
==========================================
  Files          12       13       +1     
  Lines         309      337      +28     
  Branches       36       37       +1     
==========================================
+ Hits          264      301      +37     
+ Misses         40       34       -6     
+ Partials        5        2       -3     

Impacted Files	Coverage Δ
pynumaflow/sink/decoder.py	100.00% <ø> (ø)
pynumaflow/sink/encoder.py	90.00% <ø> (ø)
pynumaflow/function/server.py	68.51% <68.51%> (ø)
pynumaflow/_constants.py	100.00% <100.00%> (ø)
pynumaflow/function/__init__.py	100.00% <100.00%> (ø)
pynumaflow/function/_dtypes.py	97.95% <100.00%> (+4.73%)	:arrow_up:
pynumaflow/sink/_dtypes.py	95.12% <100.00%> (ø)
pynumaflow/sink/handler.py	80.88% <100.00%> (ø)
pynumaflow/types.py	100.00% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[jy4096]

Need help on the repo structure... @vigith

[vigith]

Can we improve the coverage?

@ab93 can you give her some tips on how? I think @chromevoid needs some guidance there.

[vigith]

@chromevoid update the version as major/minor upgrade. @vigith this should be major right? Or should minor be okay?

I am thinking we will do minor till we reach official 1.x

[jy4096]

@chromevoid update the version as major/minor upgrade. @vigith this should be major right? Or should minor be okay?

where to update the version? which file?

toml??

[tool.poetry]
name = "pynumaflow"
version = "0.1.1"

[vigith]

where to update the version? which file?

toml??

run poetry minor update ?

[ab93]

poetry version minor

[jy4096]

poetry version minor

~minor or patch?...~

~poetry version minor got:~ ~Bumping version from 0.1.1 to 0.2.0~

Done