numaproj / numaflow-python

Numaflow Python SDK
Apache License 2.0
53 stars 18 forks source link

feat: add support to configure max message size #23

Closed jy4096 closed 1 year ago

jy4096 commented 1 year ago

Signed-off-by: jyu6 juanlu_yu@intuit.com

https://github.com/numaproj/numaflow/issues/299

Tested using numaflow-go gRPC client and numaflow-python gPRC server.

Main branch: if receive something larger than the default 4 MB limit, we will see the following error.

failed to execute c.grpcClt.MapFn(): rpc error: code = ResourceExhausted desc = Received message larger than max (4194349 vs. 4194304)

Current branch: add the option to set a higher max message size limit. (socket path field is set for testing purpose)

grpc_server = UserDefinedFunctionServicer(map_handler, sock_path="/tmp/test.sock", max_message_size=1024*1025*4)

and the tests succeed.

=== RUN   Test_server_map
--- PASS: Test_server_map (0.31s)
=== RUN   Test_server_map/server_map
    --- PASS: Test_server_map/server_map (0.31s)
PASS

TODO

Need help on the codecov... thank you

codecov[bot] commented 1 year ago

Codecov Report

Merging #23 (d357590) into main (c3b1a80) will increase coverage by 0.14%. The diff coverage is 71.42%.

:exclamation: Current head d357590 differs from pull request most recent head 4c47e1d. Consider uploading reports for the commit 4c47e1d to get more accurate results

@@            Coverage Diff             @@
##             main      #23      +/-   ##
==========================================
+ Coverage   87.31%   87.45%   +0.14%     
==========================================
  Files           6        6              
  Lines         268      271       +3     
  Branches       13       13              
==========================================
+ Hits          234      237       +3     
  Misses         32       32              
  Partials        2        2              
Impacted Files Coverage Δ
pynumaflow/function/server.py 70.68% <66.66%> (+0.51%) :arrow_up:
pynumaflow/sink/server.py 67.30% <66.66%> (+0.64%) :arrow_up:
pynumaflow/_constants.py 100.00% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more