search

grpc / grpc

The C based gRPC (C++, Python, Ruby, Objective-C, PHP, C#)
https://grpc.io
Apache License 2.0
42.04k stars 10.57k forks source link

Significant test flakiness after upgrading to grpcio 1.66.0 #37710

Open tvalentyn opened 2 months ago

tvalentyn commented 2 months ago

What version of gRPC and what language are you using?

1.66.1

What operating system (Linux, Windows,...) and version?

Linux, varies

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.8

What did you do?

In https://github.com/apache/beam/issues/32324 we have a test suite of ~60 tests. After upgrading to grpc 1.66.x, the suite became perma-red with random tests failing.

The symptoms are such that two processes that are expected to connect to each other over gprc, don't connect. In one case we see errors like: 

control.Recv failed\n\tcaused by:\nrpc error: code = Unavailable desc = error reading from server: EOF\n'

In another case, test suite gets stuck and there are many logs like 

2466 2024-09-12T19:56:32.0179123Z I0000 00:00:1726170991.935883   56889 fork_posix.cc:77] Other threads are currently calling into gRPC, skipping fork() handlers during stuckness,

but we also see the latter logs in successful runs.

In some cases, the error might be swallowed within our processes and we just see the test failure.

Setting the upper bound to grpcio<1.66.0 appears to resolve the issue.

What did you expect to see?

What did you see instead?

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

See TROUBLESHOOTING.md for how to diagnose problems better.

Anything else we should know about your project / environment?

pratikmallya commented 2 months ago

Also seeing the same issue. If there was a way to configure the logger to not write to stderr, that would also work. We are seeing this issues in 1.65.5 also, and had to roll back to 1.64.1.

tvalentyn commented 2 months ago

@pratikmallya what is the issue you see on 1.65.5? It might be a different one, since my issue has been resolved after a downgrade to 1.65.5 https://github.com/apache/beam/pull/32449

XuanWang-Amos commented 2 months ago

Hi, thanks for reporting this, can you tell us more about this issue? Like what's gRPC Pythons role in the tests?

And also is it possible for us to reproduce the issue without using testing framework from Beam?

tvalentyn commented 2 months ago

Like what's gRPC Pythons role in the tests

Inter-process communication.

And also is it possible for us to reproduce the issue without using testing framework from Beam?

I don't have better a repro at this time.

florentinchaussoy commented 2 months ago

If it helps, we collected new logs after upgrading to 1.66. We downgraded to 1.64.1, and the logs, along with our issues, have disappeared. Here are the logs:

I0000 00:00:1726678172.728387      33 call_credentials.c:168] GRPC_PHP: call credentials plugin function - begin
I0000 00:00:1726678172.728472      33 call_credentials.c:171] GRPC_PHP: call credentials plugin function - end
sourabhsinghs commented 1 month ago

@florentinchaussoy Thanks for sharing the logs. but these logs are not enough to debug this, we'll need steps to repro this issue.