VI Hangs when multiple client reading stream VIs are waiting and multiple Steam Ends occur. This used to work in 1.0.0.1.

ni / grpc-labview

gRPC client and server support for LabVIEW

MIT License

90 stars 62 forks source link

VI Hangs when multiple client reading stream VIs are waiting and multiple Steam Ends occur. This used to work in 1.0.0.1. #345

Closed NIbbuchanan closed 8 months ago

NIbbuchanan commented 9 months ago

Our application used to work with grpc 1.0.0.1, but I discovered there was a change between 1.0.0.3 and 1.0.0.4 that broke our application and caused it to hang. We are now using 1.2.0.1 (latest version available), which also results in a hang as well. When there are multiple client stream reads waiting for a message and the client streams all end in parallel, some of the reads never return. They used to all return when the streams were ended with either an error or the valid flag being false, but now a simple test VI that reproduces this issue just hangs indefinitely. A work around is to avoid reading in parallel and let each one fail/timeout sequentially when the streams are closed, but in some cases, the reads are spread across various locations so it's not trivial to coordinate when they run. I'd like to get a fix so the previous behavior works since a hang seems bad. Let me know if you need any additional information. Thanks!

GrpcHangBug.zip

AB#2645647

nischalks commented 9 months ago

@NIbbuchanan It would be great if you could move to the latest release https://github.com/ni/grpc-labview/releases/tag/v1.2.0.1 and see if it solves your problem. Thanks!

NIbbuchanan commented 9 months ago

I updated the original message because I forgot to mention we are using the latest 1.2.0.1, which is when I discovered this issue after we upgraded from 1.0.0.1. I installed older versions to determine when it broke to hopefully make it easier to track down the change that broke this behavior.

nischalks commented 8 months ago

@NIbbuchanan This is now fixed and released as part of https://github.com/ni/grpc-labview/releases/tag/v1.2.1.1

NIbbuchanan commented 8 months ago

I agree the latest export fixes this issue....Thanks! I did find a new issue with the vi.lib\gRPC\LabVIEW gRPC Library\Shared\Wait On Occurrence Wrapper.vi on the latest export:

It is not marked as re-entrant/in place so if a grpc call takes awhile, it blocks other calls from going through.
The separate compiled code flag is not checked, which is annoying when going between different contexts (ie. windows an RT) and this VI has to be saved each time. I'll follow a separate issue for this...it was causing our app to hang until I found this problem.

NIbbuchanan commented 6 months ago

@yash-ni @nischalks I am testing the latest 1.2.3.1 to verify behavior works well and I verified vi.lib\gRPC\LabVIEW gRPC Library\Shared\Wait On Occurrence Wrapper.vi now has the correct VI settings, but the issue raised in this problem is back. Could you please add this test VI to your suite so we check this behavior every release. Try taking the 1ms delay out of the top For Loop (it can happen with it as well, but it seems like a better stress test to remove it) and run 10 times to see if it hangs with the latest GRPC code. I found that it hung for me 5 out of 10 times.

Let me know if you want me to file a separate bug for this instead of commenting here. Not sure if you can just re-open this one.