Open josiah-tesfu opened 10 months ago
@josiah-tesfu include your test code please for both shared sdk and protected.
@sj608 this is a reproducible issue now. @josiah-tesfu has test codes (which should be added to this ticket soon) that can produce the error consistently.
Non-shared instances (all tests pass)
regarding this test, I just want to add that I think the reason that it is passing is not because the sdk is non-shared but because sdk init is slow the threads do not have a good chance to overlap
@sjpark608 to make all tests fail just increase number of threads from 5 to 30 here:
threads = [PropagatingThread(target=set_rtd_client) for _ in range(5)]
This comment is to sum up our brief discussion about thread safety of SDK
bad file descriptor error
which indicates accessing of a file that's closed.reference_1, reference_2, spidev.h source, reference_3, i2c-dev.h source
When calling an SDK function with multiple threads which use a shared SDK instance, one of the following exceptions occurs:
1.
2.
This specific stack trace arises from calling
set_rtd
in the ADC Service with multiple threads using a shared SDK instance, but other functions called also cause the same exceptions to be raised. The bug seems to arise when multiple SDK instances use a shared hardware resource -- in this case I2C and SPI devices.When testing for this exception, I found that when the threads each have their own instance of the SDK the tests pass. The tests fail if the threads share an SDK instance, with the rate of failure increasing with more threads. Additionally, the tests pass if the threads share an SDK instance but the SDK function call is protected with a lock.
Test code to reproduce the issue:
Shared SDK instance (subset of the tests fail):
Shared SDK instance with lock (all tests pass):
Non-shared instances (all tests pass):