Tests running in parallel can still run into concurrency issues

nengo / nengo-loihi

Run Nengo models on Intel's Loihi chip

https://www.nengo.ai/nengo-loihi/

Other

35 stars 12 forks source link

Tests running in parallel can still run into concurrency issues #217

Closed tbekolay closed 5 years ago

tbekolay commented 5 years ago

In 70ecb8a475985846c81bc5e8059056042f855a34 we started running hardware tests in parallel, and made a quick patch to ensure the C file for the snip could be access by two threads at once. It seems that this can still fail so we should try to reproduce that issue and fix it.

arvoelke commented 5 years ago

Small update for posterity: to work-around this temporarily, we partially reverted the above commit (e9cd851b01b20b774d58c82a2ba775b531cf2bab) and switched to -n 1 py-test workers in #202. But we're still seeing a different related issue (os.chdir not changing the directory back in the hardware interface's __exit__, and then coverage.xml gets put into the wrong place) if -n is omitted altogether.

Still investigating why the directory is not being switched back. This happens consistently on the same test: nengo_loihi/hardware/tests/test_allocators.py::test_block_size.

arvoelke commented 5 years ago

The above problem is solved in #218. The larger issue described in e9cd851b01b20b774d58c82a2ba775b531cf2bab is still at hand.

drasmuss commented 5 years ago

This seems to have been working OK since https://github.com/nengo/nengo-loihi/commit/ee0ca71fc341685865d26404cc5745bda070e89e#diff-bb7e722a615e3b3171b86e0b952a51fc (hard to say for sure with concurrency issues, but it's at least a pretty rare issue which can be resolved by re-running the build if it ever comes up again).