search

osmosis-labs / test-tube

Test tube for cosmos-sdk chains integration test, written in Rust!
Apache License 2.0
53 stars 31 forks source link

panic: unlinkat /tmp/.osmosis-test-tube-temp-3820962552/data: directory not empty #52

Closed Buckram123 closed 2 months ago

Buckram123 commented 4 months ago

I'm getting random panic: unlinkat /tmp/.osmosis-test-tube-temp-3820962552/data: directory not empty during tests and in CI: https://github.com/AbstractSDK/carrot-app/actions/runs/8879573463/job/24377599948?pr=84#step:8:919 The worst part that it's not consistent, restarting action just fixes it magically

Buckram123 commented 4 months ago

image

iboss-ptk commented 4 months ago

Could be a concurrency problem. Every new app creates tmp dir using os.MkdirTemp and then cleaned up after the end of it's lifetime by calling CleanUp which under the hood, calling os.RemoveAll(nodeHome) and that's probably what invoke syscall unlinkat.

Found a reference issue noting that other process might try to write to the same dir while it's cleaning up, but theoretically, rust compiler should prevent sharing the app instance and only drop after used unless there are bugs or you use some unsafe magic.

After digging into your code a bit and found that you use cw_orch which makes osmosis test app an Rc<RefCell<_>>. I suspect that there might be some miscounting happens. Would you mind strong_count the Rc where it make sense to track? I suspected that it dropped too early. If it's reproducible locally, that would be great.

also observed that the linked action failed at either install_on_sub_account or install_on_sub_account_create_position_on_install is it consistently these ones when it fails?

Buckram123 commented 2 months ago

Didn't happen again