spf13 / afero

A FileSystem Abstraction System for Go
Apache License 2.0
5.8k stars 498 forks source link

MemMapFS: missing files after write multi files in concurrency #361

Closed sko00o closed 1 year ago

sko00o commented 2 years ago

I have used afero.MemMapFs for filesystem mock test in my projects. But I got some unexpected test results.

I want to list all files in target directory that I have wrote multi files in concurrency before. Occasionally some files are not returned by Readdir function. Also I had the same problem calling Walk function on this directory.

Here is a test to reproduce my problem. https://github.com/sko00o/afero-test/blob/main/memmap/memmap_test.go#L42

failure result:

❯ go test -race -run ^TestListFilesAfterConcurrencyFileWrite$ ./... -test.count 1
--- FAIL: TestListFilesAfterConcurrencyFileWrite (4.39s)
    memmap_test.go:99: 
                Error Trace:    memmap_test.go:99
                Error:          Not equal: 
                                expected: 10000
                                actual  : 9995
                Test:           TestListFilesAfterConcurrencyFileWrite
                Messages:       missing files
FAIL
FAIL    github.com/sko00o/afero-test/memmap     4.677s
FAIL

I notice another issue #298 that found MemMapFs.Mkdir have problem when call in concurrency. In my opinion, we need a double check here before we create new directory.


So I send a PR for this.

sko00o commented 1 year ago

Ping @bep

Please check this issue.

For quick reproduce: https://go.dev/play/p/3xkOSEBVoNS

bep commented 1 year ago

Your failing test case looks correct to me (as in, it should not fail), which comes a little as a surprise to mek as I use this pretty concurrently myself ... It's not obvious to me why, though.

I also agree about the Mkdir bug, assuming that that's how most sane real file systems work?