"invalid memory address or nil pointer dereference"

ghost commented 4 years ago

I've been encountering this issue lately on Windows 10. Has been working for a year without it. This is the output I get in verbose mode:

butler.exe diff \\server\SomeFolder \\server\SomeOtherFolder \\server\patch.pwr -v
∙ Hashing \\server\SomeFolder
▐░░░░░░░░░░░░░░░░░░░░▌   0.00%                                                  panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x48 pc=0x880a00]

goroutine 1 [running]:
github.com/itchio/wharf/pwr.ComputeSignatureToWriter(0x121d800, 0xc00002a118, 0x0, 0x121dc00, 0xc000128100, 0xc000737260, 0xc00012c120, 0x0, 0x0)
        C:/Users/amos/go/pkg/mod/github.com/itchio/wharf@v0.0.0-20190703124244-659fddc29012/pwr/sign.go:54 +0x120
github.com/itchio/wharf/pwr.ComputeSignature(0x121d800, 0xc00002a118, 0x0, 0x121dc00, 0xc000128100, 0xc000737260, 0x0, 0x11efc40, 0x8, 0x20, ...)
        C:/Users/amos/go/pkg/mod/github.com/itchio/wharf@v0.0.0-20190703124244-659fddc29012/pwr/sign.go:30 +0xcb
github.com/itchio/butler/cmd/diff.Do(0xc00029eec0, 0x0, 0x0)
        C:/Multi-Runner/builds/58469115/0/itchio/butler/cmd/diff/diff.go:134 +0x18ba
github.com/itchio/butler/cmd/diff.do(0xc000170900)
        C:/Multi-Runner/builds/58469115/0/itchio/butler/cmd/diff/diff.go:61 +0xdf
main.doMain(0xc00003a090, 0x5, 0x7)
        C:/Multi-Runner/builds/58469115/0/itchio/butler/main.go:264 +0xb66
main.main()
        C:/Multi-Runner/builds/58469115/0/itchio/butler/main.go:136 +0x69

Tried reinstalling Butler (v15.17.0), same error.
It works on other content though that's sitting on \some-other-server.
I checked and the machine does have read and write access to the relevant places in \server.
Tried running the same patch from several machines on the same network, same problem everywhere. All machines have sufficient ram and disk space.

fasterthanlime commented 4 years ago

I see v15.17.0 is missing some error handling that would definitely cause a nil pointer dereference there.

Can you try one of the beta (*-head) builds from here: https://broth.itch.ovh/butler ?

You'd at least have the reason why walking either directory failed.

ghost commented 4 years ago

Thank you so much. I'm so dumb I didn't think to use one of the beta versions, my bad!

Indeed now I saw the error, which is that one of the paths was too long for Windows (261 characters instead of <=260...):

CreateFile <path-to-some-very-long-filename>: The system cannot find the path specified.
github.com/itchio/lake/tlc.WalkDir.func1.1
        C:/Users/amos/go/pkg/mod/github.com/itchio/lake@v0.0.0-20200305150023-cc4284ec2b2a/tlc/walk.go:231
path/filepath.walk
        c:/go/src/path/filepath/path.go:380
path/filepath.walk
        c:/go/src/path/filepath/path.go:384
path/filepath.walk
        c:/go/src/path/filepath/path.go:384
path/filepath.walk
        c:/go/src/path/filepath/path.go:384
path/filepath.walk
        c:/go/src/path/filepath/path.go:384
path/filepath.walk
        c:/go/src/path/filepath/path.go:384
path/filepath.walk
        c:/go/src/path/filepath/path.go:384
path/filepath.walk
        c:/go/src/path/filepath/path.go:384
path/filepath.walk
        c:/go/src/path/filepath/path.go:384
path/filepath.walk
        c:/go/src/path/filepath/path.go:384
path/filepath.Walk
        c:/go/src/path/filepath/path.go:406
github.com/itchio/lake/tlc.WalkDir
        C:/Users/amos/go/pkg/mod/github.com/itchio/lake@v0.0.0-20200305150023-cc4284ec2b2a/tlc/walk.go:342
github.com/itchio/lake/tlc.WalkAny
        C:/Users/amos/go/pkg/mod/github.com/itchio/lake@v0.0.0-20200305150023-cc4284ec2b2a/tlc/walk.go:168
github.com/itchio/butler/cmd/diff.Do
        C:/msys64/home/amos/gitlab/builds/eJ5Rnxef/1/itchio/butler/cmd/diff/diff.go:116
github.com/itchio/butler/cmd/diff.do
        C:/msys64/home/amos/gitlab/builds/eJ5Rnxef/1/itchio/butler/cmd/diff/diff.go:60
main.doMain
        C:/msys64/home/amos/gitlab/builds/eJ5Rnxef/1/itchio/butler/main.go:256
main.main
        C:/msys64/home/amos/gitlab/builds/eJ5Rnxef/1/itchio/butler/main.go:132
runtime.main
        c:/go/src/runtime/proc.go:203
runtime.goexit
        c:/go/src/runtime/asm_amd64.s:1373
github.com/itchio/lake/tlc.WalkDir
        C:/Users/amos/go/pkg/mod/github.com/itchio/lake@v0.0.0-20200305150023-cc4284ec2b2a/tlc/walk.go:344
github.com/itchio/lake/tlc.WalkAny
        C:/Users/amos/go/pkg/mod/github.com/itchio/lake@v0.0.0-20200305150023-cc4284ec2b2a/tlc/walk.go:168
github.com/itchio/butler/cmd/diff.Do
        C:/msys64/home/amos/gitlab/builds/eJ5Rnxef/1/itchio/butler/cmd/diff/diff.go:116
github.com/itchio/butler/cmd/diff.do
        C:/msys64/home/amos/gitlab/builds/eJ5Rnxef/1/itchio/butler/cmd/diff/diff.go:60
main.doMain
        C:/msys64/home/amos/gitlab/builds/eJ5Rnxef/1/itchio/butler/main.go:256
main.main
        C:/msys64/home/amos/gitlab/builds/eJ5Rnxef/1/itchio/butler/main.go:132
runtime.main
        c:/go/src/runtime/proc.go:203
runtime.goexit
        c:/go/src/runtime/asm_amd64.s:1373

I removed the path size limit on the Windows 10 Entreprise machine, rebooted it, and the issue is still there, it seems butler doesn't support these path lengths. Anything you plan to do on the short term for this? :)

fasterthanlime commented 4 years ago

Ah, MAX_PATH, good times!

I'm trying to see what the current behavior is, by creating a file whose complete path is 270 characters. It works if I give an absolute path to diff, but not if I give a relative path:

C://tmp
❯ ls ./subdir
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.txt

C://tmp
❯ butler diff C:/tmp/subdir C:/tmp/subdir2
∙ Hashing C:/tmp/subdir
√ 16 B (1 files, 0 dirs, 0 symlinks) @ 16.09 KiB/s

∙ Diffing C:/tmp/subdir2
√ 0 B (0 files, 0 dirs, 0 symlinks) @ 0 B/s

√ Re-used NaN% of old, added 0 B fresh data
√ 78 B patch (+Inf% of the full size) in 1.0044ms

C://tmp
❯ butler diff ./subdir C:/tmp/subdir2
∙ Hashing ./subdir
computing target signature: open subdir\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.txt: The system cannot find the path specified.

I'll keep investigating some more..

fasterthanlime commented 4 years ago

@dtaralla-compulsion I think you should be able to work around that issue by using UNC paths, see this article: https://blog.klauspost.com/long-windows-paths-unc-paths-in-go/

But I'm still curious if adding "long path awareness" to butler's manifest would fix it.

ghost commented 4 years ago

Hmm that could be interesting to test, to change the manifest. But it's strange that it's not the default for new apps. I did tried to use the UNC server paths (like \?\UNC\server\share\dir) for the diff source and target on the commandline but to no avail (it says The system cannot find the path specified). For now I've reduced my path lengths, but indeed a more permanent fix would be appreciated 🙏 Thanks for your support so far :)

fasterthanlime commented 4 years ago

@dtaralla-compulsion I just deployed https://github.com/itchio/butler/commit/552133ed1a9189518522b70241f08926ffb7ef9f which includes a manifest that opts into long path support - it seems to work for me:

Can you confirm it works with long paths for you too?

ghost commented 4 years ago

Yes it works now, perfect !! Thank you so much :)

itchio / butler

"invalid memory address or nil pointer dereference" #208