search

Azure / go-ntlmssp

NTLM/Negotiate authentication over HTTP
MIT License
189 stars 67 forks source link

ntlmssp.Negotiator is storing the entire body in memory and causing out of memory errors #44

Open ncw opened 11 months ago

ncw commented 11 months ago

I had a bug report in rclone which was complaining about huge memory use.

I tracked it down from the log to this

github.com/Azure/go-ntlmssp.Negotiator.RoundTrip({{0x27ff220?, 0xc0006ce230?}}, 0xc0009d6000)
    github.com/Azure/go-ntlmssp@v0.0.0-20221128193559-754e69321358/negotiator.go:52 +0x2bd fp=0xc0007aac68 sp=0xc0007aa990 pc=0x191645d
github.com/Azure/go-ntlmssp.(*Negotiator).RoundTrip(0xc0007aad48?, 0x418f48?)
    <autogenerated>:1 +0x31 fp=0xc0007aac90 sp=0xc0007aac68 pc=0x1918a31

Which points to this bit of code

https://github.com/Azure/go-ntlmssp/blob/754e69321358ada85ce213a4ec971d3e4d1bfdf7/negotiator.go#L49-L59

Where we see, yes this does load the entire body into memory.

The user is complaining about out of memory errors when trying to upload a very large file (10GB).

This might be an error we are making in the rclone code - I didn't write this bit of code so maybe we are using the library wrong, but loading a 10GB body into memory is a bit unfriendly!

Here is the rclone code if anyone wants to take a look

https://github.com/rclone/rclone/blob/1ebbc74f1d62457aea660b9e4378cc085c0dd25e/backend/webdav/webdav.go#L462