search

studio-b12 / gowebdav

A golang WebDAV client library and command line tool.
BSD 3-Clause "New" or "Revised" License
309 stars 89 forks source link

Use seeker when available on request body #50

Closed zekroTJA closed 2 years ago

zekroTJA commented 2 years ago

Problem

I tried to upload a large file (~30GB) over night with on my Raspberry Pi using the CLI client. Therefore I opened up a Docker container with the binary and the file to upload inside and started the upload from there. I noticed that the upload always gets canceled after a few minutes with log output Killed.

The problem occurs because the contents of the file are written into a buffer on upload, which filled up the RAM of the Pi (4GB) which eventually caused Linux to kill the process.

https://github.com/studio-b12/gowebdav/blob/73a7f0bf3780929a467543d2c09b2886c26170e2/requests.go#L12-L16

I was also able to reproduce this on my Windows system.

Solution

I changed the req function so that it checks if the passed body Reader implements io.Seeker. If this is the case, the streams cursor will be reset to start before request which eliminates the requirement of the retry buffer. Otherwise, if the Reader is not seekable, the stream will be teed into the retry buffer.

Also, I've renamed the buffer ab to retryBuf for better understanding.

chripo commented 2 years ago

link #24 and #15 thank you!! this is a more or less know issue and the whole mess is related to authorization handling in some servers. my initial idea was to refactor this out into strategies and on top one that handles it without fuss. still a TO DO. :)