vbauerster / mpb

multi progress bar for Go cli applications
The Unlicense
2.29k stars 123 forks source link

Add support for ReadSeeker for uploads to s3 #114

Open mskonovalov opened 1 year ago

mskonovalov commented 1 year ago

Unfortunately s3 sdk uses ReadSeeker for uploads. So in this case it is tricky to wrap a file that you trying to upload: curremt API only allows ReadCloser but not ReadSeeker.

vbauerster commented 1 year ago

I never used s3 sdk and probably will not be. IMHO it's horrible approach to use ReadSeeker for uploads. Nevertheless you can wrap ReadCloser returned by (*Bar).ProxyReader and implement whatever you need.

mskonovalov commented 1 year ago

@vbauerster I suspect they use ReadSeeker to be able to re-upload specific parts of the file. That's exactly what I do atm:

ReadSeeker{
    file,
    p.ProxyReader(file),
}

but I suspect this will show wrong results in case you try to re-upload the specific part of the file, it easily can get higher 100%

vbauerster commented 1 year ago

but I suspect this will show wrong results in case you try to re-upload the specific part of the file, it easily can get higher 100%

That's correct. There is SetCurrent which may help to keep bar in correct state after seek has been called. If you come up with working implementation, PR is welcome. I'm not able to test against s3 api.

mskonovalov commented 1 year ago

So I started to test and unfortunately it doesn't work completely: AWS SDK reads body multiple times to calculate different hashes and so on :( Because the reader is seekable they can do it without issues. Don't have ideas yet how to solve it

mskonovalov commented 1 year ago

Ok, I was able to come up with some weird naive solution (although it is not working properly in some circumstances):


type ProxyReadSeeker struct {
    F           io.ReadSeekCloser
    ProxyReader io.ReadCloser
    Bar         *mpb.Bar
}

func (rs *ProxyReadSeeker) Read(p []byte) (n int, err error) {
    return rs.ProxyReader.Read(p)
}

func (rs *ProxyReadSeeker) Close() error {
    return rs.F.Close()
}

func (rs *ProxyReadSeeker) Seek(offset int64, whence int) (int64, error) {
    if whence == io.SeekStart {
        rs.Bar.SetCurrent(offset)
    }
    return rs.F.Seek(offset, whence)
}

Not sure if it is useful for anyone else, but it seems to work. Although you cannot rely on automatic completion and have to disable it and complete the bar manually.