error: 400: "Part number must be an integer between 1 and 10000, inclusive"

rlmcpherson / s3gof3r

Fast, concurrent, streaming access to Amazon S3, including gof3r, a CLI. http://godoc.org/github.com/rlmcpherson/s3gof3r

MIT License

1.14k stars 180 forks source link

error: 400: "Part number must be an integer between 1 and 10000, inclusive" #150

Closed zam6ak closed 3 years ago

zam6ak commented 3 years ago

I am trying to upload a large file (1.5 TB) and after about 18 hours I get an error. Here is the entire command

time /backup1/gof3r put --bucket myappdb123-backups --key backups/myapp_2020-12-13.tar --path /backup1/myapp_2020-12-13.tar
gof3r error: 400: "Part number must be an integer between 1 and 10000, inclusive"

real    1082m6.438s
user    293m46.376s
sys     30m27.823s

I am using gof3r version 0.5.0

zam6ak commented 3 years ago

I added -concurrency 20 --partsize 50000000 along with --debug and after 20 hours I was able to upload the file... While uploading I saw that part size was regularly doubling also

2020/12/17 08:48:32 part size doubled to 100000000
...
2020/12/17 12:05:54 part size doubled to 200000000   
...
2020/12/18 00:52:50 part size doubled to 800000000

I am not sure if gof3r could, based on file size, start off with appropriate part size to avoid exceeding the part number limit..