search

shieldproject / shield

A standalone system that can perform backup and restore functions for a wide variety of pluggable data systems
MIT License
367 stars 69 forks source link

[BUG] Outputs bigger than 15G are not being saved #700

Open JakubKC opened 3 years ago

JakubKC commented 3 years ago

Describe the bug Outputs bigger than 15G are not being saved by the s3 plugin on the s3 bucket (they just don't show there after) despite that it says: "upload complete; uploaded xxx bytes of data" and ends with 0 exit code.

To Reproduce Steps to reproduce the behavior:

  1. Create simple pipe in shell: 
    pv --stop-at-size -s 15G /dev/zero | \
/shield/plugins/s3 store -e "${SHIELD_STORE_ENDPOINT}"
  2. Run it and verify that it ends successfully, but there is no such file on s3 after.
  3. Change size to for example "-s 14990M" and run again. Verify that now there is a file.

Expected behavior Size should be limited only by s3 limits: https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

Screenshots If applicable, add screenshots to help explain your problem.

SHIELD versions (please complete the following information):