fsouza / fake-gcs-server

Google Cloud Storage emulator & testing library.
https://pkg.go.dev/github.com/fsouza/fake-gcs-server/fakestorage?tab=doc
BSD 2-Clause "Simplified" License
1.07k stars 210 forks source link

Minimum chunk size not enforced #1080

Open ianhomer opened 1 year ago

ianhomer commented 1 year ago

We see a different behaviour on fake GCS server when chunk size is below the minimum of 262144 - https://cloud.google.com/storage/docs/performing-resumable-uploads

On fake GCS

curl -i -X PUT --data-binary "@$FILE_NAME" \
              -H "Content-Length: 500"               \
              -H "Content-Range: bytes 0-499/700000"    \
              "$UPLOAD_URL/o?uploadType=resumable&name=$FILE_NAME&upload_id=$UPLOAD_ID"

accepts the partial upload and responds status code 308 with

{"kind":"storage#object","name":"my-upload-1.txt",... }

On real GCS

curl --http1.1 -v -i -X PUT --data-binary "@$FILE_NAME" \
              -H "Content-Length: 500"               \
              -H "Content-Range: bytes 0-499/700000"    \
              "$UPLOAD_URL/o?uploadType=resumable&name=$FILE_NAME&upload_id=$UPLOAD_ID"

we get a 400 with response

Invalid request.  The number of bytes uploaded is required to be equal or greater than 262144, 
except for the final request (it's recommended to be the exact multiple of 262144).  
The received request contained 500 bytes, which does not meet this requirement.

When chunk size is multiple of 262144 both behave in a similar manner.

The small chunk size is handy for some test scenarios, however since it does not match the behaviour of real GCS, it masks a failure scenario that would be good to test with fake gcs.

Should fake GCS mirror this real GCS behaviour where a minimum chunk size is enforced?

fsouza commented 1 year ago

Hey @ianhomer, thanks for reporting. We should definitely mirror GCS' behavior.