S3 upload fails with "Need to rewind the stream" error

[westcool10]

I encountered the following error while playing with zappa: Packaging project as zip... Uploading zip (15.1MiB)... 0%| | 0.00/15.9M [00:00<?, ?it/s]Failed to upload zappa-beta-1461586412.zip to scoop-lambda/zappa-beta-1461586412.zip: Need to rewind the stream <boto3.s3.transfer.ReadFileChunk object at 0x1070a2690>, but stream is not seekable.

I manually tried to upload a file to the same bucket using AWS CLI tool and it worked without any issue. Is this a known problem?

[codingjoe]

Same here 😞

[Miserlou]

Hm, can you check if this still happens in version-minus-point-one? I merged in a PR yesterday, perhaps that's causing this.

[codingjoe]

yes

Installing collected packages: zappa
  Found existing installation: zappa 0.17.5
    Uninstalling zappa-0.17.5:
      Successfully uninstalled zappa-0.17.5
Successfully installed zappa-0.17.4
(env)Johannness-MacBook-Air-2:lex-zapa johannes$ zappa deploy prod
Updating zappa-permissions policy on ZappaLambdaExecution IAM Role.
Packaging project as zip...
Uploading zip (13.1MiB)...
  0%|                                                                                                                                                                                                                                                                                                                               | 0.00/13.8M [00:00<?, ?it/s]Failed to upload lex-zapa-prod-1461607365.zip to lex-zappa/lex-zapa-prod-1461607365.zip: Need to rewind the stream <boto3.s3.transfer.ReadFileChunk object at 0x107a1cf50>, but stream is not seekable.

An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist.

[codingjoe]

works on 0.16.0

[bennybauer]

I had this issue today too, but apparently the problem was with aws_region. Make sure you haven't already created the bucket, and if you did, make sure it's in the region you expect it to be.

[Miserlou]

What was the problem with aws_region? It should warn about that.

I'm worried this is related to dependencies. It also shows the problems of using placebo for our testing! :[

[Miserlou]

Not able to reproduce with latest code and deps.

Can you please give the command you used (deploy? update?) and your zappa_settings so that I can diagnose?

[codingjoe]

deploy

[codingjoe]

@bennybauer isn't S3 region independent?

[Miserlou]

Hm, can't reproduce. Are you using any other zappa_settings?

[westcool10]

I also tried with 0.16.0 and verified it working. This is my zappa_settings: { "beta": { "aws_region": "ap-northeast-1", "s3_bucket": "scoop-lambda", "app_function": "scoop.api.server.app" } }

[westcool10]

Hmm. Interesting. I tried from version 1.17.0 to 1.17.5 and it works with all versions now. I can't repro it anymore.

[Miserlou]

Weird. Maybe it was related to an AWS outage?

[Miserlou]

I'm going to leave this ticket open until we can resolve the source of the issue, but I hope that things are working well for you now.

Thanks for using Zappa, please let me know if there's anything else we can do to improve it!

[bennybauer]

@codingjoe S3 has regions and us-east-1 is the default. You can choose it when you create the bucket. More here: http://aws.amazon.com/s3/faqs/#regions

[Miserlou]

@bennybauer and @codingjoe - can you confirm that this problem has resolved itself?

[bennybauer]

@Miserlou I don't know if it was related, but it started working only when region in zappa_settings.json was set to 'us-east-1'. I tried to reproduce it now, and I see that if I enter a different region it still creates the bucket in US Standard (us-east-1) but then fails on (same error mentioned by @codingjoe): An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: Error occurred while GetObject. S3 Error Code: PermanentRedirect. S3 Error Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. It seems like a different issue though.

[bennybauer]

@Miserlou I was able to reproduce:

1. Go to AWS S3 console and create the bucket, choose other region than us-east-1 (I chose eu-west-1)
2. In zappa_settings.json: set "aws_region": "eu-west-1"
3. deploy I get:

Packaging project as zip...
Uploading zip (6.8MiB)...
  0%|                                                                                                                              | 0.00/7.08M [00:00<?, ?it/s]Need to rewind the stream <boto3.s3.transfer.ReadFileChunk object at 0x109838b50>, but stream is not seekable.

An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist.

[Miserlou]

Ah! Excellent!

Okay, great, thanks for reporting steps. Looks like it's not respecting the aws_region setting when creating the Lambda function.

[Miserlou]

Do you have multiple profiles in your AWS credentials file?

[bennybauer]

Yes :)

[Miserlou]

Okay, this one ended up being super weird.

I honestly still don't know what's happening here since here, but I've opened an upstream ticket to hopefully explain this.

I haven't actually fixed the bug, but I have implemented a workaround. The drawback is that there won't be a progress bar for some S3 regions, but that's not the end of the world.

Fix pushed in 0.17.6.

[codingjoe]

For the record, I had only one profile in AWS config while executing the command.