search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
9.06k stars 1.87k forks source link

boto3 1.9.199 polly synthesize_speech doesn't accept Engine parameter #2060

Closed Jun711 closed 4 years ago

Jun711 commented 5 years ago

Issue

I was trying Polly neural voices. On Boto3 Polly document, synthesize_speech method already includes Engine parameter.

Screen Shot 2019-08-01 at 11 11 59 AM

However, when I used it, it complains about Unknown parameter. Unknown parameter in input: "Engine", must be one of: LexiconNames, OutputFormat, SampleRate, SpeechMarkTypes, Text, TextType, VoiceId, LanguageCode

Steps

I tried using the following code

print('boto3 ver: ', boto3.__version__)
pollyClient = boto3.client('polly')
try:
    response = pollyClient.synthesize_speech(
                                        Engine='neural',
                                        OutputFormat='mp3',
                                        Text='test',
                                        TextType='ssml',
                                        VoiceId='Matthew'
                                    )
except Exception as e:
    print('synthesize_speech exception: ', e)
boto3 ver:  1.9.199
synthesize_speech exception:  Parameter validation failed:
Unknown parameter in input: "Engine", must be one of: LexiconNames, OutputFormat,
SampleRate, SpeechMarkTypes, Text, TextType, VoiceId, LanguageCode

Does version 1.9.199 support neural voices?

swetashre commented 5 years ago

@Jun711 - Thank you for your post. I am not able to reproduce the issue. The error you are getting is coming from botocore. When a command is executed in boto3, botocore validates the parameter with service2.json file and if mismatch occurs then you will get parameter validation failed error.

Please upgrade botocore version to 1.12.199 which is the latest version.

But if you are still getting the error even after upgrading to the latest version then please provide me with the debug log. You can enable log by adding boto3.set_stream_logger('') to your code.

Hope it helps and please let me know if you have any questions.

Jun711 commented 5 years ago

Thank you. @swetashre

Note that I am using it on using Cloud9 IDE and using run local function which call sam local invoke if I am not mistaken. Maybe runtime botocore is not updated correctly. I installed boto3 and botocore into a local folder. When it doesn't import local botocore, runtime botocore version is 1.12.42

Question: Does Lambda runtime include boto3 1.9.199 and botocore 1.12.199 or I have package them? 

from boto3Module import boto3
from botocoreModule import botocore

print('boto3 ver: ', boto3.__version__)
print('botocore ver: ', botocore.__version__)
boto3.set_stream_logger('')
pollyClient = boto3.client('polly')
try:
    response = pollyClient.synthesize_speech(
                                        Engine='neural',
                                        OutputFormat='mp3',
                                        Text='<speak>test</speak>',
                                        TextType='ssml',
                                        VoiceId='Matthew'
                                    )
except Exception as e:
    print('synthesize_speech exception: ', e)

log

boto3 ver:  1.9.199
botocore ver:  1.12.199
[DEBUG] 2019-08-02T00:49:30.254Z    52fdfc07-2182-154f-163f-5f0f9a621d72    Event before-parameter-build.polly.SynthesizeSpeech: calling handler <function generate_idempotent_uuid at 0x7f8c6cf7c7b8>
2019-08-02 00:49:30,254 botocore.hooks [DEBUG] Event before-parameter-build.polly.SynthesizeSpeech: calling handler <function generate_idempotent_uuid at 0x7f8c6cf7c7b8>
synthesize_speech exception:  Parameter validation failed:
Unknown parameter in input: "Engine", must be one of: LexiconNames, OutputFormat, SampleRate, SpeechMarkTypes, Text, TextType, VoiceId, LanguageCode
swetashre commented 5 years ago

@Jun711 - Lambda runtime by default uses boto3 version 1.9.42 and botocore version 1.12.42. You should be able to package and import any version of the SDK as you would for any other project within a Lambda function. These are the preinstalled versions of the SDKs: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

Are you still getting the error even after updating the botocore version ? If yes, please check that when you execute your code it is taking the newer version of botocore during runtime.

Jun711 commented 5 years ago

@swetashre yes, it still happens during lambda runtime. You can see the code and log I included in the previous reply. It works locally with boto3 1.9.199 and botocore 1.12.199 though.

However, even though I include botocore 1.12.199 in a folder and import it using from botocoreModule import botocore, synthesize_speech is still called using older botocore version and thus, I still get the same error. How do I make it work in lambda runtime? Thanks

swetashre commented 5 years ago

@Jun711 - Here are some of the link which demonstrates how to add package to a lambda function:

https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html#python-package-dependencies

https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html#python-package-venv

If you don't want to package a more recent boto3 version with your function, you can download boto3 with each invocation of the Lambda. Remember that /tmp/ is the directory that Lambda will allow you to download to, so you can use this to temporarily download boto3:

import sys
from pip._internal import main

main(['install', '-I', '-q', 'boto3', '--target', '/tmp/', '--no-cache-dir', '--disable-pip-version-check'])
sys.path.insert(0,'/tmp/')

import boto3
import botocore

def lambda_handler(event, context):
   print(boto3.__version__)
   print(botocore.__version__)
   pollyClient = boto3.client('polly')
   response = pollyClient.synthesize_speech(Engine='neural',OutputFormat='mp3',Text='<speak>test</speak>',TextType='ssml',VoiceId='Matthew')

Hope it helps.

Jun711 commented 5 years ago

@swetashre I installed botocore and boto3 using pip install botocore --target ./botocoreModule and pip install boto3 --target ./boto3Module and imported using

from boto3Module import boto3
from botocoreModule import botocore

print('boto3 ver: ', boto3.__version__) # boto3 ver:  1.9.199
print('botocore ver: ', botocore.__version__) # botocore ver:  1.12.199

since running locally on cloud9 doesn't work, I will try these 1) I will deploy to Lambda and run via api gateway and see if it works. 2) If not, I will try the method that downloads and install in /tmp/

Jun711 commented 5 years ago

@swetashre I just tried both ways you proposed. Both didn't seem to work. The following are print statements that I got via CloudWatch.

packaging boto3 and botocore

The following is the log that I got via CloudWatch using method 1: packaging boto3 and botocore with lambda. Even though the printed versions of boto3 and botocore are the versions I packaged but it doesn't seem to use the intended versions of boto3 and botocore. 

251
2019-08-06T12:41:43.067-07:00
[DEBUG] 2019-08-06T19:41:42.667Z    c894e3e2-8501-4ea0-bd51-1afab4eeef0f    Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f3caba9d378>
252
2019-08-06T12:41:43.067-07:00
2019-08-06 19:41:42,667 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f3caba9d378>
253
2019-08-06T12:41:43.067-07:00
[DEBUG] 2019-08-06T19:41:42.668Z    c894e3e2-8501-4ea0-bd51-1afab4eeef0f    Event creating-client-class.polly: calling handler <function add_generate_presigned_url at 0x7f3caba692f0>
254
2019-08-06T12:41:43.067-07:00
2019-08-06 19:41:42,668 botocore.hooks [DEBUG] Event creating-client-class.polly: calling handler <function add_generate_presigned_url at 0x7f3caba692f0>
255
2019-08-06T12:41:43.067-07:00
[DEBUG] 2019-08-06T19:41:42.668Z    c894e3e2-8501-4ea0-bd51-1afab4eeef0f    The s3 config key is not a dictionary type, ignoring its value of: None
256
2019-08-06T12:41:43.067-07:00
2019-08-06 19:41:42,668 botocore.args [DEBUG] The s3 config key is not a dictionary type, ignoring its value of: None
257
2019-08-06T12:41:43.067-07:00
[DEBUG] 2019-08-06T19:41:42.669Z    c894e3e2-8501-4ea0-bd51-1afab4eeef0f    Setting polly timeout as (60, 60)
258
2019-08-06T12:41:43.067-07:00
2019-08-06 19:41:42,669 botocore.endpoint [DEBUG] Setting polly timeout as (60, 60)
259
2019-08-06T12:41:43.067-07:00
[DEBUG] 2019-08-06T19:41:42.670Z    c894e3e2-8501-4ea0-bd51-1afab4eeef0f    Registering retry handlers for service: polly
260
2019-08-06T12:41:43.067-07:00
2019-08-06 19:41:42,670 botocore.client [DEBUG] Registering retry handlers for service: polly
261
2019-08-06T12:41:43.067-07:00
engine: neural
262
2019-08-06T12:41:43.067-07:00
boto3 ver: 1.9.199
263
2019-08-06T12:41:43.067-07:00
botocore ver: 1.12.199
264
2019-08-06T12:41:43.067-07:00
[DEBUG] 2019-08-06T19:41:42.670Z    c894e3e2-8501-4ea0-bd51-1afab4eeef0f    Event before-parameter-build.polly.SynthesizeSpeech: calling handler <function generate_idempotent_uuid at 0x7f3caba3f7b8>
265
2019-08-06T12:41:43.067-07:00
2019-08-06 19:41:42,670 botocore.hooks [DEBUG] Event before-parameter-build.polly.SynthesizeSpeech: calling handler <function generate_idempotent_uuid at 0x7f3caba3f7b8>
266
2019-08-06T12:41:43.067-07:00
2019-08-06 19:41:42,670 botocore.hooks [DEBUG] Event before-parameter-build.polly.SynthesizeSpeech: calling handler <function generate_idempotent_uuid at 0x7f3caba3f7b8>
267
2019-08-06T12:41:43.067-07:00
synthesize_speech exception: Parameter validation failed:
268
2019-08-06T12:41:43.067-07:00
Unknown parameter in input: "Engine", must be one of: LexiconNames, OutputFormat, SampleRate, SpeechMarkTypes, Text, TextType, VoiceId, LanguageCode

Installing boto3 and botocore in tmp

It couldn't install and the error message is there is no pip-req-tracker file / directory. 

import sys
from pip._internal import main

main(['install', '-I', '-q', 'boto3', '--target', '/tmp/', '--no-cache-dir', '--disable-pip-version-check'])
main(['install', '-I', '-q', 'botocore', '--target', '/tmp/', '--no-cache-dir', '--disable-pip-version-check'])
sys.path.insert(0,'/tmp/')

import boto3
import botocore

Got the following error 

13
2019-08-06T12:57:32.497-07:00
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-5hq7qjzg/88fe6beef949849ebf711d07cd22562df433be0
swetashre commented 5 years ago

@Jun711 - Have you followed the link i shared with you. This link demonstrates all the steps on how to achieve the same : https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html#python-package-venv

I am able to use updated version of boto3 after following all the steps from this link. And i am curious to know which version of pip do you have ?

Jun711 commented 5 years ago

I am using CodeStar so it does packaging and deploying to lambda. I reinstalled Boto3 and Botocore using the commands listed below and it still didn't work.

I will install using venv.

Installing Boto3 and Botocore

The following is the command that I use to install Boto3 and Botocore. 

711:~/environment/tts (master) $ sudo rm -rf boto3Module/                                                                                                                         

711:~/environment/tts (master) $ sudo python3 -m pip --version
pip 19.2.1 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)

711:~/environment/tts (master) $ sudo python3 -m pip install boto3==1.9.199 --upgrade -t ./boto3Module 
Collecting boto3==1.9.199
  Using cached https://files.pythonhosted.org/packages/72/1a/97ca7494fd268835f2d2ea2c6b6ea3b7cfe271f22c2adb1ef45cf007d7f3/boto3-1.9.199-py2.py3-none-any.whl
Collecting botocore<1.13.0,>=1.12.199 (from boto3==1.9.199)
  Using cached https://files.pythonhosted.org/packages/70/f2/9e591425a2543a270c7e30551472f425fd31ddc8b117f6cfa6f85c98a068/botocore-1.12.202-py2.py3-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from boto3==1.9.199)
  Using cached https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Collecting s3transfer<0.3.0,>=0.2.0 (from boto3==1.9.199)
  Using cached https://files.pythonhosted.org/packages/16/8a/1fc3dba0c4923c2a76e1ff0d52b305c44606da63f718d14d3231e21c51b0/s3transfer-0.2.1-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore<1.13.0,>=1.12.199->boto3==1.9.199)
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting urllib3<1.26,>=1.20; python_version >= "3.4" (from botocore<1.13.0,>=1.12.199->boto3==1.9.199)
  Using cached https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting docutils<0.15,>=0.10 (from botocore<1.13.0,>=1.12.199->boto3==1.9.199)
  Using cached https://files.pythonhosted.org/packages/36/fa/08e9e6e0e3cbd1d362c3bbee8d01d0aedb2155c4ac112b19ef3cae8eed8d/docutils-0.14-py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore<1.13.0,>=1.12.199->boto3==1.9.199)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, urllib3, jmespath, docutils, botocore, s3transfer, boto3
Successfully installed boto3-1.9.199 botocore-1.12.202 docutils-0.14 jmespath-0.9.4 python-dateutil-2.8.0 s3transfer-0.2.1 six-1.12.0 urllib3-1.25.3

711:~/environment/tts (master) $ sudo python3 -m pip install botocore==1.12.202 --upgrade -t ./boto3Module                                                                        
Collecting botocore==1.12.202
  Using cached https://files.pythonhosted.org/packages/70/f2/9e591425a2543a270c7e30551472f425fd31ddc8b117f6cfa6f85c98a068/botocore-1.12.202-py2.py3-none-any.whl
Collecting docutils<0.15,>=0.10 (from botocore==1.12.202)
  Using cached https://files.pythonhosted.org/packages/36/fa/08e9e6e0e3cbd1d362c3bbee8d01d0aedb2155c4ac112b19ef3cae8eed8d/docutils-0.14-py3-none-any.whl
Collecting urllib3<1.26,>=1.20; python_version >= "3.4" (from botocore==1.12.202)
  Using cached https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.12.202)
  Using cached https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore==1.12.202)
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore==1.12.202)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: docutils, urllib3, jmespath, six, python-dateutil, botocore
Successfully installed botocore-1.12.202 docutils-0.14 jmespath-0.9.4 python-dateutil-2.8.0 six-1.12.0 urllib3-1.25.3

Importing Boto3 and Botocore

from boto3Module import boto3
from boto3Module import botocore

yaml

tts:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: index.handler
      Runtime: python3.7
...
Jun711 commented 5 years ago

@swetashre It didn't work when I installed using venv too.

Installation Commands

711:~/environment/tts (master) $ python3 -m venv v-env

711:~/environment/tts (master) $ source v-env/bin/activate

(v-env) 711:~/environment/tts (master) $ pip --version
pip 19.0.3 from /home/ec2-user/environment/tts/v-env/lib/python3.7/site-packages/pip (python 3.7)

(v-env) 711:~/environment/tts (master) $ pip install boto3 -t ./venvBoto3
The directory '/home/ec2-user/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/ec2-user/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting boto3
  Downloading https://files.pythonhosted.org/packages/6b/fd/18c862d197105b6fbe2d27d62ae541bd41e883f4e2b5ab6aab8c9d1a2396/boto3-1.9.202-py2.py3-none-any.whl (128kB)
    100% |████████████████████████████████| 133kB 13.9MB/s 
Collecting s3transfer<0.3.0,>=0.2.0 (from boto3)
  Downloading https://files.pythonhosted.org/packages/16/8a/1fc3dba0c4923c2a76e1ff0d52b305c44606da63f718d14d3231e21c51b0/s3transfer-0.2.1-py2.py3-none-any.whl (70kB)
    100% |████████████████████████████████| 71kB 13.2MB/s 
Collecting jmespath<1.0.0,>=0.7.1 (from boto3)
  Downloading https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Collecting botocore<1.13.0,>=1.12.202 (from boto3)
  Downloading https://files.pythonhosted.org/packages/70/f2/9e591425a2543a270c7e30551472f425fd31ddc8b117f6cfa6f85c98a068/botocore-1.12.202-py2.py3-none-any.whl (5.6MB)
    100% |████████████████████████████████| 5.6MB 10.3MB/s 
Collecting urllib3<1.26,>=1.20; python_version >= "3.4" (from botocore<1.13.0,>=1.12.202->boto3)
  Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
    100% |████████████████████████████████| 153kB 29.8MB/s 
Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore<1.13.0,>=1.12.202->boto3)
  Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
    100% |████████████████████████████████| 235kB 29.2MB/s 
Collecting docutils<0.15,>=0.10 (from botocore<1.13.0,>=1.12.202->boto3)
  Downloading https://files.pythonhosted.org/packages/36/fa/08e9e6e0e3cbd1d362c3bbee8d01d0aedb2155c4ac112b19ef3cae8eed8d/docutils-0.14-py3-none-any.whl (543kB)
    100% |████████████████████████████████| 552kB 28.9MB/s 
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore<1.13.0,>=1.12.202->boto3)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: jmespath, urllib3, six, python-dateutil, docutils, botocore, s3transfer, boto3
Successfully installed boto3-1.9.202 botocore-1.12.202 docutils-0.14 jmespath-0.9.4 python-dateutil-2.8.0 s3transfer-0.2.1 six-1.12.0 urllib3-1.25.3
You are using pip version 19.0.3, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

(v-env) 711:~/environment/tts (master) $ pip install botocore -t ./venvBoto3                                                         
The directory '/home/ec2-user/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/ec2-user/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting botocore
  Downloading https://files.pythonhosted.org/packages/70/f2/9e591425a2543a270c7e30551472f425fd31ddc8b117f6cfa6f85c98a068/botocore-1.12.202-py2.py3-none-any.whl (5.6MB)
    100% |████████████████████████████████| 5.6MB 10.2MB/s 
Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore)
  Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
    100% |████████████████████████████████| 235kB 28.7MB/s 
Collecting urllib3<1.26,>=1.20; python_version >= "3.4" (from botocore)
  Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
    100% |████████████████████████████████| 153kB 29.6MB/s 
Collecting docutils<0.15,>=0.10 (from botocore)
  Downloading https://files.pythonhosted.org/packages/36/fa/08e9e6e0e3cbd1d362c3bbee8d01d0aedb2155c4ac112b19ef3cae8eed8d/docutils-0.14-py3-none-any.whl (543kB)
    100% |████████████████████████████████| 552kB 26.7MB/s 
Collecting jmespath<1.0.0,>=0.7.1 (from botocore)
  Downloading https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, urllib3, docutils, jmespath, botocore
Successfully installed botocore-1.12.202 docutils-0.14 jmespath-0.9.4 python-dateutil-2.8.0 six-1.12.0 urllib3-1.25.3
Target directory /home/ec2-user/environment/tts/venvBoto3/urllib3 already exists. Specify --upgrade to force replacement.
Target directory /home/ec2-user/environment/tts/venvBoto3/dateutil already exists. Specify --upgrade to force replacement.
Target directory /home/ec2-user/environment/tts/venvBoto3/python_dateutil-2.8.0.dist-info already exists. Specify --upgrade to force replacement.
Target directory /home/ec2-user/environment/tts/venvBoto3/botocore already exists. Specify --upgrade to force replacement.
Target directory /home/ec2-user/environment/tts/venvBoto3/botocore-1.12.202.dist-info already exists. Specify --upgrade to force replacement.
Target directory /home/ec2-user/environment/tts/venvBoto3/__pycache__ already exists. Specify --upgrade to force replacement.
Target directory /home/ec2-user/environment/tts/venvBoto3/urllib3-1.25.3.dist-info already exists. Specify --upgrade to force replacement.
Target directory /home/ec2-user/environment/tts/venvBoto3/jmespath-0.9.4.dist-info already exists. Specify --upgrade to force replacement.
Target directory /home/ec2-user/environment/tts/venvBoto3/six-1.12.0.dist-info already exists. Specify --upgrade to force replacement.
Target directory /home/ec2-user/environment/tts/venvBoto3/docutils-0.14.dist-info already exists. Specify --upgrade to force replacement.
Target directory /home/ec2-user/environment/tts/venvBoto3/docutils already exists. Specify --upgrade to force replacement.
Target directory /home/ec2-user/environment/tts/venvBoto3/jmespath already exists. Specify --upgrade to force replacement.
Target directory /home/ec2-user/environment/tts/venvBoto3/six.py already exists. Specify --upgrade to force replacement.
Target directory /home/ec2-user/environment/tts/venvBoto3/bin already exists. Specify --upgrade to force replacement.
You are using pip version 19.0.3, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(v-env) 711:~/environment/tts (master) $ deactivate

Python

from venvBoto3 import boto3
from venvBoto3 import botocore

print('boto3 ver: ', boto3.__version__) # 1.9.202
print('botocore ver: ', botocore.__version__) # 1.12.202
client = boto3.client('polly')
Jun711 commented 5 years ago

@swetashre I created a new project and tried to use polly neural tts using sam local invoke on my machine, not via cloud9. It also doesn't recognize newer botocore installed locally within the project folder. Have you tried using it this way?

Is there a way to let boto3 client use a specific version of botocore?

index.py

import json
import datetime
from boto3Module import boto3
from boto3Module import botocore

def handler(event, context):
    data = {
        'output': 'Hello World',
        'timestamp': datetime.datetime.utcnow().isoformat()
    }
    print('boto3 ver: ', boto3.__version__)
    print('botocore ver: ', botocore.__version__)
    boto3.set_stream_logger('')
    pollyClient = boto3.client('polly')
    try:
        response = pollyClient.synthesize_speech(
                                            Engine='neural',
                                            OutputFormat='mp3',
                                            Text='<speak>test</speak>',
                                            TextType='ssml',
                                            VoiceId='Matthew'
                                        )
        print('tts response: ', response)
    except Exception as e:
        print('synthesize_speech exception: ', e)
    return {'statusCode': 200,
            'body': json.dumps(data),
            'headers': {'Content-Type': 'application/json'}}

yaml

Resources:
  Tts190808:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: python3.7
      Role:
        Fn::GetAtt:
        - LambdaExecutionRole
        - Arn
      Events:
        GetEvent:
          Type: Api
          Properties:
            Path: /
            Method: get
        PostEvent:
          Type: Api
          Properties:
            Path: /
            Method: post

log

2019-08-08 15:23:30 Invoking index.handler (python3.7)
2019-08-08 15:23:30 Found credentials in shared credentials file: ~/.aws/credentials

Fetching lambci/lambda:python3.7 Docker container image......
2019-08-08 15:23:32 Mounting /Users/711/tts as /var/task:ro,delegated inside runtime container
START RequestId: 52fdfc07-2182-154f-163f-5f0f9a6227d1 Version: $LATEST
boto3 ver:  1.9.199
botocore ver:  1.12.204
[DEBUG] 2019-08-08T22:23:33.77Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane

2019-08-08 22:23:33,077 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
[DEBUG] 2019-08-08T22:23:33.80Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Changing event name from before-call.apigateway to before-call.api-gateway

2019-08-08 22:23:33,080 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
[DEBUG] 2019-08-08T22:23:33.81Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict

2019-08-08 22:23:33,081 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
[DEBUG] 2019-08-08T22:23:33.83Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration

2019-08-08 22:23:33,083 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
[DEBUG] 2019-08-08T22:23:33.84Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Changing event name from before-parameter-build.route53 to before-parameter-build.route-53

2019-08-08 22:23:33,084 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
[DEBUG] 2019-08-08T22:23:33.84Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search

2019-08-08 22:23:33,084 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
[DEBUG] 2019-08-08T22:23:33.85Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section

2019-08-08 22:23:33,085 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
[DEBUG] 2019-08-08T22:23:33.88Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask

2019-08-08 22:23:33,088 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
[DEBUG] 2019-08-08T22:23:33.88Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section

2019-08-08 22:23:33,088 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
[DEBUG] 2019-08-08T22:23:33.88Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search

2019-08-08 22:23:33,088 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
[DEBUG] 2019-08-08T22:23:33.89Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section

2019-08-08 22:23:33,089 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
[DEBUG] 2019-08-08T22:23:33.95Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Looking for credentials via: env

2019-08-08 22:23:33,095 botocore.credentials [DEBUG] Looking for credentials via: env
[INFO]  2019-08-08T22:23:33.95Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Found credentials in environment variables.

2019-08-08 22:23:33,095 botocore.credentials [INFO] Found credentials in environment variables.
[DEBUG] 2019-08-08T22:23:33.96Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Loading JSON file: /var/runtime/botocore/data/endpoints.json

2019-08-08 22:23:33,096 botocore.loaders [DEBUG] Loading JSON file: /var/runtime/botocore/data/endpoints.json
[DEBUG] 2019-08-08T22:23:33.99Z 52fdfc07-2182-154f-163f-5f0f9a6227d1    Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f645beaf950>

2019-08-08 22:23:33,099 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f645beaf950>
[DEBUG] 2019-08-08T22:23:33.106Z    52fdfc07-2182-154f-163f-5f0f9a6227d1    Loading JSON file: /var/runtime/botocore/data/polly/2016-06-10/service-2.json

2019-08-08 22:23:33,106 botocore.loaders [DEBUG] Loading JSON file: /var/runtime/botocore/data/polly/2016-06-10/service-2.json
[DEBUG] 2019-08-08T22:23:33.108Z    52fdfc07-2182-154f-163f-5f0f9a6227d1    Event creating-client-class.polly: calling handler <function add_generate_presigned_url at 0x7f645beeb8c8>

2019-08-08 22:23:33,108 botocore.hooks [DEBUG] Event creating-client-class.polly: calling handler <function add_generate_presigned_url at 0x7f645beeb8c8>
[DEBUG] 2019-08-08T22:23:33.109Z    52fdfc07-2182-154f-163f-5f0f9a6227d1    The s3 config key is not a dictionary type, ignoring its value of: None

2019-08-08 22:23:33,109 botocore.args [DEBUG] The s3 config key is not a dictionary type, ignoring its value of: None
[DEBUG] 2019-08-08T22:23:33.111Z    52fdfc07-2182-154f-163f-5f0f9a6227d1    Setting polly timeout as (60, 60)

2019-08-08 22:23:33,111 botocore.endpoint [DEBUG] Setting polly timeout as (60, 60)
[DEBUG] 2019-08-08T22:23:33.112Z    52fdfc07-2182-154f-163f-5f0f9a6227d1    Loading JSON file: /var/runtime/botocore/data/_retry.json

2019-08-08 22:23:33,112 botocore.loaders [DEBUG] Loading JSON file: /var/runtime/botocore/data/_retry.json
[DEBUG] 2019-08-08T22:23:33.113Z    52fdfc07-2182-154f-163f-5f0f9a6227d1    Registering retry handlers for service: polly

2019-08-08 22:23:33,113 botocore.client [DEBUG] Registering retry handlers for service: polly
[DEBUG] 2019-08-08T22:23:33.114Z    52fdfc07-2182-154f-163f-5f0f9a6227d1    Event before-parameter-build.polly.SynthesizeSpeech: calling handler <function generate_idempotent_uuid at 0x7f645be45d90>

2019-08-08 22:23:33,114 botocore.hooks [DEBUG] Event before-parameter-build.polly.SynthesizeSpeech: calling handler <function generate_idempotent_uuid at 0x7f645be45d90>
synthesize_speech exception:  Parameter validation failed:
Unknown parameter in input: "Engine", must be one of: LexiconNames, OutputFormat, SampleRate, SpeechMarkTypes, Text, TextType, VoiceId, LanguageCode
END RequestId: 52fdfc07-2182-154f-163f-5f0f9a6227d1
REPORT RequestId: 52fdfc07-2182-154f-163f-5f0f9a6227d1  Duration: 43.73 ms  Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 32 MB
Jun711 commented 5 years ago

Hi @swetashre, I wonder if there is any workaround or I have to wait lambda runtime botocore to be updated to a version that supports Engine parameter?

Thank you

swetashre commented 5 years ago

@Jun711 - You can try using chalice which will handle the packaging things for you. Here is the step by step instruction: 

 pip install chalice #install chalice 
chalice new-project test-package #create a package named test-package using chalice
cd test-package
ls
app.py          requirements.txt
echo "boto3==1.9.212" > requirements.txt 
cat requirements.txt #this requirement file will basically do pip install for you 
boto3==1.9.212
open app.py #this will open the app.py file and you can write your code inside the file 

#app.py file content

import boto3

from chalice import Chalice

app = Chalice(app_name='test-package')

@app.lambda_function()
def fn(event, context):
    pollyClient = boto3.client('polly')
    response = pollyClient.synthesize_speech(Engine='neural',OutputFormat='mp3',Text='test',TextType='ssml',VoiceId='Matthew')
    return response
 chalice deploy
chalice invoke -n fn

This code will create a lambda function in the console with name test-package-dev-fn with all the required dependency. So you don't have to worry about the dependency and you can test the code easily.

Hope it helps.

Jun711 commented 5 years ago

@swetashre thanks for the detailed instructions. I will give it a try for testing purpose.

because we already have our service running on lambda already using CodeStar as CI/CD tools. Adding chalice will require us to migrate what we on CodeStar?

swetashre commented 5 years ago

I tried using virtual environment and i am not getting any error after that. These are all the steps i tried to update boto3 version in my lambda function.

python3 -m venv lambda-env
source lambda-env/bin/activate
pip install boto3
deactivate
cd lambda-env/lib/python3.7/site-packages
zip -r9 ${OLDPWD}/function.zip .
cd $OLDPWD
zip -g function.zip lambda_function.py  #lambda_function.py will contain your code 
aws lambda update-function-code --function-name function-name --zip-file fileb://function.zip

These are the steps to update boto3 version in lambda function using virtual environment.

Jun711 commented 5 years ago

@swetashre I didn't try using chalice since using virtual environment is more convenient for my use case. I can try neural voices via lambda now. Thank you.

and, because my code is in multiple folders and files, I had to run zip -g function.zip -r folder1/* for folders and zip -g function.zip python_file.py to include all the needed files. Maybe there is a shorter command. There are other yml files, test files etc that can't be included.

some follow-up questions if you don't mind: 1) what does zip -r9 do? -r is used to replace, right? 2) I noticed lambda code size increased from 1.7MB to 14.6MB. would it affect cost or cold start? 3) because I am using aws codestar to automate build and deployment process. Now that I have to use CLI directly, which cli to deploy to my API Gateway directly? is it aws api-gateway update-deployment

swetashre commented 4 years ago

I don't have much idea about lambda pricing but as far as i know it should not affect the cost. I am closing the issue as i believe this issue has been resolved.

Please reopen if you still have any concerns.