search

aws / aws-toolkit-azure-devops

AWS Toolkit for Azure DevOps
Other
245 stars 104 forks source link

Python Boto and boto3 task #143

Open ghost opened 5 years ago

ghost commented 5 years ago

Hello, Can we take a branch copy of the Microsoft Python task for running Python scripts for AWS in the same way that we have the Powershell Task for AWS.

We just need to have the ability to run using the credentials task helper and to install boto and boto3

ghost commented 5 years ago

Hi, just checking to see if there are any resource available to look at this request ? it should be an easy one to add add.

ghost commented 5 years ago

Hello, just seeing if there is any planned updates to this request, our workaround of using AWS Shell Script to call the python binary with the .py file and some arguments has now broken with the update to 1.3 >

it was a poor workaround in the 1st place, would be good to have native support for python on windows and linux build agents.

hunterwerlla commented 5 years ago

@steven-cuthill-otm In what way did the 1.3 update break your script? I'd like to get that fixed if there is an issue

ghost commented 5 years ago

Hello @hunterwerlla ,

I am still trying to put together what has went wrong, basically this pipeline gets triggered on a weekly basis, and failed to run after it looks like we did a version bump, i cant switch back to a lower version to confirm this sadly.

file it says it cant find is 100% there, i think its someone getting lost when translating in the line script, i have attached the task input and the output for both the working logs and the now broken logs.

task input

- task: AmazonWebServices.aws-vsts-tools.AWSShellScript.AWSShellScript@1
  displayName: 'AWS Shell Script - scale up'
  inputs:
    awsCredentials: 'XXXXXXXXXXXXXXX'
    regionName: 'eu-west-2'
    scriptType: inline
    inlineScript: |
     which python
     pip --version
     pip install boto3
     pip install setuptools
     pip install wheel
     pip install awscli --upgrade
     aws --version
     python _OnTheMarket_internal-infrastructure/appsupport/helper-scripts/set-asg.py internal-prd-asg-eu-west-2-packer 1 1 eu-west-2

Working Task Output 

2019-06-09T03:00:31.3486656Z e3b81a72-c761-4a6a-8373-e87825107277 exists true
2019-06-09T03:00:31.3487469Z ...configuring AWS credentials from service endpoint 'e3b81a72-c761-4a6a-8373-e87825107277'
2019-06-09T03:00:31.3488227Z ...endpoint defines standard access/secret key credentials
2019-06-09T03:00:31.3509789Z Configuring region for task
2019-06-09T03:00:31.3512518Z ...configured to use region eu-west-2, defined in task.
2019-06-09T03:00:31.3762578Z [command]"C:\Program Files\Git\bin\bash.exe" d:\a\_temp\awsshellscript_4304.sh
2019-06-09T03:00:31.9423093Z /c/hostedtoolcache/windows/Python/3.6.8/x64/python
2019-06-09T03:00:35.5813873Z pip 19.1.1 from c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\pip (python 3.6)
2019-06-09T03:00:35.5814077Z 
2019-06-09T03:00:36.1084843Z Collecting boto3
2019-06-09T03:00:37.0802715Z   Downloading https://files.pythonhosted.org/packages/5c/4e/ed845436bdf61235401224ced23319c7016d5837c9d1b5dcd2456c082d24/boto3-1.9.164-py2.py3-none-any.whl (128kB)
2019-06-09T03:00:37.2077369Z Collecting jmespath<1.0.0,>=0.7.1 (from boto3)
2019-06-09T03:00:37.2521204Z   Downloading https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
2019-06-09T03:00:37.2843941Z Collecting botocore<1.13.0,>=1.12.164 (from boto3)
2019-06-09T03:00:37.7205046Z   Downloading https://files.pythonhosted.org/packages/e1/81/f63beb56e22e4f3d96cf3ec93f949646b4e68257380e95a6e9293fd98dc6/botocore-1.12.164-py2.py3-none-any.whl (5.5MB)
2019-06-09T03:00:39.3016184Z Collecting s3transfer<0.3.0,>=0.2.0 (from boto3)
2019-06-09T03:00:39.3485520Z   Downloading https://files.pythonhosted.org/packages/16/8a/1fc3dba0c4923c2a76e1ff0d52b305c44606da63f718d14d3231e21c51b0/s3transfer-0.2.1-py2.py3-none-any.whl (70kB)
2019-06-09T03:00:39.4372741Z Collecting urllib3<1.26,>=1.20; python_version >= "3.4" (from botocore<1.13.0,>=1.12.164->boto3)
2019-06-09T03:00:39.5165336Z   Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
2019-06-09T03:00:39.6129967Z Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore<1.13.0,>=1.12.164->boto3)
2019-06-09T03:00:39.6689109Z   Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
2019-06-09T03:00:39.7438947Z Collecting docutils>=0.10 (from botocore<1.13.0,>=1.12.164->boto3)
2019-06-09T03:00:39.7887360Z   Downloading https://files.pythonhosted.org/packages/36/fa/08e9e6e0e3cbd1d362c3bbee8d01d0aedb2155c4ac112b19ef3cae8eed8d/docutils-0.14-py3-none-any.whl (543kB)
2019-06-09T03:00:40.1113177Z Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore<1.13.0,>=1.12.164->boto3)
2019-06-09T03:00:40.1587853Z   Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
2019-06-09T03:00:40.2239615Z Installing collected packages: jmespath, urllib3, six, python-dateutil, docutils, botocore, s3transfer, boto3
2019-06-09T03:00:47.6215982Z Successfully installed boto3-1.9.164 botocore-1.12.164 docutils-0.14 jmespath-0.9.4 python-dateutil-2.8.0 s3transfer-0.2.1 six-1.12.0 urllib3-1.25.3
2019-06-09T03:00:49.3293805Z Requirement already satisfied: setuptools in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (40.6.2)
2019-06-09T03:00:50.0264839Z Collecting wheel
2019-06-09T03:00:50.4147563Z   Downloading https://files.pythonhosted.org/packages/bb/10/44230dd6bf3563b8f227dbf344c908d412ad2ff48066476672f3a72e174e/wheel-0.33.4-py2.py3-none-any.whl
2019-06-09T03:00:50.4924122Z Installing collected packages: wheel
2019-06-09T03:00:50.5860807Z Successfully installed wheel-0.33.4
2019-06-09T03:00:51.1435549Z Collecting awscli
2019-06-09T03:00:51.9869570Z   Downloading https://files.pythonhosted.org/packages/0e/f0/5478c9831ca7724fb7c8d63563d51e9788ce218ed2263abc30f3b214a684/awscli-1.16.174-py2.py3-none-any.whl (1.6MB)
2019-06-09T03:00:54.0143447Z Collecting colorama<=0.3.9,>=0.2.5 (from awscli)
2019-06-09T03:00:54.0782547Z   Downloading https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
2019-06-09T03:00:54.1051376Z Requirement already satisfied, skipping upgrade: botocore==1.12.164 in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from awscli) (1.12.164)
2019-06-09T03:00:54.1205025Z Requirement already satisfied, skipping upgrade: docutils>=0.10 in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from awscli) (0.14)
2019-06-09T03:00:54.1290656Z Collecting PyYAML<=3.13,>=3.10 (from awscli)
2019-06-09T03:00:54.2002692Z   Downloading https://files.pythonhosted.org/packages/4f/ca/5fad249c5032270540c24d2189b0ddf1396aac49b0bdc548162edcf14131/PyYAML-3.13-cp36-cp36m-win_amd64.whl (206kB)
2019-06-09T03:00:54.2610066Z Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
2019-06-09T03:00:54.3097560Z   Downloading https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl (46kB)
2019-06-09T03:00:54.3495370Z Requirement already satisfied, skipping upgrade: s3transfer<0.3.0,>=0.2.0 in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from awscli) (0.2.1)
2019-06-09T03:00:54.3548499Z Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from botocore==1.12.164->awscli) (0.9.4)
2019-06-09T03:00:54.3583466Z Requirement already satisfied, skipping upgrade: urllib3<1.26,>=1.20; python_version >= "3.4" in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from botocore==1.12.164->awscli) (1.25.3)
2019-06-09T03:00:54.3746999Z Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from botocore==1.12.164->awscli) (2.8.0)
2019-06-09T03:00:54.3830841Z Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
2019-06-09T03:00:54.4828328Z   Downloading https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl (73kB)
2019-06-09T03:00:54.5436604Z Requirement already satisfied, skipping upgrade: six>=1.5 in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore==1.12.164->awscli) (1.12.0)
2019-06-09T03:00:54.5656963Z Installing collected packages: colorama, PyYAML, pyasn1, rsa, awscli
2019-06-09T03:01:02.3991220Z Successfully installed PyYAML-3.13 awscli-1.16.174 colorama-0.3.9 pyasn1-0.4.5 rsa-3.4.2
2019-06-09T03:01:04.7102853Z aws-cli/1.16.174 Python/3.6.8 Windows/10 exec-env/VSTS-AWSShellScript botocore/1.12.164
2019-06-09T03:01:05.7999155Z INFO:root:setting instance min count to 1
2019-06-09T03:01:05.8942958Z INFO:root:checking status of instances...
2019-06-09T03:01:46.3676166Z INFO:root:There are currently 0 of 1 healthy instances in service
2019-06-09T03:01:46.3676750Z INFO:root:waited for..00:40 minutes:seconds
2019-06-09T03:01:46.3676917Z INFO:root:pausing for 10 seconds
2019-06-09T03:02:06.5500771Z INFO:root:There are currently 0 of 1 healthy instances in service
2019-06-09T03:02:06.5503082Z INFO:root:waited for..01:00 minutes:seconds
2019-06-09T03:02:06.5503212Z INFO:root:pausing for 10 seconds
2019-06-09T03:02:26.6730628Z INFO:root:There are currently 1 of 1 healthy instances in service
2019-06-09T03:02:27.5665890Z INFO:root:waited for..01:20 minutes:seconds
2019-06-09T03:02:27.5666550Z INFO:root:pausing for 10 seconds
2019-06-09T03:02:46.6894693Z INFO:root:Instances Now in service
2019-06-09T03:02:46.7431808Z ##[section]Finishing: AWS Shell Script - scale up

Broken Task Output 

2019-06-18T15:53:12.4136313Z Configuring credentials for task
2019-06-18T15:53:12.4141136Z e3b81a72-c761-4a6a-8373-e87825107277 exists true
2019-06-18T15:53:12.4141873Z ...configuring AWS credentials from service endpoint 'e3b81a72-c761-4a6a-8373-e87825107277'
2019-06-18T15:53:12.4142139Z ...endpoint defines standard access/secret key credentials
2019-06-18T15:53:12.4165552Z Configuring region for task
2019-06-18T15:53:12.4167505Z ...configured to use region eu-west-2, defined in task.
2019-06-18T15:53:12.4946255Z [command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc -c pwd
2019-06-18T15:53:15.1188073Z /d/a/_temp
2019-06-18T15:53:15.1259177Z [command]"C:\Program Files\Git\bin\bash.exe" /d/a/_temp/awsshellscript_4400.sh
2019-06-18T15:53:15.3895344Z /c/hostedtoolcache/windows/Python/3.6.8/x64/python
2019-06-18T15:53:30.6243430Z pip 19.1.1 from c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages\pip (python 3.6)
2019-06-18T15:53:30.6244085Z 
2019-06-18T15:53:31.1409793Z Collecting boto3
2019-06-18T15:53:34.0169115Z   Downloading https://files.pythonhosted.org/packages/d0/e4/8871d9c3d27bedadabe3da72b3435875d1a401e58ce15f0cc2122b0791a2/boto3-1.9.170-py2.py3-none-any.whl (128kB)
2019-06-18T15:53:34.1748803Z Collecting botocore<1.13.0,>=1.12.170 (from boto3)
2019-06-18T15:53:34.6589908Z   Downloading https://files.pythonhosted.org/packages/b8/06/6fb79b75bba6f6515d2f2f42ee7f91d85eea29d3d164a03981184efbaeee/botocore-1.12.170-py2.py3-none-any.whl (5.5MB)
2019-06-18T15:53:36.3763583Z Collecting s3transfer<0.3.0,>=0.2.0 (from boto3)
2019-06-18T15:53:36.4195832Z   Downloading https://files.pythonhosted.org/packages/16/8a/1fc3dba0c4923c2a76e1ff0d52b305c44606da63f718d14d3231e21c51b0/s3transfer-0.2.1-py2.py3-none-any.whl (70kB)
2019-06-18T15:53:36.5497219Z Collecting jmespath<1.0.0,>=0.7.1 (from boto3)
2019-06-18T15:53:36.6504635Z   Downloading https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
2019-06-18T15:53:36.7607241Z Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore<1.13.0,>=1.12.170->boto3)
2019-06-18T15:53:36.8464920Z   Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
2019-06-18T15:53:37.1847246Z Collecting docutils>=0.10 (from botocore<1.13.0,>=1.12.170->boto3)
2019-06-18T15:53:37.3046320Z   Downloading https://files.pythonhosted.org/packages/36/fa/08e9e6e0e3cbd1d362c3bbee8d01d0aedb2155c4ac112b19ef3cae8eed8d/docutils-0.14-py3-none-any.whl (543kB)
2019-06-18T15:53:39.6238626Z Collecting urllib3<1.26,>=1.20; python_version >= "3.4" (from botocore<1.13.0,>=1.12.170->boto3)
2019-06-18T15:53:39.7563771Z   Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
2019-06-18T15:53:40.3799033Z Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore<1.13.0,>=1.12.170->boto3)
2019-06-18T15:53:40.4538767Z   Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
2019-06-18T15:53:40.8141466Z Installing collected packages: six, python-dateutil, docutils, urllib3, jmespath, botocore, s3transfer, boto3
2019-06-18T15:53:58.2608440Z Successfully installed boto3-1.9.170 botocore-1.12.170 docutils-0.14 jmespath-0.9.4 python-dateutil-2.8.0 s3transfer-0.2.1 six-1.12.0 urllib3-1.25.3
2019-06-18T15:53:59.4314064Z Requirement already satisfied: setuptools in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (40.6.2)
2019-06-18T15:54:00.1026071Z Collecting wheel
2019-06-18T15:54:00.4700623Z   Downloading https://files.pythonhosted.org/packages/bb/10/44230dd6bf3563b8f227dbf344c908d412ad2ff48066476672f3a72e174e/wheel-0.33.4-py2.py3-none-any.whl
2019-06-18T15:54:00.5425698Z Installing collected packages: wheel
2019-06-18T15:54:00.6600579Z Successfully installed wheel-0.33.4
2019-06-18T15:54:01.1648628Z Collecting awscli
2019-06-18T15:54:01.9725959Z   Downloading https://files.pythonhosted.org/packages/24/3d/632f17377d5fc43f51658009f9cfd79c549d104915da7cb2977ac4c0b9e8/awscli-1.16.180-py2.py3-none-any.whl (1.6MB)
2019-06-18T15:54:06.5439750Z Collecting PyYAML<=5.1,>=3.10; python_version != "2.6" (from awscli)
2019-06-18T15:54:06.6824115Z   Downloading https://files.pythonhosted.org/packages/72/da/386634cb17f33b12278af017ccb3314cede4806af70285f396b8f6af1203/PyYAML-5.1-cp36-cp36m-win_amd64.whl (214kB)
2019-06-18T15:54:07.0640478Z Requirement already satisfied, skipping upgrade: botocore==1.12.170 in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from awscli) (1.12.170)
2019-06-18T15:54:07.0798933Z Requirement already satisfied, skipping upgrade: docutils>=0.10 in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from awscli) (0.14)
2019-06-18T15:54:07.0871872Z Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
2019-06-18T15:54:07.1804855Z   Downloading https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl (46kB)
2019-06-18T15:54:07.5045687Z Collecting colorama<=0.3.9,>=0.2.5 (from awscli)
2019-06-18T15:54:07.5972093Z   Downloading https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
2019-06-18T15:54:07.7459376Z Requirement already satisfied, skipping upgrade: s3transfer<0.3.0,>=0.2.0 in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from awscli) (0.2.1)
2019-06-18T15:54:07.7507321Z Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from botocore==1.12.170->awscli) (0.9.4)
2019-06-18T15:54:07.7540826Z Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from botocore==1.12.170->awscli) (2.8.0)
2019-06-18T15:54:07.7585446Z Requirement already satisfied, skipping upgrade: urllib3<1.26,>=1.20; python_version >= "3.4" in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from botocore==1.12.170->awscli) (1.25.3)
2019-06-18T15:54:07.7776756Z Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
2019-06-18T15:54:08.0074130Z   Downloading https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl (73kB)
2019-06-18T15:54:08.4910201Z Requirement already satisfied, skipping upgrade: six>=1.5 in c:\hostedtoolcache\windows\python\3.6.8\x64\lib\site-packages (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore==1.12.170->awscli) (1.12.0)
2019-06-18T15:54:08.5129465Z Installing collected packages: PyYAML, pyasn1, rsa, colorama, awscli
2019-06-18T15:54:36.2023295Z Successfully installed PyYAML-5.1 awscli-1.16.180 colorama-0.3.9 pyasn1-0.4.5 rsa-3.4.2
2019-06-18T15:54:39.2281438Z aws-cli/1.16.180 Python/3.6.8 Windows/10 exec-env/VSTS-AWSShellScript botocore/1.12.170
2019-06-18T15:54:39.3096194Z C:\hostedtoolcache\windows\Python\3.6.8\x64\python.exe: can't open file '_OnTheMarket_internal-infrastructure/appsupport/helper-scripts/set-asg.py': [Errno 2] No such file or directory
2019-06-18T15:54:39.3276966Z ##[error]Error: The process 'C:\Program Files\Git\bin\bash.exe' failed with exit code 2
2019-06-18T15:54:39.3334825Z ##[section]Finishing: AWS Shell Script - scale up
hunterwerlla commented 5 years ago

@steven-cuthill-otm I think it's the same bug as https://github.com/aws/aws-vsts-tools/issues/254 , it is fixed in today's 1.4.0 release, please try it out

ghost commented 5 years ago

hi @hunterwerlla yup, thats fixed now.

any chance we can get python task anytime soon ? python is such a big language in the aws space its a big gap.

lowpast commented 4 years ago

+1 on this feature request.

Current work around is to use the AWS Bash task to call a local python file, not ideal but it works. Id prefer to just run python under the context of AWS.

ghost commented 4 years ago

Just Bumping this again...

Adesanya-Toba commented 8 months ago

Also need this ✋ Is there an Azure DevOps task available now or at least a better walk around?