dcfranca commented 4 years ago

Please describe which feature you have a question about? Amplify Console CI/CD

Provide additional details On the build phase of the default amplify configuration for deployment: - amplifyPush --simple I'm getting an error as it can't find the Python3.8 installed

2020-04-16T08:03:36.931Z [INFO]: python3 found but version Python 3.7.4 is less than the minimum required version. You must have python >= 3.8 installed and available on your PATH as "python3". It can be installed from https://www.python.org/downloads

The build image I'm using is the "Amazon Linux:2", which according to the documentation contains Python3.8

but even running yum list python shows only python3.7 version, no python3.8 available

I also tried to use a custom image from the Docker hub (i.e: python3.8-alpine), but they fail on the build phase without any log information.

I also tried to change my runtime requirements to Python3.8, changed it on Cloudformation files for the functions and on the Pipfile, but for some reason (I guess it requires Python3.7 for other things), it still fails with the same error.

The only way I have made it works is building python3.8 from the source, but it takes ages and it is far from an ideal solution

dcfranca commented 4 years ago

Any updates?

sudeep-quantelai commented 4 years ago

I am encountering this issue too.

dcfranca commented 4 years ago

@swaminator It is almost a month in pending triage state, do you have any updates on it?

dcfranca commented 4 years ago

1 month anniversary 🎂

appasahebs commented 4 years ago

Any updates on this? I am facing same issue.

matiboy commented 4 years ago

Same issue; just to mention one workaround (as mentioned by @danielfranca) is to build from source: add the following to amplify.yml in the AWS console under App settings -> Build settings:

backend:
  phases:
    preBuild:
      commands:
        - export BASE_PATH=$(pwd)
        - yum install -y gcc openssl-devel bzip2-devel libffi-devel python3.8-pip
        - cd /opt && wget https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tgz
        - cd /opt && tar xzf Python-3.8.2.tgz 
        - cd /opt/Python-3.8.2 && ./configure --enable-optimizations
        - cd /opt/Python-3.8.2 && make altinstall
        - pip3.8 install --user pipenv
        - ln -fs /usr/local/bin/python3.8 /usr/bin/python3
        - ln -fs /usr/local/bin/pip3.8 /usr/bin/pip3
        - cd $BASE_PATH

It would be nice if Amplify's build used the same features/options as CodeBuild's one (which includes specifying the runtime versions of Python/Java/etc)

ravibanks54 commented 4 years ago

Any updates on this? It's driving up my build time and costs un-necessarily.

dcfranca commented 4 years ago

Please, post some updates regarding this issue, it is really annoying and it is slowing down the development workflow by large

@swaminator

roeemz commented 4 years ago

Same here, I am assuming that starting from some version Python 3.8 is a requirement, haven't investigated further.

roeemz commented 4 years ago

Is there a way to make the Amplify CLI work with Python 3.7 for now? a verified working version or some tweaking on its files?

I'll update if I find where the dependency comes from and how to temporarily fix it for now. (installing Python 3.8 manually consumes too much time...), just need to find the time to get to it, hopefully there will be a fix before then.

samputer commented 4 years ago

Another sufferer here! Would really appreciate an update on this please guys?

I suppose we could use a custom build image for this, but it seems unnecessarily complex.

lorengordon commented 4 years ago

Using a custom image does indeed work. Tested using nikolaik/python-nodejs:python3.8-nodejs10. Much faster than installing python3.8 in a pre-build step!

https://hub.docker.com/r/nikolaik/python-nodejs/

shawnmmatthews commented 4 years ago

Any updates on this? It looks like 3.8 should be supported, but I had to add the prebuild steps mentioned by @matiboy to get the backend to build. Which is rather frustrating and slow.

dcfranca commented 4 years ago

@shawnmmatthews I think the best solution so far is to use a custom image like mentioned on @lorengordon post I'm using the same image and it is working fine: https://hub.docker.com/r/nikolaik/python-nodejs/

shawnmmatthews commented 4 years ago

@dcfranca is there a recommended method for using a custom image? Or should I be looking into something like this https://docs.aws.amazon.com/lambda/latest/dg/runtimes-walkthrough.html. I see a dropdown from the Lambda console for custom, but not somewhere to specify which image to use. Thanks for reaching out.

dcfranca commented 4 years ago

@shawnmmatthews You can do it from the Amplify Console. Go to the Amplify Console, open the menu on the left, Click on "Build Settings", scroll down until you see "Build Image Settings", on the dropdown select Custom, then enter the image name on the field just below it

shawnmmatthews commented 4 years ago

Awesome. I assumed I was just missing the setting somewhere. Thank you so much @dcfranca!

pchevallier commented 4 years ago

cf: https://github.com/aws-amplify/amplify-console/issues/595#issuecomment-633165993 The only issue I came across using this Python3.8 support patch, is that my aws-exports.js was still not generated in my CI/CD environment.

To workaround this issue, I added the original command amplifhPush --simple at the end of the build section and my export file is now generated with the proper values.

backend:
  phases:
    build:
      commands:
        - export BASE_PATH=$(pwd)
        - yum install -y gcc openssl-devel bzip2-devel libffi-devel python3.8-pip
        - cd /opt && wget https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tgz
        - cd /opt && tar xzf Python-3.8.2.tgz 
        - cd /opt/Python-3.8.2 && ./configure --enable-optimizations
        - cd /opt/Python-3.8.2 && make altinstall
        - pip3.8 install --user pipenv
        - ln -fs /usr/local/bin/python3.8 /usr/bin/python3
        - ln -fs /usr/local/bin/pip3.8 /usr/bin/pip3
        - cd $BASE_PATH
        - amplifyPush --simple

christophebeling commented 4 years ago

I'm astonished that there is still no fix for this. Pretty disappointing.

roeemz commented 4 years ago

@christophebeling , its pretty crazy that practically Amplify Console is broken and this is not getting a fix... Even if this fix requires some cross-teams work.

siegerts commented 4 years ago

@christophebeling @roeemz , the build has been working fine utilizing the solution mentioned above (i.e. swapping the build image). This can be done with the https://hub.docker.com/r/nikolaik/python-nodejs/ or your own.

lorengordon commented 4 years ago

Any maintainers tracking or working this issue? While it is easy enough to workaround the problem using the custom image when the source is GitHub, it's more of a pain when the source is CodeCommit. This is because the "cloning repository" step attempts to run the aws command to setup the credential-helper, but the custom image does not have the aws-cli installed. So we're back to either installing python3.8 in the preBuild phase, or having to manage builds of custom images ourselves. It's sad when AWS services don't really support use of AWS services. 😢

2020-09-14T16:17:53.246Z [INFO]: aws codecommit credential-helper $@ get: 1: aws codecommit credential-helper $@ get: aws: not found
2020-09-14T16:17:53.246Z [INFO]: fatal: could not read Username for 'https://git-codecommit.us-east-1.amazonaws.com/v1/repos/.....': No such device or address
2020-09-14T16:17:53.247Z [ERROR]: !!! Unable to clone repository.

wvidana commented 4 years ago

Here is a publicly available build of this project's fork (the above PR), with Python 3.8 added wvidana/amplifybuildimage:python38.

Don't forget to install pipenv on your amplify.yml:

backend:
  phases:
    build:
      commands:
        - pip3 install --user pipenv

michaelbrewer commented 3 years ago

Hopefully this can be resolved soon so we can officially support python functions.

dcfranca commented 3 years ago

Quite often I get the error that I have reached the pull limits for the custom image from Docker hub

[BUILD_CONTAINER_UNABLE_TO_PULL_IMAGE: Unable to pull customer's container image. CannotPullContainerError: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading

Is it at least possible to login into Docker hub from the Amplify console to increase this limit?

johnhill2424 commented 3 years ago

Using the Codebuild AWS image worked for me:

AlessandroVol23 commented 3 years ago

Worked for me as well with the image: aws/codebuild/amazonlinux2-x86_64-standard:3.0

Left in the menu on Build Settings -> Scroll down to Build image settings -> Edit -> Build image in Dropdown -> Enter aws/codebuild/amazonlinux2-x86_64-standard:3.0

wvidana commented 3 years ago

@swaminator can you clarify? Should we use the codebuild image or wait for https://github.com/aws-amplify/amplify-console/pull/1139 ? I don't mind on discarding the PR if not necessary

chazarabriseis commented 3 years ago

@wvidana Thanks for the wvidana/amplifybuildimage:python38 docker image! It helped me to get my python lambdas working with amplify :)

Java path problem

But now I also have java lambdas that I would like to host on the same project with amplify so I thought I just fork your image, add java and gradle to it and I should be good to go. However, using the updated dockerimage chazarabriseis/amplifybuildimage:python38java11gradle6 I am still getting the error from amplify backend build: Unable to find Java compiler version >=11 on the path. Download link: https://amzn.to/2UUljp9 Unable to find Gradle version >=5 on the path. Download link: https://bit.ly/3aGYDj6

I am new to dockerfiles and was hoping you or someone else can give me a hint what I am missing. Do I need to take extra steps to actually add it to the path after installing? Putting it in usr/bin is not enough?

What I did:

I forked your github, added java and gradle to the dockerfile, built it successfully and then put it on docker hub: chazarabriseis/amplifybuildimage:python38java11gradle6

This is what I added to the amplifybuildimage:python38 dockerfile:

( following https://tecadmin.net/install-java-on-amazon-linux/ ) RUN amazon-linux-extras install java-openjdk11

( following https://linuxize.com/post/how-to-install-gradle-on-centos-7/ ) RUN wget https://services.gradle.org/distributions/gradle-6.7-bin.zip -P /tmp && \ unzip -d /gradle /tmp/gradle-6.7-bin.zip && \ mv /gradle /usr/bin/gradle && \ rm -rf /tmp/gradle-6.7-bin.zip

Thanks in advance for comments!

wvidana commented 3 years ago

@chazarabriseis without the output from the build nor trying it myself, I can only guess, but it looks like:

You are not updating java alternatives, which means that even if Java 11 is installed, the command java is pointing to another version. From the link you shared, check the step 3 https://tecadmin.net/install-java-on-amazon-linux/ I assume you can do something similar of what I did for Python 3.8 https://github.com/wvidana/amplify-console/blob/0e5432213a0d999e5fa53539ac707b2ed31c054a/images/latest/Dockerfile#L79-L81
Maybe you need to set the environment variables to set the gradle version and path, on step 3 https://linuxize.com/post/how-to-install-gradle-on-centos-7/#3-setup-environment-variables

chazarabriseis commented 3 years ago

Update: Moved this problem below to a new issue: https://github.com/aws-amplify/amplify-console/issues/1243

Thanks @wvidana for your fast reply! I was trying out stuff and I got the java installation working now by installing it from a wget and setting the path at the end. I also set the path for gradle and it finds it now but complains about the version: Update Gradle to >=5. Download link: https://bit.ly/3aGYDj6

Not sure what could go wrong there since I am installing and pointing to verison 6.7 and if asking on a local image build for the gradle verison it is 6.7. Any idea what could go wrong here? @wvidana do you think I should start a new issue for this since it is not a problem with the python version? I really don't know what could be the problem here and that way other people could help?

The changes to the wvidana/amplifybuildimage:python38 Dockerfile and then pushed to chazarabriseis/amplifybuildimage:python38java11gradle6:

RUN wget https://corretto.aws/downloads/latest/amazon-corretto-11-x64-linux-jdk.tar.gz && \ tar -xf amazon-corretto-11-x64-linux-jdk.tar.gz amazon-corretto-11.0.9.12.1-linux-x64 -C / && \ mv /amazon-corretto-11.0.9.12.1-linux-x64 /usr/bin/java && \ rm -rf amazon-corretto-11-x64-linux-jdk.tar.gz

RUN wget https://services.gradle.org/distributions/gradle-6.7-bin.zip -P /tmp && unzip -d /gradle /tmp/gradle-6.7-bin.zip && mv /gradle /usr/bin/gradle && rm -rf /tmp/gradle-6.7-bin.zip

Environment Setup

RUN echo export PATH="\ /usr/local/rvm/gems/ruby-${VERSION_RUBY_DEFAULT}/bin:\ /usr/local/rvm/gems/ruby-${VERSION_RUBY_DEFAULT}@global/bin:\ /usr/local/rvm/rubies/ruby-${VERSION_RUBY_DEFAULT}/bin:\ /usr/bin/java/bin:\ /usr/bin/gradle/gradle-6.7/bin:\ /usr/local/rvm/bin:\ /root/.yarn/bin:\ /root/.config/yarn/global/node_modules/.bin:\ /root/.nvm/versions/node/${VERSION_NODE_DEFAULT}/bin:\ $(python3 -m site --user-base)/bin:\ $PATH" >> ~/.bashrc && \ echo export GEM_PATH="/usr/local/rvm/gems/ruby-${VERSION_RUBY_DEFAULT}" >> ~/.bashrc && \ echo "nvm use ${VERSION_NODE_DEFAULT} 1> /dev/null" >> ~/.bashrc

nirkra commented 3 years ago

proper resolution:

version: 1
backend:
  phases:
    build:
      commands:
        - amazon-linux-extras install python3.8
        - ln -fs /usr/bin/python3.8 /usr/bin/python3
        - pip3.8 install pipenv
        - amplifyPush --simple

poulter7 commented 3 years ago

Does this installation on the fly approach dramatically increase build times though?

On Thu, Nov 19, 2020 at 11:00 AM Nir Krakowski notifications@github.com wrote:

proper resolution:

version: 1 backend: phases: build: commands:

amazon-linux-extras install python3.8

ln -fs /usr/bin/python3.8 /usr/bin/python3

pip3.8 install pipenv

amplifyPush --simple

frontend: phases: preBuild: commands:

npm ci build: commands:

npm run build artifacts: baseDirectory: dist files:

'*/' cache: paths:

node_modules/*/

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/aws-amplify/amplify-console/issues/595#issuecomment-730471130, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB2OYQ25NHYSDR4V3SVNDTSQU6J7ANCNFSM4MJLW3IQ .

nirkra commented 3 years ago

yes, and you get to stay on the latest image. which should've had 3.8, but for some reason only has 3.7

johspaeth commented 3 years ago

The solution by @nirkra worked for me. Thanks for sharing!

subtlegiant commented 3 years ago

Worked for me as well, thanks @nirkra

litwicki commented 3 years ago

In addition to @nirkra comment above, we have also updated our default build image with the latest version of python. Please reopen this issue if you notice anything working incorrectly. Thanks!

andre-scheffer commented 3 years ago

@litwicki the current version of the build image is still 3.7.9:

Since I am not a contributor I'm not able to reopen this issue.

armonge commented 3 years ago

After the new image update i got my python builds working with

alternatives --install /usr/bin/python3 python3 /usr/local/bin/python3.8 10

python --version
python2 --version
python3 --version

That produces the following output

2020-12-10T10:52:32.860Z [WARNING]: + alternatives --install /usr/bin/python3 python3 /usr/local/bin/python3.8 10
2020-12-10T10:52:32.862Z [WARNING]: + python --version
2020-12-10T10:52:32.863Z [WARNING]: Python 2.7.18
                                    + python2 --version
2020-12-10T10:52:32.865Z [WARNING]: Python 2.7.18
2020-12-10T10:52:32.865Z [WARNING]: + python3 --version
2020-12-10T10:52:33.007Z [INFO]: Python 3.8.0

litwicki commented 3 years ago

@andre-scheffer is https://github.com/aws-amplify/amplify-console/issues/595#issuecomment-745174724 an option for you?

andre-scheffer commented 3 years ago

@litwicki I already deploy with the installation of Python 3.8 via script. But you said in your previous post that Amplify is using the current Python version and that we should contact you if this is not the case. To me it still says that Python 3.7.9 is used if I deploy without an installation script.

abhi7cr commented 3 years ago

@andre-scheffer the latest build image has both python3 (python 3.7.9) and python3.8 (python 3.8) installed to be backwards compatible . Could you try running under the python3.8 cli command. You can check the exact version of python 3.8 by running python3.8 --version

SJI-ParthaSarathi commented 3 years ago

This saved my day! @pchevallier brother you are awesome! <3

wvidana commented 3 years ago

The new installation is a bit messy: all python binaries are under /usr/bin but the new python 3.8 is on /usr/local/bin, which took me several tries with convinations of update-alternatives and pip install to get it right. Also /usr/bin/pip3 is not a link but the binary from the 3.7 installation, so it can't be override with update-alternatives...

If this is the definitive solution then it should be clearly stated somewhere, so the people using python 3.8 functions can setup their amplify.yml correctly.

These lines gave me a succesfull backend deploy using the default image provided by AWS

version: 1
backend:
  phases:
    build:
      commands:
        - update-alternatives --install /usr/bin/python3 python3 /usr/local/bin/python3.8 11
        - /usr/local/bin/pip3.8 install --user pipenv
        - amplifyPush --simple

litwicki commented 3 years ago

@wvidana Sorry for the delayed response, but thank you for your feedback. We're going to review this and will follow up.

SJI-ParthaSarathi commented 3 years ago

@jake This is really frustrating that Amplify can't find python3 environment path on Windows 10 even if it is added! it only happens when I run Amplify push command :-(

On Thu, Jan 21, 2021 at 12:51 PM Jake Litwicki notifications@github.com wrote:

@wvidana https://github.com/wvidana Sorry for the delayed response, but thank you for your feedback. We're going to review this and will follow up.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/aws-amplify/amplify-console/issues/595#issuecomment-764424064, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMO6G7KNL7GU6ZFUN6J5X4LS27FHRANCNFSM4MJLW3IQ .

-- Partha Sarathi Kundu Software Development Engineer at SJ Innovation LLC http://sjinnovation.com/ Where employee happiness generates client success Get me in touch on LinkedIn https://www.linkedin.com/in/partha-sarathi-kundu/ Follow SJ Innovation on Facebook https://www.facebook.com/sjinnovation/ or Twitter https://twitter.com/sjinnovation

abhi7cr commented 3 years ago

we have updated our FAQ with the commands posted by @wvidana, and will update our docs shortly. Please let us know if you are still facing issues.

michaelbrewer commented 3 years ago

we have updated our FAQ with the commands posted by @wvidana, and will update our docs shortly. Please let us know if you are still facing issues.

Looking forward to this being the default behavior

glynjackson commented 3 years ago

March 17th 2021 and this is still an issue! So frustrating, I really want to like Amplify this should be default behavior by now, right?

The workaround works for me by modifying the build script to create symlink to link python3 to python3.8. Thank you to @wvidana who posted this, saved me a lot of time

swaminator commented 3 years ago

Please refer to: https://github.com/aws-amplify/amplify-console/blob/master/FAQ.md#how-do-i-run-amplify-functions-with-python-runtime

aws-amplify / amplify-hosting

Amplify can't find Python3.8 on build phase of CI/CD #595

Java path problem

What I did:

Environment Setup