actions / setup-python

Set up your GitHub Actions workflow with a specific version of Python
MIT License
1.66k stars 532 forks source link

pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available. #93

Closed utapyngo closed 2 years ago

utapyngo commented 4 years ago

Trying to use Python 3.6 on Ubuntu 16.04:

Execute installation script
##[debug]Check if Python hostedtoolcache folder exist...
##[debug]Deleting Python 3.6.10
##[debug]Create Python 3.6.10 folder
##[debug]Copy Python binaries to hostedtoolcache folder
##[debug]Create additional symlinks (Required for UsePythonVersion VSTS task)
##[debug]Upgrading PIP...
##[debug]Looking in links: /tmp/tmp5xjr9lqz
##[debug]Requirement already satisfied: setuptools in /opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages (40.6.2)
##[debug]Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages (18.1)
##[debug]Collecting pip
##[debug]Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)) - skipping
##[debug]Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)) - skipping
##[error]The process '/bin/bash' failed with exit code 1
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Set up Python
konradpabjan commented 4 years ago

Could you provide a minimal repo? Not sure if this has been fixed or maybe there is some extra configurations or perhaps you're using a self-hosted runner?

I tried the following and it everything seems to be working.

    runs-on: ubuntu-16.04
    steps:
    - name: Setup-Python Test
      uses: actions/setup-python@v2
      with:
        python-version: 3.6.10 (also tried with 3.6 as that defaults to 3.6.10)
utapyngo commented 4 years ago

We are using a self-hosted runner. I have kind of fixed this by compiling openssl1.1 and specifying the path in an environment variable. Now the problem is that I must do it manually for every Python version because the Github Actions runner does not work with openssl1.1.

konradpabjan commented 4 years ago

Yup, it's failing towards the end of the the setup script, ssl related: https://github.com/actions/python-versions/blob/10f5e8e4f5b3118624f86839a5c64c6e095094f7/installers/nix-setup-template.sh#L54

If you try to do ./python -m pip install --ignore-installed pip locally, it spits out the same message. After the extensive testing with other machines, I got this to show up once. I used a combination of solutions outlined here: https://stackoverflow.com/questions/45954528/pip-is-configured-with-locations-that-require-tls-ssl-however-the-ssl-module-in/57240184 (did you use something similar?) Seems like this is highly dependent on the self-hosted machine and what is installed already.

I don't know of a good solution at this time... we could update the README with some instructions, but ideally I would like it to just work. Another option is to maybe update the setup-templates which I think should do the trick. During compile time, we have all the SSL modules installed beforehand, so I there isn't anything else we can do: https://github.com/actions/python-versions/blob/10f5e8e4f5b3118624f86839a5c64c6e095094f7/builders/ubuntu-python-builder.psm1#L68

jakemwood commented 4 years ago

We are also experiencing problems with this on a self-hosted runner. Has this been solved yet?

Upgrading PIP...
Looking in links: /tmp/tmpopvlgd1t
Requirement already satisfied: setuptools in /opt/hostedtoolcache/Python/3.7.8/x64/lib/python3.7/site-packages (47.1.0)
Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.7.8/x64/lib/python3.7/site-packages (20.1.1)
##[error]WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
##[error]WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
##[error]WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
##[error]WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
##[error]WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
##[error]WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
ERROR: Could not find a version that satisfies the requirement pip (from versions: none)
ERROR: No matching distribution found for pip
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
##[error]WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
##[error]The process '/bin/bash' failed with exit code 1
debtha commented 4 years ago

Has anyone found a resolution for this issue? I just upgraded from Python 2.7.5 to Python 3.8.1 and upgraded Pip and having this same issue.

EvanPHamilton commented 4 years ago

https://help.dreamhost.com/hc/en-us/articles/360001435926-Installing-OpenSSL-locally-under-your-username seems like it might help

utapyngo commented 4 years ago

@EvanPHamilton, that's what I did, and it helped. The problem is that the environment variables must be set individually for every Python version because the Github Actions runner does not work with openssl1.1.

BjornWF commented 4 years ago

@utapyngo , i'm trying to install tesseract-ocr on Windows 10 I need to install the "tesserocr-2.4 .0-cp37-cp37m-win-amd64.whl" wheel and then get the error

image

I set my environment variable for path for tesseract-ocr. Any suggestions? I would really appreciate any assistance, Regard bjorn

BjornWF commented 4 years ago

Has anyone found a resolution for this issue? I just upgraded from Python 2.7.5 to Python 3.8.1 and upgraded Pip and having this same issue.

@debtha , did you come right here?

mathangpeddi commented 3 years ago

Screenshot (582)

I am trying to install flask in Windows and it shows a similar error. Any idea how to resolve this error? Thanks in advance!

chasays commented 3 years ago

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.
l1771250640 commented 3 years ago

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.

thanks, resolved.

kvamsi7 commented 3 years ago

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.

could you please elaborate on the second step, please ?

l1771250640 commented 3 years ago

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.

could you please elaborate on the second step, please ?

download python source from https://www.python.org/downloads/source/ such as Python-x.x.x.tgz, then tar -xvf Python-x.x.x.tgz, and cd Python-x.x.x, do /configure --with-openssl=/home/username/openssl.

zimventures commented 3 years ago

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.

I was able to get a version of python built to use a openssl in a home directory, however the setup-python doesn't seem to ever want to use that version. Even setting up the python and python3 symlink in /usr/bin still result it in trying to install its own version and failing. Any advice on how to force setup-python to use a specific python binary?

git8951 commented 3 years ago

resolved.

1. first install openssl, please refer to [this page](https://help.dreamhost.com/hc/en-us/articles/360001435926-Installing-OpenSSL-locally-under-your-username)

2. install python and `./configure --with-openssl=/home/username/openssl`

3. at last, run `python3 -m ssl` and nothing outputs, it's ok.

I install ssl and use same command , last command not working "python3 -m ssl" , same problem not resolved

ghost commented 3 years ago

I have the problem to not be able to find the bash_profile

I am using Kali Linux on VMWare, and it copied the vmware iso (not the installer, i don't know if there is a difference)

I did all others steps but I am guessing the change in the bash_profile is mandatory, unfortunately i cant do it

Can someone help ?

rlf89 commented 3 years ago

I have the problem to not be able to find the bash_profile

I am using Kali Linux on VMWare, and it copied the vmware iso (not the installer, i don't know if there is a difference)

I did all others steps but I am guessing the change in the bash_profile is mandatory, unfortunately i cant do it

Can someone help ?

try . ~/.profile instead

or editing bashrc

https://askubuntu.com/questions/510709/i-cannot-find-bash-profile-in-ubuntu

rlf89 commented 3 years ago

I managed to include SSL library with

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/username/openssl/lib/

before step ./configure --with-openssl=/home/username/openssl

now python3 -m ssl give no errors

awerchniak commented 3 years ago

I was encountering this problem while attempting to RUN pip install numpy in a docker image that extends one of the Amazon Sagemaker Tensorflow Serving images (reference), which runs Ubuntu 16.04. In case anyone encounters this in the future, the reason that this was failing for me is that my custom script was modifying the LD_LIBRARY_PATH environment variable. After moving the RUN pip install numpy call to before that change, it's working!

stmckeni commented 3 years ago

https://github.com/actions/setup-python/issues/93#issuecomment-716433622

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.

https://github.com/actions/setup-python/issues/93#issuecomment-770637487

I managed to include SSL library with

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/username/openssl/lib/

before step ./configure --with-openssl=/home/username/openssl

now python3 -m ssl give no errors

this worked with the openssl 1.1.1k version for python 3.9.5 on Linux lite ( or a custom kernel based on ubuntu 20.04) .

I had installed openssl source built it following the instructions from that link and then configured from python readme.rst with ./configure --with-openssl=/home/username/openssl --enable-optimizations followed the python make and build instructions and after python3 -m ssl I got no errors and pip worked. I even finally got to update to pip 21.1.2. These posted have helped me the most on a barely mainstream custom linux kernel thank you again everyone.

DiKorsch commented 3 years ago

The solution used here (https://bugs.python.org/issue34028#msg339455) helped me:

  1. I installed openssl with the default prefix (to /usr/local)
  2. Then I needed to set LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBARY_PATH:/usr/local/lib64:/usr/local/lib
  3. Afterwards, so that python configuration succeeds, I needed to set LD_FLAGS: export LDFLAGS="-L/usr/local/lib -L/usr/local/lib64 -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/local/lib64"
  4. Finally, after configuring with ./configure --with-openssl=/usr/local/ and installing with make && make install python3 -m ssl worked
arpanbaruah commented 3 years ago

If any body still facing issue Follow these steps install openssl with /home/username

set export PATH=$HOME/openssl/bin:$PATH export LD_LIBRARY_PATH=/home/username/openssl/lib export LC_ALL="en_US.UTF-8" export LDFLAGS="-L/home/username/openssl/lib -Wl,-rpath,/home/username/openssl/lib" export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/username/openssl/lib/

execute in Python folder ./configure -prefix=/home/username/Python38/ --with-openssl=/home/username/openssl make make install python3 -m ssl

rtadros125 commented 2 years ago

Nothing worked for me until I yum installed openssl-devel and bzip2-devel before building python on CentOS. Neither LD manipulation nor setting --with-openssl while configuring the build was necessary. The names of those packages for apt would be libssl-dev and libbz2-dev, but I have not tried it.

Huyassin commented 2 years ago

In Ubuntu 20.04 I fixed the similar error, trying to install Django by using super user sudo -s And # pip3 install django

leeco-22 commented 2 years ago

resolved.

  1. first install openssl, please refer to this page
  2. install python and ./configure --with-openssl=/home/username/openssl
  3. at last, run python3 -m ssl and nothing outputs, it's ok.

This is great resolved, thank bro

leeco-22 commented 2 years ago

The solution used here (https://bugs.python.org/issue34028#msg339455) helped me:

  1. I installed openssl with the default prefix (to /usr/local)
  2. Then I needed to set LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBARY_PATH:/usr/local/lib64:/usr/local/lib
  3. Afterwards, so that python configuration succeeds, I needed to set LD_FLAGS: export LDFLAGS="-L/usr/local/lib -L/usr/local/lib64 -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/local/lib64"
  4. Finally, after configuring with ./configure --with-openssl=/usr/local/ and installing with make && make install python3 -m ssl worked

I followed your guide but still got some error bro, "/usr/local/bin/python3.10: No module named openssl"

Lazik commented 2 years ago

Worked for me. I had to install openssl and then it worked like a charm.

qudos-tech commented 2 years ago

3. python3 -m ssl

working but still getting this error: WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available. Please advise...

IvanZosimov commented 2 years ago

Hi, @qudos-tech 👋 Could you provide us with some more details about your case in order to be able to reproduce it. You can use this template as example.

gezabohus commented 2 years ago

This is on CentOS 7.

Openssl-devel and bzip2-devel mentioned here are up to date.

I have installed Python 3.10 as root so everyone can use it. Got the failures mentioned in the subject when trying to do

python3 -m pip install requests

Installed openssl for myself as suggested above and I still get the same errors when trying to install python packages.

python3 -m ssl

results in

Traceback (most recent call last): File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.10/ssl.py", line 98, in import _ssl # if we can't import it, let the error propagate ModuleNotFoundError: No module named '_ssl'

panticmilos commented 2 years ago

Hi @gezabohus,

Your case is a bit specific because you are using CentOS compared to other self-hosted runners. The issue here is that prebuilt Python binaries are prebuilt for Ubuntu so we cannot guarantee this Python build would work on other distributions. If you would like to build Python on your own for CentOS there is no need to use the setup-python action, since this action is using prebuilt binaries as explained above. Your issue is not related to setup-python action, probably you can open a topic on the python bug tracker: https://bugs.python.org or https://github.com/python/cpython/issues.

Cheers

gezabohus commented 2 years ago

Hi @panticmilos,

Thanks for the info.

I wasn't completely clear, sorry. When I wrote "installed" I meant going through the get tar - configure - make cycle. I suppose that should build from source. Right?

panticmilos commented 2 years ago

Hi @gezabohus,

You are welcome. Yes, that is correct, these are building phases. And since you are building it on your own, that's why I think raising the issue on one of these two bug trackers could help you. If I can help you additionally feel free to continue the discussion. :)

gezabohus commented 2 years ago

Hi @panticmilos,

Got it. Thanks for the info.

gezabohus commented 2 years ago

Hi all,

A bit of fudging with this answer solved my problem. (Plus using virtual env.)

shuoyinn commented 2 years ago

Thanks a lot for all these answers. For me, I also found that it also worked even after I removed the openssl installed in my home directory and cancel export LD_LIBRARY_PATH for it's lib. So I just tried many other times for digging into it. And I summarize as follows:

Firstly. I installed openssl in my home directory and set its related LD_LIBRARY_PATH before I installed python3.9 from source code with ./configure --with-openssl={openssl path I just installed manually}. And ./pip3 install worked without any SSL error.

Afterwards, I removed the openssl and unset the environment variable for it, but still no SSL error when I use ./pip3 to install packages.

Another time, I remove all the things mentioned above and install openssl again. This time I didn't set its LD_LIBRARY_PATH and only set --with-openssl={openssl path I just installed manually} and then install python3.9 before I removed openssl again. So there is no manually installed openssl any more. Still, it worked.

In conclusion, I guess the procedure of 'configure' python3.9 needs an openssl directory and only needs for 'configure', for some reason it cannot find one (I mean it cannot find the default openssl the system already has). So just install and set one explicitly. After configuring, making and installing python, it is of no use for normally running python and pip.

I encountered this problem on one of my lab servers (labeled as 1), but everything went well on another (labeled as 0). I cannot find where is different between them since both default openssl are on /usr (but for 1 is on /usr/local/bin and for 0 on /usr/bin).

dsame commented 2 years ago

hello @utapyngo

With the fact that self-hosted environment must have openssl libraries installed and with the solution provided @chasays i believe the issue can be closed. But please feel free to reopen this issue or create another one in case if the problem still exists.

clh15683 commented 1 year ago

On Ubuntu (tested on 22.04) it is sufficient to install libssl-dev and then rerun configure. There is no need to install a local openssl.

bozimmerman commented 1 year ago

On Ubuntu (tested on 22.04) it is sufficient to install libssl-dev and then rerun configure. There is no need to install a local openssl.

If re-running configure on a machine that already has the required libs is all that's needed, can that be added to setup-python? I feel like almost none of the comments are focusing on the actual Action we are supposedly trying to use.

BlackFuffey commented 1 year ago

resolved.

1. first install openssl, please refer to [this page](https://help.dreamhost.com/hc/en-us/articles/360001435926-Installing-OpenSSL-locally-under-your-username)

2. install python and `./configure --with-openssl=/home/username/openssl`

3. at last, run `python3 -m ssl` and nothing outputs, it's ok.

Thank you very much this pretty much saved my life :D

asanc115 commented 1 year ago

Got this working for installing Python 3.11.2 on Amazon Linux 2. (An OS Similar to CentOS 7/Rhel 7) I had to install a different openssl version, which Amazon Repo's provide. You can still leave the regular openssl rpm installed, but you must REMOVE openssl-devel if installed. The three packages you must install AFTER REMOVING openssl-devel are:

Once installed just run ./configure --enable-optimizations (no need for the --with-openssl as the configure script already has a defined locations to look for your openssl, you only need to provide it if your openssl is not within those standard directories, you can find the locations in the script)

Do not let the g fool you, amazon updates their RPM's without changing the letter at the end, so yes it's patched for vulnerabilities.

Reference: https://alas.aws.amazon.com/AL2/ALAS-2023-1934.html

Hope this helps!

Also some more info if interested:

Everytime you run ./configure it creates a config.log. I saw it was finding my original openssl in the config.log, however, it was not happy with the version I was using, and the capabilities of that version based on some tests it would do. Which is what led me to find out there was a different openssl Amazon provided in which python3.11.2 was happy with.

miaogen123 commented 1 year ago

I managed to include SSL library with

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/username/openssl/lib/

before step ./configure --with-openssl=/home/username/openssl

now python3 -m ssl give no errors

this solution works for me too!

DDvO commented 11 months ago

The reason can also be that the OpenSSL installation (e.g, in /usr/local/lib) is messed up. Invoking python and in the interactive shell import ssl reveals where it tries to find libssl. See also https://stackoverflow.com/questions/58280484/ssl-module-in-python-is-not-available-on-osx

enahwe commented 7 months ago

Those commands will fix the issue:

$ sudo yum install yum-utils
$ sudo yum-builddep python3

or

$ sudo apt build-dep python3
$ sudo apt install pkg-config

or

$ sudo dnf install dnf-plugins-core
$ sudo dnf builddep python3

See: https://devguide.python.org/getting-started/setup-building/index.html#linux

zackees commented 6 months ago

In case anyone is running into this issue with the amazonlinux:2 docker image, here is a Dockerfile that allows python 3.11.8 to be compiled and installed, by first building openssl from source and setting the proper link flags for the python build.

Thanks to everyone that's contributed to working through this issue. Collectively I was able to piece the proper magic settings to get it working. And right when I was almost about to give up!

# Use an official Amazon Linux 2 base image
FROM amazonlinux:2
# Notes: amazonlinux:2023 uses php 8.1, and does not support
# php 7.2-7.4.

# Install Apache, PHP, and other necessary packages
RUN yum update -y \
    && amazon-linux-extras enable php7.2 \
    && yum install -y \
    # Install commands.
    httpd \
    openssh-server \
    cronie \
    sudo \
    dos2unix \
    which \
    # Optimnizes yum install
    deltarpm \
    procps \
    nano \
    os-prober \
    mariadb-server \
    bash-completion \
    ppp \
    postfix \
    # Install nohup which needs the coreutils
    coreutils nohup \
    # Install php
    php \
    php-cli \
    php-fpm \
    php-bcmath \
    php-bz2 \
    php-calendar \
    php-ctype \
    php-curl \
    php-date \
    php-dom \
    php-exif \
    php-fileinfo \
    php-filter \
    php-ftp \
    php-gd \
    php-gettext \
    php-hash \
    php-iconv \
    php-intl \
    php-json \
    php-libxml \
    php-mbstring \
    php-mysqli \
    php-mysqlnd \
    php-openssl \
    php-pcntl \
    php-pcre \
    php-pdo \
    php-pdo_mysql \
    php-pdo_pgsql \
    php-pdo_sqlite \
    php-pgsql \
    php-phar \
    php-posix \
    php-readline \
    php-reflection \
    php-session \
    php-shmop \
    php-simplexml \
    php-sockets \
    php-spl \
    php-sqlite3 \
    php-standard \
    php-sysvmsg \
    php-sysvsem \
    php-sysvshm \
    php-tokenizer \
    php-xml \
    php-xmlreader \
    php-xmlwriter \
    php-xsl \
    php-zip \
    php-zlib \
    mod_ssl \
    # Don't leave garbage behind, we are a container so we won't need it.
    && yum clean all

# More php packages, todo: merge above with below.
RUN yum install -y php-{pear,cgi,common,curl,mbstring,gd,mysqlnd,gettext,bcmath,json,xml,fpm,intl,zip,imap} && yum clean all

# Install C++ toolchain for building native add-ons for Node.js/python. Note
# that this must be done here while python is python 2.7. After python 3 is install
# this command will fail to work.
RUN yum install -y gcc44 gcc-c++ libgcc44 cmake wget tar gzip make && yum clean all
RUN yum install -y @development zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel xz xz-devel libffi-devel openssl-devel && yum clean all
# RUN yum install -y compat-openssl10-devel
RUN yum install -y openssl-devel yum-utils ncurses-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel expat-devel && yum clean all
RUN yum-builddep python3 -y

# Install local openssl

# wget https://www.openssl.org/source/openssl-1.1.1g.tar.gz
RUN wget https://www.openssl.org/source/openssl-1.1.1g.tar.gz
RUN tar -xvzf openssl-1.1.1g.tar.gz
RUN cd openssl-1.1.1g && ./config --prefix=/root/openssl --openssldir=/root/openssl no-ssl2 && make && make install
RUN yum groupinstall "Development Tools"

# Install python 3.9.6
# As of right now we cannot build any python version higher than 3.9.6
ARG PYTHON_MAJOR_VERSION=3
ARG PYTHON_MINOR_VERSION=11
ARG PYTHON_PATCH_VERSION=8
ARG PYTHON_VERSION=${PYTHON_MAJOR_VERSION}.${PYTHON_MINOR_VERSION}.${PYTHON_PATCH_VERSION}
ARG PYTHON_SHORT_VERSION=${PYTHON_MAJOR_VERSION}.${PYTHON_MINOR_VERSION}
WORKDIR /opt
RUN wget https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz
RUN tar xzf Python-${PYTHON_VERSION}.tgz
WORKDIR /opt/Python-${PYTHON_VERSION}

ENV LD_LIBRARY_PATH="/root/openssl/lib:/usr/local/lib64:/usr/local/lib"
ENV LDFLAGS="-L/root/openssl/lib -L/usr/local/lib -L/usr/local/lib64 -Wl,-rpath,/root/openssl/lib -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/local/lib64"
ENV CPPFLAGS="-I/usr/local/include"
RUN ./configure --enable-optimizations --with-openssl=/root/openssl
# RUN ./configure --enable-optimizations
RUN make altinstall
RUN rm -f /opt/Python-${PYTHON_VERSION}.tgz
RUN rm /usr/bin/python3 || true
RUN ln -s /opt/Python-${PYTHON_VERSION}/python /usr/bin/python3
RUN rm /usr/bin/pip3 || true
RUN ln -s /usr/local/bin/pip${PYTHON_SHORT_VERSION} /usr/bin/pip3
WORKDIR /

# Configure python environment
RUN pip3 install --upgrade pip
RUN pip3 install setuptools wheel
RUN pip3 install cryptography==41.0.5
RUN pip3 install wormhole-tx

# Ensure /usr/bin and /usr/sbin (where yum is located) are in the PATH
ENV PATH="/usr/bin:/usr/sbin:${PATH}"

# INSTALL NODE.js AND NPM
# Note that the highest version of Node we can install is 17, because of our version of
# glibc. Fixing this requires a different base image (not amazon linux 2).
ARG NODE_VERSION=17
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash \
    && . ~/.nvm/nvm.sh \
    && nvm install $NODE_VERSION \
    && nvm alias default $NODE_VERSION \
    && nvm use default \
    && echo 'export NVM_DIR="$HOME/.nvm"' >> /root/.bashrc \
    && echo '[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"' >> /root/.bashrc \
    && echo '[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"' >> /root/.bashrc \
    && echo 'export PATH="$NVM_DIR/versions/node/$(nvm version)/bin:$PATH"' >> /root/.bashrc \
    && source /root/.bashrc

# Install npm packages
RUN . ~/.bashrc && npm install --location=global pm2 http-server

# Install ripgrep
RUN curl -LO https://github.com/BurntSushi/ripgrep/releases/download/13.0.0/ripgrep-13.0.0-x86_64-unknown-linux-musl.tar.gz \
    && tar xvf ripgrep-13.0.0-x86_64-unknown-linux-musl.tar.gz \
    && mv ripgrep-13.0.0-x86_64-unknown-linux-musl/rg /usr/local/bin/ \
    && rm -rf ripgrep-13.0.0-x86_64-unknown-linux-musl.tar.gz ripgrep-13.0.0-x86_64-unknown-linux-musl

# TODO Move up to the top of the file.
RUN if ! id apache > /dev/null 2>&1; then echo "Apache user does not exist" && exit 1; fi
RUN mkdir -p /var/lib/php/sessions && chown apache:apache /var/lib/php/sessions

# Install glibc and AWS CLI v2
RUN yum install -y glibc
WORKDIR /tmp
RUN if [ "$(uname -m)" = "x86_64" ]; then \
    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"; \
    elif [ "$(uname -m)" = "aarch64" ]; then \
    curl "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" -o "awscliv2.zip"; \
    else \
    echo "Unsupported architecture"; \
    exit 1; \
    fi \
    && unzip awscliv2.zip \
    && sudo ./aws/install \
    && rm -rf /tmp/awscliv2.zip /tmp/aws

WORKDIR /root

# Ensure the AWS CLI binary is in the PATH
ENV PATH="/usr/local/aws-cli/aws/v2/current/bin:${PATH}"

# Copy AWS config file
COPY image_src/home/ec2-user/.aws /root/.aws

# Set the default value of BUILD_ENV to "test"
ARG BUILD_ENV=test
ENV BUILD_ENV=${BUILD_ENV}

# Setup SSH.
RUN mkdir -p /etc/ssh/host_keys/
# Copy SSH host keys
COPY ssh/test1.pem /etc/ssh/ssh_host_rsa_key
RUN chmod 600 /etc/ssh/ssh_host_rsa_key
COPY ssh/test1.pem /etc/ssh/host_keys/

ARG SSH_PASS=tssx2hJr229Xo8WpOyKmR0cZBqvl7n
ARG USE_SSHD=
# if USE_SSHD is set, then set it as an environment variable
ENV USE_SSHD=${USE_SSHD}

# Set the URL
ARG URL=
# if URL is set, then set it as an environment variable
ENV URL=${URL}
RUN \
    if [ -z "$URL" ]; \
    then echo "Error: URL environment variable is not set. Please set it to the current URL." && exit 1; \
    else echo "export URL=$URL" >> /root/.bashrc; \
    fi

# Set root password
RUN echo "root:${SSH_PASS}" | chpasswd

# COPY ENTIRE SOURCE DIRECTORY ONTO IMAGE.
COPY image_src/ /
# Apply dos2unix recursively to the copied files
RUN find /home/ -type f -name "*.sh" -exec dos2unix {} \;

# As far as I know, docker COPY commands are unconditional. So we copy the
# directory unconditionally and if the URL is localhost then we don't symlink
# but instead mount the twdb directory on the host machine to /var/www/twdb.
COPY --chown=apache:apache twdb/ /twdb/
# Note that linking is not workign and I don't know why, but directory moving
# seems to work just fine.
#RUN if [ "$URL" != "localhost" ]; then ln -s /twdb /var/www/twdb; fi
# move folder instaad
ARG USE_VOLUME=0
RUN if [ "$USE_VOLUME" != "1" ]; \
    # Move the copy of the twdb directory to /var/www/twdb \
    then mv /twdb /var/www/twdb; \
    # Otherwise remove the twdb directory, since the real twdb directory will be mounted \
    else rm -rf /twdb; \
    fi

COPY entrypoint /home/ec2-user/entrypoint
RUN find /home/ -name "entrypoint" -exec dos2unix {} \;
COPY jobs.config.js /home/ec2-user/jobs.config.js
COPY fmin_db_metrics_cron /home/ec2-user/fmin_db_metrics_cron
# COPY ./image_src/etc/cron.d /etc/cron.d
COPY docker /home/ec2-user/docker

# install twdb/python/requirments.txt
COPY twdb/python /var/www/twdb/python
RUN pip3 install /var/www/twdb/python

RUN /home/ec2-user/docker/final_install.sh

# Open necessary ports
EXPOSE 80 443 22
ENV PORT 80

# Start Apache and cron
CMD ["sh", "-c", "/home/ec2-user/entrypoint"]
enahwe commented 2 months ago

I found a really good and complete explanation bellow.

https://medium.com/@enahwe/how-to-06bc8a042345

adarshan-intel commented 1 month ago

The solution used here (https://bugs.python.org/issue34028#msg339455) helped me:

  1. I installed openssl with the default prefix (to /usr/local)
  2. Then I needed to set LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBARY_PATH:/usr/local/lib64:/usr/local/lib
  3. Afterwards, so that python configuration succeeds, I needed to set LD_FLAGS: export LDFLAGS="-L/usr/local/lib -L/usr/local/lib64 -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/local/lib64"
  4. Finally, after configuring with ./configure --with-openssl=/usr/local/ and installing with make && make install python3 -m ssl worked

This is the only comment that worked for me, Thanks a ton!!!!!!!