search

cossacklabs / themis

Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.
https://www.cossacklabs.com/themis
Apache License 2.0
1.87k stars 143 forks source link

Docker: Unable to locate package libthemis-dev #969

Open albertbuchard opened 2 years ago

albertbuchard commented 2 years ago

Describe the bug When trying to dockerize themis using debian (buster or bullseye) - the dockerization stops at the installation of themis with error: Unable to locate package libthemis-dev

To Reproduce Steps to reproduce the behavior:

  1. Create Dockerfile 
    
FROM node:18-buster
RUN apt update && \
apt -y install apt-transport-https

Install themis

RUN wget -qO - https://pkgs-ce.cossacklabs.com/gpg | apt-key add - RUN echo "deb https://pkgs-ce.cossacklabs.com/stable/debian buster main" | tee /etc/apt/sources.list.d/cossacklabs.list RUN apt update && apt install -y libthemis-dev

2. Run `docker build .`

error

[ 5/11] RUN apt update && apt install -y libthemis-dev:

9 0.223

9 0.223 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

9 0.223

9 0.326 Hit:1 http://deb.debian.org/debian buster InRelease

9 0.330 Hit:2 http://deb.debian.org/debian-security buster/updates InRelease

9 0.342 Hit:3 http://deb.debian.org/debian buster-updates InRelease

9 0.350 Get:4 https://pkgs-ce.cossacklabs.com/stable/debian buster InRelease [1709 B]

9 0.816 Fetched 1709 B in 1s (3049 B/s)

9 0.816 Reading package lists...

9 1.159 Building dependency tree...

9 1.223 Reading state information...

9 1.235 21 packages can be upgraded. Run 'apt list --upgradable' to see them.

9 1.241

9 1.241 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

9 1.241

9 1.246 Reading package lists...

9 1.595 Building dependency tree...

9 1.666 Reading state information...

9 1.715 E: Unable to locate package libthemis-dev

executor failed running [/bin/sh -c apt update && apt install -y libthemis-dev]: exit code: 100

Expected behavior Should find libthemis-dev

Environment (please complete the following information):

Additional context .

shadinua commented 1 year ago

Hi, @albertbuchard!

Thank you for the detailed description of the problem. We have built and added packages for the arm64 architecture and tested according to the process you have described above.

Could you please check if it solve the issue and confirm that?

yen936 commented 1 year ago

Hi @shadinua,

I have the exact same issue bug report as @albertbuchard.

Am I using the wrong base image? Thank you so much for your help.

Describe the bug Docker image fails to build

To Reproduce Steps to reproduce the behavior:

  1. Here is my docker file 
    
ARG FUNCTION_DIR="/function"

FROM python:3.10 as build-image

Include global arg in this stage of the build

ARG FUNCTION_DIR

Copy function code

RUN mkdir -p ${FUNCTION_DIR} COPY . ${FUNCTION_DIR}

Install the function's dependencies

RUN pip install \ --target ${FUNCTION_DIR} \ awslambdaric

RUN apt update && \ apt -y install apt-transport-https

Install themis

RUN wget -qO - https://pkgs-ce.cossacklabs.com/gpg | apt-key add - RUN echo "deb https://pkgs-ce.cossacklabs.com/stable/debian buster main" | tee /etc/apt/sources.list.d/cossacklabs.list RUN apt update && apt install -y libthemis-dev

Use a slim version of the base Python image to reduce the final image size

FROM python:3.10-slim

Include global arg in this stage of the build

ARG FUNCTION_DIR

Set working directory to function root directory

WORKDIR ${FUNCTION_DIR}

Copy in the built dependencies

COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}

Set runtime interface client as default command for the container runtime

ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]

Pass the name of the function handler as an argument to the runtime

CMD [ "lambda_function.handler" ]

3. Run `docker build -t docker-image:test .`
4. See the following error:

dockerfile:24

22 | RUN wget -qO - https://pkgs-ce.cossacklabs.com/gpg | apt-key add - 23 | RUN echo "deb https://pkgs-ce.cossacklabs.com/stable/debian buster main" | tee /etc/apt/sources.list.d/cossacklabs.list 24 | >>> RUN apt update && apt install -y libthemis-dev 25 |
26 | # Use a slim version of the base Python image to reduce the final image size

ERROR: failed to solve: process "/bin/sh -c apt update && apt install -y libthemis-dev" did not complete successfully: exit code: 100



**Expected behavior**
Docker to build

**Environment (please complete the following information):**
- OS: OSX 13.3.1 Ventura 
- Hardware: 64-bit Mac
- Themis version:  0.15.0 Newest
- Installation way:
   - [ ] via package manager
   - [X] built from docker

**Additional context**
Add any other relevant context for the problem here. Share an example project, if you can.
Lagovas commented 1 year ago

python:3.10 image uses debian:bookworm as base image that is not supported yet. please, try to use python:3.10-bullseye

yen936 commented 1 year ago 
FROM public.ecr.aws/lambda/python:3.10

# Copy requirements.txt
COPY requirements.txt ${LAMBDA_TASK_ROOT}

# Copy function code
COPY lambda_function.py ${LAMBDA_TASK_ROOT}

RUN yum update -y

# RUN yum install -y httpd

RUN rpm --import https://pkgs-ce.cossacklabs.com/gpg
RUN wget -qO - https://pkgs-ce.cossacklabs.com/stable/centos/cossacklabs.repo | tee /etc/yum.repos.d/cossacklabs.repo
RUN yum install libthemis-devel

# Install the specified packages
RUN pip install -r requirements.txt

# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
CMD [ "lambda_function.handler" ]

that worked thank you @Lagovas.

now I need to transition the dockerfile to one of AWS's base images. This image is Amazon Linux based on RedHat to my knowledge. Do you have an idea why this fails?

I am not clear why yum cannot find libthemis-devel


[+] Building 4.4s (11/12)                                                                                                                                                                                   
 => [internal] load build definition from dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 2.05kB                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                        0.0s
 => [internal] load metadata for public.ecr.aws/lambda/python:3.10                                                                                                                                     0.9s
 => [1/8] FROM public.ecr.aws/lambda/python:3.10@sha256:19b2ea7fc3484f112f63b8dbc836847c1cd5ac41cf0305af8fdfa09b4470c9b6                                                                               0.0s
 => [internal] load build context                                                                                                                                                                      0.0s
 => => transferring context: 75B                                                                                                                                                                       0.0s
 => CACHED [2/8] COPY requirements.txt /var/task                                                                                                                                                       0.0s
 => CACHED [3/8] COPY lambda_function.py /var/task                                                                                                                                                     0.0s
 => CACHED [4/8] RUN yum update -y                                                                                                                                                                     0.0s
 => [5/8] RUN rpm --import https://pkgs-ce.cossacklabs.com/gpg                                                                                                                                         1.4s
 => [6/8] RUN wget -qO - https://pkgs-ce.cossacklabs.com/stable/centos/cossacklabs.repo | tee /etc/yum.repos.d/cossacklabs.repo                                                                        0.3s
 => ERROR [7/8] RUN yum install libthemis-devel                                                                                                                                                        1.8s
------                                                                                                                                                                                                      
 > [7/8] RUN yum install libthemis-devel:                                                                                                                                                                   
#0 0.389 Loaded plugins: ovl
#0 1.683 No package libthemis-devel available.
#0 1.801 Error: Nothing to do
------
dockerfile:15
--------------------
  13 |     RUN rpm --import https://pkgs-ce.cossacklabs.com/gpg
  14 |     RUN wget -qO - https://pkgs-ce.cossacklabs.com/stable/centos/cossacklabs.repo | tee /etc/yum.repos.d/cossacklabs.repo
  15 | >>> RUN yum install libthemis-devel
  16 |     
  17 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c yum install libthemis-devel" did not complete successfully: exit code: 1
Lagovas commented 1 year ago

I'm unfamiliar with AWS and centos too much, and I'm not sure what repo structure AWS expects. We support, test and maintain repositories for centos 7/8, and temporarily latest version deployed only for x86_64. You can try to download .rpm manually from https://pkgs-ce.cossacklabs.com/stable/centos/8/x86_64/ and try to install it locally from the file. It may works if AWS expects different repo structure in comparison with centos 7/8 but okay with .rpm packages packed for them. And it may work only for x86_64, not arm64.

As I see, base image uses scratch base image and install FS structure from archives - https://github.com/aws/aws-lambda-base-images/blob/python3.10/Dockerfile.python3.10 . So, it needs a time to investigate and debug.

yen936 commented 1 year ago

Thanks for the response--and thanks for working on this lib. Yes, the base image was the key. By pivoting into the base image that you mentioned I finally got it, thank God. So, here's what I did for future reference:

# Define custom function directory
ARG FUNCTION_DIR="/function"

FROM python:3.10-bullseye as build-image

# Include global arg in this stage of the build
ARG FUNCTION_DIR

# Copy function code
RUN mkdir -p ${FUNCTION_DIR}
COPY . ${FUNCTION_DIR}

# Install the function's dependencies
RUN pip install --target ${FUNCTION_DIR} awslambdaric boto3 botocore pythemis

RUN apt update && apt install -y apt-transport-https wget gnupg

# Install themis
RUN wget -qO - https://pkgs-ce.cossacklabs.com/gpg | apt-key add -
RUN echo "deb https://pkgs-ce.cossacklabs.com/stable/debian bullseye main" | tee /etc/apt/sources.list.d/cossacklabs.list
RUN apt update && apt install -y libthemis-dev

WORKDIR ${FUNCTION_DIR}