search

DriLLFreAK100 / codefee-works-api

Just an experimental API project written in Rust
9 stars 2 forks source link

dep conflict while building with docker #7

Open Kactus83 opened 1 year ago

Kactus83 commented 1 year ago

I was really interested in your backend model but I couldn't build the image with docker compose.

When I run the command, I get this in the terminal:

0 508.8 Compiling codefee-works-api v0.1.0 (/usr/src/codefee-works-api)

0 509.4 error[E0308]: mismatched types

0 509.4 --> src/configs/open_api.rs:45:71

0 509.4 |

0 509.4 45 | SwaggerUi::new("/swagger-ui/{_:.*}").url("/api-doc/openapi.json", ApiDoc::openapi())

0 509.4 | --- ^^^^^^^^^^^^^^^^^ expected struct utoipa::openapi::OpenApi, found a different struct utoipa::openapi::OpenApi

0 509.4 | |

0 509.4 | arguments to this function are incorrect

0 509.4 |

0 509.4 = note: struct utoipa::openapi::OpenApi and struct utoipa::openapi::OpenApi have similar names, but are actually distinct types

0 509.4 note: struct utoipa::openapi::OpenApi is defined in crate utoipa

0 509.4 --> /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/utoipa-2.4.2/src/openapi.rs:36:1

0 509.4 |

0 509.4 36 | / builder! {

0 509.4 37 | | /// # Examples

0 509.4 38 | | ///

0 509.4 39 | | /// Create [OpenApi] using [OpenApiBuilder].

0 509.4 ... |

0 509.4 114 | | }

0 509.4 115 | | }

0 509.4 | |_^

0 509.4 note: struct utoipa::openapi::OpenApi is defined in crate utoipa

0 509.4 --> /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/utoipa-3.0.3/src/openapi.rs:37:1

0 509.4 |

0 509.4 37 | / builder! {

0 509.4 38 | | /// # Examples

0 509.4 39 | | ///

0 509.4 40 | | /// Create [OpenApi] using [OpenApiBuilder].

0 509.4 ... |

0 509.4 115 | | }

0 509.4 116 | | }

0 509.4 | |_^

0 509.4 = note: perhaps two different versions of crate utoipa are being used?

0 509.4 note: associated function defined here

0 509.4 --> /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/utoipa-swagger-ui-3.0.2/src/lib.rs:226:12

0 509.4 |

0 509.4 226 | pub fn url<U: Into<Url<'static>>>(mut self, url: U, openapi: OpenApi) -> Self {

0 509.4 | ^^^

0 509.4 = note: this error originates in the macro builder (in Nightly builds, run with -Z macro-backtrace for more info)

0 509.4

0 509.6 For more information about this error, try rustc --explain E0308.

0 509.6 error: could not compile codefee-works-api due to previous error

0 511.8 error: failed to compile codefee-works-api v0.1.0 (/usr/src/codefee-works-api), intermediate artifacts can be found at /usr/src/codefee-works-api/target

failed to solve: executor failed running [/bin/sh -c cargo install --path .]: exit code: 101

I apologize in advance if this is a noob error but I did not find the solution ..

DriLLFreAK100 commented 1 year ago

Hey @Kactus83, thanks so much for pointing out the issue. It was due to some version conflicts with the utoipa crates apparently. I have resolved it and merged the changes into the main branch. Would you mind giving it a try and let me know if it works for you?

Kactus83 commented 1 year ago

Conflict with Utoipa is solved ! Next there is a error while copying libs from builder to debian image.

Libs seems to be in /usr/lib/x86_64-linux-gnu/ in the builder image so i'm trying to change COPY lines with this kind of path :

" COPY --from=build /usr/lib/x86_64-linux-gnu/libpq.so* /usr/lib/x86_64-linux-gnu/ "

I come back later with some good new (i hope !)

edit : I changed copy pathin the Dockerfile like this :

libpq related (required by diesel)

COPY --from=build /usr/lib/x86_64-linux-gnu/libpq.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libkrb5.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libk5crypto.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libkrb5support.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/liblber-2.4.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libsasl2.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libgnutls.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libp11-kit.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libidn2.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libunistring.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libtasn1.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libnettle.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libhogweed.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libgmp.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /usr/lib/x86_64-linux-gnu/libffi.so /usr/lib/x86_64-linux-gnu/ COPY --from=build /lib/x86_64-linux-gnu/libcom_err.so /lib/x86_64-linux-gnu/ COPY --from=build /lib/x86_64-linux-gnu/libkeyutils.so* /lib/x86_64-linux-gnu/

Build and compose are Ok now. I still have a little doubt because I also changed the path of the copy destination, and I don't know if it was a good thing!

DriLLFreAK100 commented 1 year ago

@Kactus83 Nice stuff, yep you are right on that. Basically, this is related to the build target architecture. I am using an M1 Mac here (arm64). I supposed you're on an amd64, i.e. x86_64 architecture here. Hence, the issue. This is very interesting, something new to me too. Haha.. Anyway, here's the PR for it. Feel free to give it a try.

Kactus83 commented 1 year ago

Thanks for the update! I tried working on a dynamic method this morning but couldn't get anything that is clear and functional. I found this which might interest you: https://docs.docker.com/engine/reference/builder/#automatic-platform-args-in-the-global-scope

I tried something like this:


# Use shell
SHELL ["/bin/bash", "-c"]

# Determine architecture from TARGETPLATFORM
ARG TARGETPLATFORM
# Set the environment variable based on the architecture
RUN if [ "$TARGETPLATFORM" = "linux/amd64" ]; then export ARCH=x86_64-linux-gnu; \
    elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then export ARCH=aarch64-linux-gnu; \
    else echo "Unsupported architecture" && exit 1; \
    fi
ENV ARCH=$ARCH

Several conclusions:

When putting it in the builder, I couldn't get the variable in the debian image, and the files ended up in a folder without a name. (but maybe I'm doing it wrong)

When putting it in the second stage, the lack of a shell caused a problem and prevented me from using "case" or "if" to handle the different cases. That's why I tried adding SHELL but it didn't work, and I feel like I'm getting lost, so I'm passing on what I learned, and I'm going back to my authentication issues!

--------------------------------------- EDIT ------------------------------

I tried another method with creatin a script and executing it from dockerfile but dosen't work neither. I share you the code maybe it will be usefull : 

#Updates and installation
apt-get update && apt-get install libpq5 -y

mkdir -p temp_libs

# Libs we want to copy
libs=(
    "libpq.so*" 
    "libgssapi_krb5.so*"
    "libldap_r-2.4.so*"
    "libkrb5.so*"
    "libk5crypto.so*"
    "libkrb5support.so*"
    "liblber-2.4.so*"
    "libsasl2.so*"
    "libgnutls.so*"
    "libp11-kit.so*"
    "libidn2.so*"
    "libunistring.so*"
    "libtasn1.so*"
    "libnettle.so*"
    "libhogweed.so*"
    "libgmp.so*"
    "libffi.so*"
    "libcom_err.so*"
    "libkeyutils.so*"
)

# Loop for find and copy
for lib in "${libs[@]}"
do
  find /usr/lib -name "$lib" -exec cp --parents {} ./temp_libs \;
done

I thought it was an good idea to copy all libs in a folder and then only copy the folder wich contain subfolders with right name in stage 2

DriLLFreAK100 commented 1 year ago

@Kactus83 Yes, it will not work in stage 2 since the distroless image itself does not contain any utility (like shell) for the conditional logic construction purposes.

Hence, in this pull request, I took another approach to let us specify the architecture directly as part of the build arguments when running the commands instead.

Have you tried pulling the latest changes and execute the following 2 commands in succession? 

# You need to specify the ARCH argument accordingly
docker-compose build --build-arg "ARCH=x86_64"
docker-compose up

The first command carries out the build by passing in the build argument "ARCH=x86_64". While the second command is basically spinning up the containers with the built images.

I have tested this approach on both an M1 Mac (aarch64) and a Windows machine (x86_64) by setting the ARCH value accordingly. It seems to be working

Kactus83 commented 1 year ago

yes, I tried it this morning and it works without any problem!

I wanted to go further because I realized that there were two other architectures less common but still used. So I wondered how to automate the process so that it could be applied to other types of projects and architectures.

(this is based on the assumption that there are other cases that I haven't encountered yet where it can be interesting to start from a distroless image base to which you only add the necessary libs, but maybe it's useless or very rare, I have no idea!)

So I thought of creating a blank repository with only the installation of libs in the builder and the automatic movement of those in the distroless image no matter what architecture is used. The idea was to work in 3 steps:

1- Install the libs needed for your project and get the list of those you want in the final image (it can't be automated for the moment)

2- Loop through the libs you want to copy to find them and move them to a temporary folder keeping all the folder architecture (from usr for example)

3- When building the distroless image, you just have to copy the content of the builder's temporary folder, which should logically follow the structure of the destination folder, since you have kept the tree structure that was in the builder.

I was thinking that with this process it should have worked even for architectures I didn't know... maybe I'm ambitious

On Thu, Mar 9, 2023 at 5:18 PM Ng Charn Chuen @.***> wrote:

@Kactus83 https://github.com/Kactus83 Yes, it will not work in stage 2 since the distroless image itself does not contain any utility (like shell) for the conditional logic construction purposes.

Hence, in this pull request https://github.com/DriLLFreAK100/codefee-works-api/pull/10, I took another approach to let us specify the architecture directly as part of the build arguments when running the commands instead.

Have you tried pulling the latest changes and execute the following 2 commands in succession?

You need to specify the ARCH argument accordingly

docker-compose build --build-arg "ARCH=x86_64" docker-compose up

The first command carries out the build by passing in the build argument "ARCH=x86_64". While the second command is basically spinning up the containers with the built images.

I have tested this approach on both an M1 Mac (aarch64) and a Windows machine (x86_64) by setting the ARCH value accordingly. It seems to be working

— Reply to this email directly, view it on GitHub https://github.com/DriLLFreAK100/codefee-works-api/issues/7#issuecomment-1462348180, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWMFTO247O76MLCTGBQWTZDW3H7EHANCNFSM6AAAAAAVSLTNWQ . You are receiving this because you were mentioned.Message ID: @.***>

DriLLFreAK100 commented 1 year ago

The distroless image serves a different purpose. The goal is to minimize the final image size, so that it does not include anything else aside from the components required to run the application.

That being side, it's the ideal case to be able to self detect the architecture without requiring any additional configurations from a dev's point of view. Perhaps, we could do something like this

  1. install the shell util in stage 2 (distroless base)
  2. perform the dynamic arch configurations
  3. cleanup - remove the shell util

That way, we might be able to achieve the dynamicity, at the same time preserving the distroless' initial state.

I'll try to explore when I have the buffer. Cheers