facebook / mcrouter

Mcrouter is a memcached protocol router for scaling memcached deployments.
MIT License
3.26k stars 538 forks source link

Cannot build ubuntu20 docker file #394

Open biancoari opened 2 years ago

biancoari commented 2 years ago

Hi,

I've been trying to get in touch with the MCROUTER community with 0 success, I've been trying to build mcrouter in ubuntu20 for the past couple of day, I've used their installer (https://github.com/facebook/mcrouter/blob/main/mcrouter/scripts/install_ubuntu_20.04.sh) and I've also used the docker file (https://github.com/facebook/mcrouter/blob/main/mcrouter/scripts/docker/ubuntu/Dockerfile)

This is the tail that I get when trying a docker build of that dockerfile:

#10 1167.5 mv -f .deps/libmcroutercore_a-ThriftAcceptor.Tpo .deps/libmcroutercore_a-ThriftAcceptor.Po
#10 1167.5 g++ -DHAVE_CONFIG_H   -I.. -I/usr/local/mcrouter/install/include   -DLIBMC_FBTRACE_DISABLE -DDISABLE_COMPRESSION  -Wno-missing-field-initializers -Wno-deprecated -W -Wall -Wextra -Wno-unused-parameter -fno-strict-aliasing -std=c++17 -g -O2 -MT routes/libmcroutercore_a-FailoverRateLimiter.o -MD -MP -MF routes/.deps/libmcroutercore_a-FailoverRateLimiter.Tpo -c -o routes/libmcroutercore_a-FailoverRateLimiter.o `test -f 'routes/FailoverRateLimiter.cpp' || echo './'`routes/FailoverRateLimiter.cpp
#10 1170.9 mv -f routes/.deps/libmcroutercore_a-FailoverRateLimiter.Tpo routes/.deps/libmcroutercore_a-FailoverRateLimiter.Po
#10 1170.9 g++ -DHAVE_CONFIG_H   -I.. -I/usr/local/mcrouter/install/include   -DLIBMC_FBTRACE_DISABLE -DDISABLE_COMPRESSION  -Wno-missing-field-initializers -Wno-deprecated -W -Wall -Wextra -Wno-unused-parameter -fno-strict-aliasing -std=c++17 -g -O2 -MT routes/libmcroutercore_a-KeySplitRoute.o -MD -MP -MF routes/.deps/libmcroutercore_a-KeySplitRoute.Tpo -c -o routes/libmcroutercore_a-KeySplitRoute.o `test -f 'routes/KeySplitRoute.cpp' || echo './'`routes/KeySplitRoute.cpp
#10 1185.0 mv -f routes/.deps/libmcroutercore_a-BigValueRoute.Tpo routes/.deps/libmcroutercore_a-BigValueRoute.Po
#10 1185.0 g++ -DHAVE_CONFIG_H   -I.. -I/usr/local/mcrouter/install/include   -DLIBMC_FBTRACE_DISABLE -DDISABLE_COMPRESSION  -Wno-missing-field-initializers -Wno-deprecated -W -Wall -Wextra -Wno-unused-parameter -fno-strict-aliasing -std=c++17 -g -O2 -MT routes/libmcroutercore_a-L1L2SizeSplitRoute.o -MD -MP -MF routes/.deps/libmcroutercore_a-L1L2SizeSplitRoute.Tpo -c -o routes/libmcroutercore_a-L1L2SizeSplitRoute.o `test -f 'routes/L1L2SizeSplitRoute.cpp' || echo './'`routes/L1L2SizeSplitRoute.cpp
#10 1191.9 mv -f routes/.deps/libmcroutercore_a-KeySplitRoute.Tpo routes/.deps/libmcroutercore_a-KeySplitRoute.Po
#10 1191.9 g++ -DHAVE_CONFIG_H   -I.. -I/usr/local/mcrouter/install/include   -DLIBMC_FBTRACE_DISABLE -DDISABLE_COMPRESSION  -Wno-missing-field-initializers -Wno-deprecated -W -Wall -Wextra -Wno-unused-parameter -fno-strict-aliasing -std=c++17 -g -O2 -MT routes/libmcroutercore_a-LatencyInjectionRoute.o -MD -MP -MF routes/.deps/libmcroutercore_a-LatencyInjectionRoute.Tpo -c -o routes/libmcroutercore_a-LatencyInjectionRoute.o `test -f 'routes/LatencyInjectionRoute.cpp' || echo './'`routes/LatencyInjectionRoute.cpp
#10 1205.3 mv -f routes/.deps/libmcroutercore_a-L1L2SizeSplitRoute.Tpo routes/.deps/libmcroutercore_a-L1L2SizeSplitRoute.Po
#10 1205.3 g++ -DHAVE_CONFIG_H   -I.. -I/usr/local/mcrouter/install/include   -DLIBMC_FBTRACE_DISABLE -DDISABLE_COMPRESSION  -Wno-missing-field-initializers -Wno-deprecated -W -Wall -Wextra -Wno-unused-parameter -fno-strict-aliasing -std=c++17 -g -O2 -MT routes/libmcroutercore_a-McBucketRoute.o -MD -MP -MF routes/.deps/libmcroutercore_a-McBucketRoute.Tpo -c -o routes/libmcroutercore_a-McBucketRoute.o `test -f 'routes/McBucketRoute.cpp' || echo './'`routes/McBucketRoute.cpp
#10 1223.7 mv -f routes/.deps/libmcroutercore_a-LatencyInjectionRoute.Tpo routes/.deps/libmcroutercore_a-LatencyInjectionRoute.Po
#10 1223.7 g++ -DHAVE_CONFIG_H   -I.. -I/usr/local/mcrouter/install/include   -DLIBMC_FBTRACE_DISABLE -DDISABLE_COMPRESSION  -Wno-missing-field-initializers -Wno-deprecated -W -Wall -Wextra -Wno-unused-parameter -fno-strict-aliasing -std=c++17 -g -O2 -MT routes/libmcroutercore_a-McImportResolver.o -MD -MP -MF routes/.deps/libmcroutercore_a-McImportResolver.Tpo -c -o routes/libmcroutercore_a-McImportResolver.o `test -f 'routes/McImportResolver.cpp' || echo './'`routes/McImportResolver.cpp
#10 1224.6 mv -f routes/.deps/libmcroutercore_a-McImportResolver.Tpo routes/.deps/libmcroutercore_a-McImportResolver.Po
#10 1224.6 g++ -DHAVE_CONFIG_H   -I.. -I/usr/local/mcrouter/install/include   -DLIBMC_FBTRACE_DISABLE -DDISABLE_COMPRESSION  -Wno-missing-field-initializers -Wno-deprecated -W -Wall -Wextra -Wno-unused-parameter -fno-strict-aliasing -std=c++17 -g -O2 -MT routes/libmcroutercore_a-McRouteHandleProvider.o -MD -MP -MF routes/.deps/libmcroutercore_a-McRouteHandleProvider.Tpo -c -o routes/libmcroutercore_a-McRouteHandleProvider.o `test -f 'routes/McRouteHandleProvider.cpp' || echo './'`routes/McRouteHandleProvider.cpp
#10 1246.6 mv -f routes/.deps/libmcroutercore_a-McBucketRoute.Tpo routes/.deps/libmcroutercore_a-McBucketRoute.Po
#10 1246.6 g++ -DHAVE_CONFIG_H   -I.. -I/usr/local/mcrouter/install/include   -DLIBMC_FBTRACE_DISABLE -DDISABLE_COMPRESSION  -Wno-missing-field-initializers -Wno-deprecated -W -Wall -Wextra -Wno-unused-parameter -fno-strict-aliasing -std=c++17 -g -O2 -MT routes/libmcroutercore_a-McRouteHandleProvider-AllFastestRoute.o -MD -MP -MF routes/.deps/libmcroutercore_a-McRouteHandleProvider-AllFastestRoute.Tpo -c -o routes/libmcroutercore_a-McRouteHandleProvider-AllFastestRoute.o `test -f 'routes/McRouteHandleProvider-AllFastestRoute.cpp' || echo './'`routes/McRouteHandleProvider-AllFastestRoute.cpp
#10 1346.3 mv -f .deps/mcrouter-StandaloneUtils.Tpo .deps/mcrouter-StandaloneUtils.Po
#10 1359.4 mv -f routes/.deps/libmcroutercore_a-McRouteHandleProvider-AllFastestRoute.Tpo routes/.deps/libmcroutercore_a-McRouteHandleProvider-AllFastestRoute.Po
#10 1406.3 mv -f routes/.deps/libmcroutercore_a-McRouteHandleProvider.Tpo routes/.deps/libmcroutercore_a-McRouteHandleProvider.Po
#10 1406.3 make[3]: Leaving directory '/usr/local/mcrouter/repo/mcrouter/mcrouter'
#10 1406.3 make[2]: Leaving directory '/usr/local/mcrouter/repo/mcrouter/mcrouter'
#10 1406.4 make[1]: Leaving directory '/usr/local/mcrouter/repo/mcrouter/mcrouter'
------
executor failed running [/bin/sh -c /tmp/scripts/build.sh]: exit code: 2

Does anybody had any success with mcrouter and ubuntu20?

Thanks!

charlypa commented 2 years ago

Any Update?

biancoari commented 2 years ago

no response from the community, any plans on making a package for focal?

alynx282 commented 2 years ago

I implemented the Dockerfile this winter. Haven't tested that with the latest versions and with Ubuntu for a while. With almalinux it worked till the end of May, but now doesn't. Seems like it's due to some dependency update (fmt/fbthrift, etc. mcrouter scripts are using the master versions from github. Searching for a solution at the moment)

alexxed commented 1 year ago

I didn't find docs on how mcrouter is actually built from these tags that are cut, as the latest release is really really old, so managed to do it with this Dockerfile. This is for focal, and builds a distroless image; other versions might need some other sed stuff or apt packages. Although I managed to pin some software used, not everything is pinned, so expect this to be broken occasionally as stuff goes in to some dependencies that are cloned from their main branch.

FROM ubuntu:focal AS base

# Prevents TZ data prompt
ENV DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC

RUN apt-get update -y && \
    apt-get upgrade -y && \
    apt-get install -y git sudo tzdata && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

FROM base as builder
# Clone the repo
WORKDIR /workspace
# this is the latest tag from https://github.com/facebook/mcrouter/tags
ENV PIN_TAG=v2023.07.03.00
RUN git clone --depth 1 --branch $PIN_TAG https://github.com/facebook/mcrouter.git
# reduce concurrency to 1 as the build process is memory hungry and can fail in some CIs
RUN sed -i 's/ -j3 / -j1 -w /g' /workspace/mcrouter/mcrouter/scripts/get_and_build_by_make.sh
RUN sed -i 's/MAKE_ARGS="\$@ -j\$(nproc)"/MAKE_ARGS=" -j1 -w "/g' /workspace/mcrouter/mcrouter/scripts/common.sh
# pin fmtlib as master fails to build
RUN sed -i 's#https://github.com/fmtlib/fmt#--depth 1 --branch 10.0.0 https://github.com/fmtlib/fmt#' /workspace/mcrouter/mcrouter/scripts/recipes/fmtlib.sh
# pin all repos to the same tag
RUN echo $PIN_TAG > /workspace/mcrouter/mcrouter/FBTHRIFT_COMMIT && \
        echo $PIN_TAG > /workspace/mcrouter/mcrouter/FIZZ_COMMIT && \
        echo $PIN_TAG > /workspace/mcrouter/mcrouter/FOLLY_COMMIT && \
        echo $PIN_TAG > /workspace/mcrouter/mcrouter/WANGLE_COMMIT
WORKDIR /workspace/mcrouter

# Install mcrouter from source
RUN mkdir -p /build
# reduce concurrency to 1 as the build process is memory hungry and can fail in some CIs
ENV MAKE_ARGS=' -j1 -w '
ENV CXXFLAGS=' -w '
# if not done, version will be .0 which breaks php getStats
RUN echo 41:1 > /workspace/mcrouter/mcrouter/VERSION
RUN mcrouter/scripts/install_ubuntu_20.04.sh /build

# Get all the linked libraries for mcrouter and copy them to /lib
RUN ldd /build/install/bin/mcrouter | grep "=> /" | awk '{print $3}' | xargs -I '{}' cp -v '{}' /build/install/lib

FROM base
ENV LD_LIBRARY_PATH=/mcrouter/lib
WORKDIR /mcrouter
COPY --from=builder /build/install /mcrouter
RUN ln -s /mcrouter/bin/mcrouter /usr/local/bin/mcrouter

USER 1000:1000
CMD ["mcrouter", "--version"]
bmm-alc commented 7 months ago

I didn't find docs on how mcrouter is actually built from these tags that are cut, as the latest release is really really old, so managed to do it with this Dockerfile.

Great work, it builds fine. Do you maintain this Docker file in a repo ? (for instance if you update the pinning on a regular basis)

Thanks