search

shaka-project / shaka-packager

A media packaging and development framework for VOD and Live DASH and HLS applications, supporting Common Encryption for Widevine and other DRM Systems.
https://shaka-project.github.io/shaka-packager/
Other
1.95k stars 504 forks source link

support for alpine linux #164

Closed bmurphy1976 closed 6 years ago

bmurphy1976 commented 7 years ago

I made a half-assed attempt at building shaka packager on Alpine Linux (which uses musl as it's standard library instead of libc). Massive failure.

Are there any plans to change the build process to make porting to alternate distributions easier?

The biggest problems I ran into so far are dependencies on binaries that are linked to libc, clang flags that aren't valid in what appears to be newer version of clang provided by the distribution, pathing problems finding system wide header files, and what looks like either mis-identification of the host os or assumptions that the host os is debian or a debian derivative.

Ideally I'd like a statically compiled binary that can run inside a very small docker container, ala alpine (~5mb) instead of ubuntu (~250mb).

kqyang commented 7 years ago

Yes, I do plan to tackle this problem. I haven't dug into this problem in detail. I will update this bug once I have a solution.

Btw, does disabling clang work, i.e. set clang=0,

GYP_DEFINES='clang=0' gclient runhooks
ninja -C out/Debug
bmurphy1976 commented 7 years ago

I can get a little farther but now it's referencing a copy of binutils that aren't compatible with musl: 

[6/1560] CXX obj/third_party/protobuf/src/google/protobuf/stubs/protobuf_full_do_not_use.substitute.o
FAILED: c++ -MMD -MF obj/third_party/protobuf/src/google/protobuf/stubs/protobuf_full_do_not_use.substitute.o.d -DV8_DEPRECATION_WARNINGS -D_FILE_OFFSET_BITS=64 -DCHROMIUM_BUILD -DCR_CLANG_REVISION=274369-1 -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_AURA=1 -DUSE_PANGO=1 -DUSE_CAIRO=1 -DUSE_DEFAULT_RENDER_THEME=1 -DUSE_LIBJPEG_TURBO=1 -DENABLE_WEBRTC=1 -DENABLE_MEDIA_ROUTER=1 -DENABLE_PEPPER_CDMS -DENABLE_NOTIFICATIONS -DUSE_UDEV -DFIELDTRIAL_TESTING_ENABLED -DENABLE_TASK_MANAGER=1 -DENABLE_EXTENSIONS=1 -DENABLE_PDF=1 -DENABLE_PLUGINS=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_THEMES=1 -DENABLE_PRINTING=1 -DENABLE_BASIC_PRINTING=1 -DENABLE_PRINT_PREVIEW=1 -DENABLE_SPELLCHECK=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DENABLE_SUPERVISED_USERS=1 -DENABLE_MDNS=1 -DENABLE_SERVICE_DISCOVERY=1 -DV8_USE_EXTERNAL_STARTUP_DATA -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DUSE_LIBPCI=1 -DUSE_NSS_CERTS=1 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -Igen -I../../packager/third_party/protobuf/src -fstack-protector --param=ssp-buffer-size=4 -pthread -fno-strict-aliasing -Wno-extra -Wno-unused-parameter -Wno-missing-field-initializers -fvisibility=hidden -pipe -fPIC -B/shaka_packager/src/packager/third_party/binutils/Linux_x64/Release/bin -Wno-unused-local-typedefs -m64 -march=x86-64 -O2 -fno-ident -fdata-sections -ffunction-sections -funwind-tables -fno-exceptions -fno-rtti -fno-threadsafe-statics -fvisibility-inlines-hidden -Wno-deprecated -std=gnu++11 -Wno-narrowing  -c ../../packager/third_party/protobuf/src/google/protobuf/stubs/substitute.cc -o obj/third_party/protobuf/src/google/protobuf/stubs/protobuf_full_do_not_use.substitute.o
c++: error trying to exec '/shaka_packager/src/packager/third_party/binutils/Linux_x64/Release/bin/as': execv: No such file or directory
ninja: build stopped: subcommand failed.

FYI, this is the Dockerfile I'm using to reproduce this:

FROM alpine:3.4
CMD [ "/bin/bash" ]

RUN apk add --no-cache \
    bash \
    build-base \
    curl \
    findutils \
    git \
    ninja \
    python \
    tar \
    wget

RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

# put /depot_tools at end so system binaries take precedence
ENV PATH $PATH:/depot_tools
RUN mkdir shaka_packager

WORKDIR shaka_packager
RUN gclient config https://www.github.com/google/shaka-packager.git --name=src
RUN gclient sync --no-history

WORKDIR src
ENV GYP_DEFINES clang=0
RUN gclient runhooks
RUN ninja -C out/Release

I'm looking into circumventing this dependency now.

bmurphy1976 commented 7 years ago

Got further with:

GYP_DEFINES='linux_use_bundled_binutils=0 clang=0'

Output:

bash-4.3# ./gyp_packager.py
Updating projects from gyp files...
bash-4.3# ninja -C out/Release
ninja: Entering directory `out/Release'
[323/1560] LINK protoc
FAILED: c++ -Wl,-z,now -Wl,-z,relro -Wl,--fatal-warnings -Wl,-z,defs -pthread -Wl,-z,noexecstack -fPIC -B/shaka_packager/src/packager/third_party/binutils/Linux_x64/Release/bin -Wl,--disable-new-dtags -m64 -Wl,-O1 -Wl,--as-needed -Wl,--gc-sections -Wl,--no-as-needed -lpthread -Wl,--as-needed -o protoc -Wl,--start-group obj/third_party/protobuf/src/google/protobuf/compiler/protoc.main.o obj/third_party/protobuf/libprotoc_lib.a obj/third_party/protobuf/libprotobuf_full_do_not_use.a -Wl,--end-group
collect2: error trying to exec '/shaka_packager/src/packager/third_party/binutils/Linux_x64/Release/bin/ld': execvp: No such file or directory
collect2: error: ld returned 255 exit status
[323/1560] CXX obj/third_party/icu/source/common/icuuc.rbbiscan.o
ninja: build stopped: subcommand failed.
bmurphy1976 commented 7 years ago

More progress, but the next error is of an entirely different nature:

GYP_DEFINES='linux_use_bundled_binutils=0 linux_use_bundled_gold=0

Output:

bash-4.3# export GYP_DEFINES='linux_use_bundled_binutils=0 linux_use_bundled_gold=0 clang=0'
bash-4.3# ./gyp_packager.py
Updating projects from gyp files...
bash-4.3# ninja -C out/Release
ninja: Entering directory `out/Release'
[26/1228] CXX obj/media/base/gen/protoc_out/packager/media/base/widevine_pssh_data_proto.widevine_pssh_data.pb.o
FAILED: c++ -MMD -MF obj/media/base/gen/protoc_out/packager/media/base/widevine_pssh_data_proto.widevine_pssh_data.pb.o.d -DV8_DEPRECATION_WARNINGS -D_FILE_OFFSET_BITS=64 -DCHROMIUM_BUILD -DCR_CLANG_REVISION=274369-1 -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_AURA=1 -DUSE_PANGO=1 -DUSE_CAIRO=1 -DUSE_DEFAULT_RENDER_THEME=1 -DUSE_LIBJPEG_TURBO=1 -DENABLE_WEBRTC=1 -DENABLE_MEDIA_ROUTER=1 -DENABLE_PEPPER_CDMS -DENABLE_NOTIFICATIONS -DUSE_UDEV -DFIELDTRIAL_TESTING_ENABLED -DENABLE_TASK_MANAGER=1 -DENABLE_EXTENSIONS=1 -DENABLE_PDF=1 -DENABLE_PLUGINS=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_THEMES=1 -DENABLE_PRINTING=1 -DENABLE_BASIC_PRINTING=1 -DENABLE_PRINT_PREVIEW=1 -DENABLE_SPELLCHECK=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DENABLE_SUPERVISED_USERS=1 -DENABLE_MDNS=1 -DENABLE_SERVICE_DISCOVERY=1 -DV8_USE_EXTERNAL_STARTUP_DATA -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DPROTOBUF_USE_DLLS -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DUSE_LIBPCI=1 -DUSE_NSS_CERTS=1 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -D_FORTIFY_SOURCE=2 -Igen -I../../packager -I../.. -Igen/protoc_out -I../../packager/third_party/protobuf/src -fstack-protector --param=ssp-buffer-size=4 -Werror -pthread -fno-strict-aliasing -Wall -Wno-extra -Wno-unused-parameter -Wno-missing-field-initializers -fvisibility=hidden -pipe -fPIC -Wno-unused-local-typedefs -m64 -march=x86-64 -O2 -fno-ident -fdata-sections -ffunction-sections -funwind-tables -fno-exceptions -fno-rtti -fno-threadsafe-statics -fvisibility-inlines-hidden -std=gnu++11 -Wno-narrowing  -c gen/protoc_out/packager/media/base/widevine_pssh_data.pb.cc -o obj/media/base/gen/protoc_out/packager/media/base/widevine_pssh_data_proto.widevine_pssh_data.pb.o
<command-line>:0:0: error: "_FORTIFY_SOURCE" redefined [-Werror]
<built-in>: note: this is the location of the previous definition
cc1plus: all warnings being treated as errors
[26/1228] CXX obj/mpd/gen/protoc_out/packager/mpd/base/media_info_proto.media_info.pb.o
FAILED: c++ -MMD -MF obj/mpd/gen/protoc_out/packager/mpd/base/media_info_proto.media_info.pb.o.d -DV8_DEPRECATION_WARNINGS -D_FILE_OFFSET_BITS=64 -DCHROMIUM_BUILD -DCR_CLANG_REVISION=274369-1 -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_AURA=1 -DUSE_PANGO=1 -DUSE_CAIRO=1 -DUSE_DEFAULT_RENDER_THEME=1 -DUSE_LIBJPEG_TURBO=1 -DENABLE_WEBRTC=1 -DENABLE_MEDIA_ROUTER=1 -DENABLE_PEPPER_CDMS -DENABLE_NOTIFICATIONS -DUSE_UDEV -DFIELDTRIAL_TESTING_ENABLED -DENABLE_TASK_MANAGER=1 -DENABLE_EXTENSIONS=1 -DENABLE_PDF=1 -DENABLE_PLUGINS=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_THEMES=1 -DENABLE_PRINTING=1 -DENABLE_BASIC_PRINTING=1 -DENABLE_PRINT_PREVIEW=1 -DENABLE_SPELLCHECK=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DENABLE_SUPERVISED_USERS=1 -DENABLE_MDNS=1 -DENABLE_SERVICE_DISCOVERY=1 -DV8_USE_EXTERNAL_STARTUP_DATA -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DPROTOBUF_USE_DLLS -DUSE_LIBPCI=1 -DUSE_NSS_CERTS=1 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -D_FORTIFY_SOURCE=2 -Igen -I../../packager -I../.. -Igen/protoc_out -I../../packager/third_party/protobuf/src -fstack-protector --param=ssp-buffer-size=4 -Werror -pthread -fno-strict-aliasing -Wall -Wno-extra -Wno-unused-parameter -Wno-missing-field-initializers -fvisibility=hidden -pipe -fPIC -Wno-unused-local-typedefs -m64 -march=x86-64 -O2 -fno-ident -fdata-sections -ffunction-sections -funwind-tables -fno-exceptions -fno-rtti -fno-threadsafe-statics -fvisibility-inlines-hidden -std=gnu++11 -Wno-narrowing  -c gen/protoc_out/packager/mpd/base/media_info.pb.cc -o obj/mpd/gen/protoc_out/packager/mpd/base/media_info_proto.media_info.pb.o
<command-line>:0:0: error: "_FORTIFY_SOURCE" redefined [-Werror]
<built-in>: note: this is the location of the previous definition
cc1plus: all warnings being treated as errors
[26/1228] CC obj/third_party/libxml/src/libxml.xmlschemas.o
ninja: build stopped: subcommand failed.
bmurphy1976 commented 7 years ago

More progress, had to install a few additional headers and comment out the _FORTIFY_SOURCE line in common.gypi. I may have hit the wall though, I'm not sure how to even approach the next error:

[835/1560] CXX obj/third_party/tcmalloc/chromium/src/base/allocator.spinlock_internal.o
FAILED: c++ -MMD -MF obj/third_party/tcmalloc/chromium/src/base/allocator.spinlock_internal.o.d -DV8_DEPRECATION_WARNINGS -D_FILE_OFFSET_BITS=64 -DCHROMIUM_BUILD -DCR_CLANG_REVISION=274369-1 -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_AURA=1 -DUSE_PANGO=1 -DUSE_CAIRO=1 -DUSE_DEFAULT_RENDER_THEME=1 -DUSE_LIBJPEG_TURBO=1 -DENABLE_WEBRTC=1 -DENABLE_MEDIA_ROUTER=1 -DENABLE_PEPPER_CDMS -DENABLE_NOTIFICATIONS -DUSE_UDEV -DFIELDTRIAL_TESTING_ENABLED -DENABLE_TASK_MANAGER=1 -DENABLE_EXTENSIONS=1 -DENABLE_PDF=1 -DENABLE_PLUGINS=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_THEMES=1 -DENABLE_PRINTING=1 -DENABLE_BASIC_PRINTING=1 -DENABLE_PRINT_PREVIEW=1 -DENABLE_SPELLCHECK=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DENABLE_SUPERVISED_USERS=1 -DENABLE_MDNS=1 -DENABLE_SERVICE_DISCOVERY=1 -DV8_USE_EXTERNAL_STARTUP_DATA -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DNO_HEAP_CHECK -DTCMALLOC_DONT_REPLACE_SYSTEM_ALLOC -DUSE_LIBPCI=1 -DUSE_NSS_CERTS=1 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -Igen -I../../packager/base/allocator -I../../packager/third_party/tcmalloc/chromium/src/base -I../../packager/third_party/tcmalloc/chromium/src -I../../packager -fstack-protector --param=ssp-buffer-size=4 -pthread -fno-strict-aliasing -Wno-extra -Wno-unused-parameter -Wno-missing-field-initializers -fvisibility=hidden -pipe -fPIC -Wno-unused-local-typedefs -Wno-sign-compare -Wno-unused-result -m64 -march=x86-64 -O2 -fno-ident -fdata-sections -ffunction-sections -funwind-tables -fno-exceptions -fno-rtti -fno-threadsafe-statics -fvisibility-inlines-hidden -Wno-deprecated -std=gnu++11 -Wno-narrowing  -c ../../packager/third_party/tcmalloc/chromium/src/base/spinlock_internal.cc -o obj/third_party/tcmalloc/chromium/src/base/allocator.spinlock_internal.o
In file included from ../../packager/third_party/tcmalloc/chromium/src/base/spinlock_linux-inl.h:38:0,
                 from ../../packager/third_party/tcmalloc/chromium/src/base/spinlock_internal.cc:51:
../../packager/third_party/tcmalloc/chromium/src/base/linux_syscall_support.h:1932:37: error: '__off64_t' has not been declared
                                     __off64_t o) {
                                     ^
ninja: build stopped: subcommand failed.

Updated Dockerfile:

FROM alpine:3.4
CMD [ "/bin/bash" ]

RUN apk add --no-cache \
    bash \
    build-base \
    bsd-compat-headers \
    curl \
    findutils \
    git \
    ninja \
    python \
    tar \
    wget

RUN apk add --no-cache --repository "http://dl-3.alpinelinux.org/alpine/edge/community" \
    linux-headers

RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

# put /depot_tools at end so system binaries take precedence
ENV PATH $PATH:/depot_tools
RUN mkdir shaka_packager

WORKDIR shaka_packager
RUN gclient config https://www.github.com/google/shaka-packager.git --name=src
RUN gclient sync --no-history

WORKDIR src
ENV GYP_DEFINES "linux_use_bundled_binutils=0 linux_use_bundled_gold=0 clang=0"
RUN gclient runhooks

RUN sed -i "s/'_FORTIFY_SOURCE=2',/#'_FORTIFY_SOURCE=2',/" packager/build/common.gypi
RUN ./gyp_packager.py

RUN ninja -C out/Release
dbpolito commented 6 years ago

It's been a while no one comments here, wondering if someone got it working.

dbpolito commented 6 years ago

I followed the steps and i get:

Could not find a version that satisfies the requirement cffi==1.10.0 (from -r /tmp/vpython_bootstrap440952574/requirements.txt (line 4)) (from versions: )
kqyang commented 6 years ago

@dbpolito Sorry for not getting to this issue earlier. I will post a solution soon.

kqyang commented 6 years ago

@dbpolito I have added instructions to build Shaka Packager on Alpine Linux: https://github.com/google/shaka-packager/blob/master/docs/source/build_instructions.md#alpine-linux. And here is docker config that works: https://github.com/google/shaka-packager/blob/master/packager/testing/dockers/Alpine_Dockerfile. Let us know if you have more questions.

dbpolito commented 6 years ago

That's awesome. ❤️

wader commented 6 years ago

@kqyang Hello! I have problem using the alpine Dockerfile you provided. Not sure what i'm doing wrong, see log below. The error seems to be same as for @dbpolito:

Could not find a version that satisfies the requirement cffi==1.10.0 (from -r /tmp/vpython_bootstrap868511423/requirements.txt (line 4)) (from versions: )

My goal is to build static binaries using the alpine build toolchain. My plan is to run ninja with GYP_GENERATORS="make" and then export CFLAGS/LDFLAGS etc to produce static binaries. Should that work? will all dependencies also use the flags?

Im using docker for mac 18.06.0-ce-mac70 (26399) and when running docker run /shacke-packager is a volume mount of a directory where shaka-packageas is cloned and HEAD is at 715ed939f1712a398281e56bbccaec25116828ea (commit where you added alpine support)

Full build log:

$ docker build -t test .
Sending build context to Docker daemon  3.072kB
Step 1/7 : FROM alpine:3.8
 ---> 11cd0b38bc3c
Step 2/7 : RUN apk add --no-cache bash build-base curl findutils git ninja python                        bsd-compat-headers linux-headers libexecinfo-dev
 ---> Using cache
 ---> 18336f624b29
Step 3/7 : WORKDIR /
 ---> Using cache
 ---> 07b9ff44db0c
Step 4/7 : RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
 ---> Using cache
 ---> dd2aea162d97
Step 5/7 : ENV PATH $PATH:/depot_tools
 ---> Using cache
 ---> 5d1a02e03fe8
Step 6/7 : RUN sed -i     '/malloc_usable_size/a \\nstruct mallinfo {\n  int arena;\n  int hblkhd;\n  int uordblks;\n};'     /usr/include/malloc.h
 ---> Using cache
 ---> dcb71b6ce523
Step 7/7 : ENV GYP_DEFINES="linux_use_bundled_binutils=0 linux_use_bundled_gold=0 clang=0 use_experimental_allocator_shim=0 use_allocator=none musl=1"
 ---> Using cache
 ---> dce08f72c195
Successfully built dce08f72c195
Successfully tagged test:latest
$ docker run -ti --rm -v $PWD:/shaka-packager test
/ # cd /shaka-packager/
/shaka-packager # ls
AUTHORS            CONTRIBUTORS       Dockerfile.static  appveyor.yml       gyp_packager.py
CHANGELOG.md       DEPS               LICENSE            chromium-LICENSE   kokoro
CONTRIBUTING.md    Dockerfile         README.md          docs               packager
/shaka-packager #  gclient config https://www.github.com/google/shaka-packager.git --name=src --unmanaged
WARNING: Your metrics.cfg file was invalid or nonexistent. A new one will been created.
/shaka-packager # gclient sync -r 715ed939f1712a398281e56bbccaec25116828ea

________ running 'git -c core.deltaBaseCacheLimit=2g clone --no-checkout --progress https://www.github.com/google/shaka-packager.git /shaka-packager/_gclient_src_zihCIa' in '/shaka-packager'
Cloning into '/shaka-packager/_gclient_src_zihCIa'...
warning: redirecting to https://github.com/google/shaka-packager.git/
remote: Counting objects: 663830, done.
remote: Compressing objects: 100% (1550/1550), done.
Receiving objects:  46% (309589/663830), 201.69 MiB | 2.66 MiB/s
[0:01:00] Still working on:
[0:01:00]   src
Receiving objects:  54% (358469/663830), 236.00 MiB | 3.60 MiB/s
[0:01:10] Still working on:
[0:01:10]   src
Receiving objects:  61% (408419/663830), 269.90 MiB | 3.38 MiB/s
[0:01:20] Still working on:
[0:01:20]   src
Receiving objects:  69% (461099/663830), 303.43 MiB | 2.78 MiB/s
[0:01:30] Still working on:
[0:01:30]   src
Receiving objects:  76% (506898/663830), 331.81 MiB | 3.03 MiB/s
[0:01:40] Still working on:
[0:01:40]   src
Receiving objects:  84% (557618/663830), 364.61 MiB | 3.49 MiB/s
[0:01:50] Still working on:
[0:01:50]   src
Receiving objects:  91% (607101/663830), 397.16 MiB | 3.00 MiB/s
[0:02:00] Still working on:
[0:02:00]   src
Receiving objects:  96% (639842/663830), 437.27 MiB | 5.13 MiB/s
[0:02:10] Still working on:
[0:02:10]   src
Receiving objects:  97% (648780/663830), 494.12 MiB | 6.87 MiB/s
[0:02:20] Still working on:
[0:02:20]   src
remote: Total 663830 (delta 1199), reused 1589 (delta 772), pack-reused 661499
Receiving objects: 100% (663830/663830), 534.39 MiB | 3.66 MiB/s, done.
Resolving deltas:   0% (0/463993)
[0:02:30] Still working on:
[0:02:30]   src
Resolving deltas:   3% (13941/463993)
[0:02:40] Still working on:
[0:02:40]   src
Resolving deltas:   8% (41690/463993)
[0:02:50] Still working on:
[0:02:50]   src

[0:03:00] Still working on:
[0:03:00]   src
Resolving deltas:  22% (102875/463993)
[0:03:10] Still working on:
[0:03:10]   src
Resolving deltas:  26% (121576/463993)
[0:03:20] Still working on:
[0:03:20]   src
Resolving deltas:  36% (167041/463993)
[0:03:30] Still working on:
[0:03:30]   src
Resolving deltas:  37% (175527/463993)
[0:03:40] Still working on:
[0:03:40]   src
Resolving deltas:  47% (218283/463993)
[0:03:50] Still working on:
[0:03:50]   src
Resolving deltas:  52% (244628/463993)
[0:04:00] Still working on:
[0:04:00]   src
Resolving deltas:  62% (290825/463993)
[0:04:10] Still working on:
[0:04:10]   src
Resolving deltas:  69% (322393/463993)
[0:04:20] Still working on:
[0:04:20]   src
Resolving deltas:  76% (355446/463993)
[0:04:30] Still working on:
[0:04:30]   src
Resolving deltas:  82% (383462/463993)
[0:04:40] Still working on:
[0:04:40]   src
Resolving deltas:  90% (417680/463993)
[0:04:50] Still working on:
[0:04:50]   src
Resolving deltas:  96% (448791/463993)
[0:05:00] Still working on:
[0:05:00]   src
Resolving deltas: 100% (463993/463993), done.

[0:05:10] Still working on:
[0:05:10]   src

[0:05:20] Still working on:
[0:05:20]   src
Syncing projects:   0% ( 0/ 2)
[0:05:22] Still working on:
[0:05:22]   src
Syncing projects: 100% (20/20), done.

________ running '/usr/bin/python src/packager/build/linux/sysroot_scripts/install-sysroot.py --running-as-hook' in '/shaka-packager'
Installing Debian Wheezy amd64 root image: /shaka-packager/src/packager/build/linux/debian_wheezy_amd64-sysroot
Downloading https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/c52471d9dec240c8d0a88fa98aa1eefeee32e22f/debian_wheezy_amd64_sysroot.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 41.6M  100 41.6M    0     0  12.1M      0  0:00:03  0:00:03 --:--:-- 12.1M
Installing Debian Wheezy i386 root image: /shaka-packager/src/packager/build/linux/debian_wheezy_i386-sysroot
Downloading https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/c52471d9dec240c8d0a88fa98aa1eefeee32e22f/debian_wheezy_i386_sysroot.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 44.6M  100 44.6M    0     0  4349k      0  0:00:10  0:00:10 --:--:-- 7677k
Hook '/usr/bin/python src/packager/build/linux/sysroot_scripts/install-sysroot.py --running-as-hook' took 102.72 secs
Running hooks:  50% ( 3/ 6) binutils
________ running '/usr/bin/python src/packager/third_party/binutils/download.py' in '/shaka-packager'
Downloading /shaka-packager/src/packager/third_party/binutils/Linux_x64/binutils.tar.bz2
0> Failed to fetch file gs://chromium-binutils/d9064388bed0e7225b1366d80b59289b1509d7c2 for /shaka-packager/src/packager/third_party/binutils/Linux_x64/binutils.tar.bz2, skipping. [Err: [E2018-08-23T14:41:58.979214Z 804 0 venv.go:942] Command (cwd=): [/root/.vpython-root/17ddb1/bin/python -B -E -s -m pip install --isolated --compile --no-index --find-links /tmp/vpython_bootstrap743910297/packages --requirement /tmp/vpython_bootstrap743910297/requirements.txt --no-binary=:none: --only-binary=:all: --use-wheel]
Process output:
Collecting appdirs==1.4.3 (from -r /tmp/vpython_bootstrap743910297/requirements.txt (line 1))
Collecting asn1crypto==0.22.0 (from -r /tmp/vpython_bootstrap743910297/requirements.txt (line 2))
Collecting boto==2.48.0 (from -r /tmp/vpython_bootstrap743910297/requirements.txt (line 3))
Collecting cffi==1.10.0 (from -r /tmp/vpython_bootstrap743910297/requirements.txt (line 4))
  Could not find a version that satisfies the requirement cffi==1.10.0 (from -r /tmp/vpython_bootstrap743910297/requirements.txt (line 4)) (from versions: )
No matching distribution found for cffi==1.10.0 (from -r /tmp/vpython_bootstrap743910297/requirements.txt (line 4))

Environment:
AWS_CREDENTIAL_FILE=
BOTO_CONFIG=
GYP_DEFINES=linux_use_bundled_binutils=0 linux_use_bundled_gold=0 clang=0 use_experimental_allocator_shim=0 use_allocator=none musl=1
HOME=/tmp/vpython_bootstrap743910297
HOSTNAME=d634cd4a67ca
OLDPWD=/
PATH=/root/.vpython-root/f72659/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/depot_tools
PIP_NO_BINARY=:none:
PIP_ONLY_BINARY=:all:
PIP_USE_WHEEL=1
PWD=/shaka-packager
PYTHONDONTWRITEBYTECODE=1
PYTHONNOUSERSITE=1
SHLVL=1
TEMP=/tmp
TERM=xterm
TMP=/tmp
TMPDIR=/tmp
VIRTUAL_ENV=/root/.vpython-root/f72659
[E2018-08-23T14:41:58.989146Z 804 0 annotate.go:242] original error: exit status 1
[E2018-08-23T14:41:58.989212Z 804 0 annotate.go:242]
[E2018-08-23T14:41:58.989241Z 804 0 annotate.go:242] goroutine 1:
[E2018-08-23T14:41:58.989270Z 804 0 annotate.go:242] #0 go.chromium.org/luci/vpython/venv/venv.go:730 - venv.(*Env).installWheels()
[E2018-08-23T14:41:58.989313Z 804 0 annotate.go:242]   reason: failed to install wheels
[E2018-08-23T14:41:58.989344Z 804 0 annotate.go:242]
[E2018-08-23T14:41:58.989393Z 804 0 annotate.go:242] #1 go.chromium.org/luci/vpython/venv/venv.go:527 - venv.(*Env).createLocked.func1()
[E2018-08-23T14:41:58.989439Z 804 0 annotate.go:242]   reason: failed to install wheels
[E2018-08-23T14:41:58.989469Z 804 0 annotate.go:242]
[E2018-08-23T14:41:58.989512Z 804 0 annotate.go:242] #2 go.chromium.org/luci/common/system/filesystem/tempdir.go:55 - filesystem.(*TempDir).With()
[E2018-08-23T14:41:58.989538Z 804 0 annotate.go:242] #3 go.chromium.org/luci/vpython/venv/venv.go:103 - venv.withTempDir()
[E2018-08-23T14:41:58.989646Z 804 0 annotate.go:242] #4 go.chromium.org/luci/vpython/venv/venv.go:497 - venv.(*Env).createLocked()
[E2018-08-23T14:41:58.989694Z 804 0 annotate.go:242] #5 go.chromium.org/luci/vpython/venv/venv.go:272 - venv.(*Env).ensure.func1()
[E2018-08-23T14:41:58.989724Z 804 0 annotate.go:242]   reason: failed to create new VirtualEnv
[E2018-08-23T14:41:58.989754Z 804 0 annotate.go:242]
[E2018-08-23T14:41:58.989798Z 804 0 annotate.go:242] #6 go.chromium.org/luci/vpython/venv/venv.go:957 - venv.mustReleaseLock()
[E2018-08-23T14:41:58.989828Z 804 0 annotate.go:242] #7 go.chromium.org/luci/vpython/venv/venv.go:258 - venv.(*Env).ensure()
[E2018-08-23T14:41:58.989870Z 804 0 annotate.go:242] #8 go.chromium.org/luci/vpython/venv/venv.go:328 - venv.(*Env).withImpl()
[E2018-08-23T14:41:58.989897Z 804 0 annotate.go:242] #9 go.chromium.org/luci/vpython/venv/venv.go:168 - venv.With()
[E2018-08-23T14:41:58.989941Z 804 0 annotate.go:242] #10 go.chromium.org/luci/vpython/run.go:63 - vpython.Run()
[E2018-08-23T14:41:58.989985Z 804 0 annotate.go:242] #11 go.chromium.org/luci/vpython/application/application.go:318 - application.(*application).mainImpl()
[E2018-08-23T14:41:58.990014Z 804 0 annotate.go:242] #12 go.chromium.org/luci/vpython/application/application.go:402 - application.(*Config).Main.func1()
[E2018-08-23T14:41:58.990057Z 804 0 annotate.go:242] #13 go.chromium.org/luci/vpython/application/support.go:47 - application.run()
[E2018-08-23T14:41:58.990102Z 804 0 annotate.go:242] #14 go.chromium.org/luci/vpython/application/application.go:401 - application.(*Config).Main()
[E2018-08-23T14:41:58.990147Z 804 0 annotate.go:242] #15 vpython/main.go:93 - main.mainImpl()
[E2018-08-23T14:41:58.990173Z 804 0 annotate.go:242] #16 vpython/main.go:99 - main.main()
[E2018-08-23T14:41:58.990218Z 804 0 annotate.go:242] #17 runtime/proc.go:198 - runtime.main()
[E2018-08-23T14:41:58.990263Z 804 0 annotate.go:242] #18 runtime/asm_amd64.s:2361 - runtime.goexit()
]Failed to fetch file gs://chromium-binutils/d9064388bed0e7225b1366d80b59289b1509d7c2 for /shaka-packager/src/packager/third_party/binutils/Linux_x64/binutils.tar.bz2. [Err: [E2018-08-23T14:41:58.979214Z 804 0 venv.go:942] Command (cwd=): [/root/.vpython-root/17ddb1/bin/python -B -E -s -m pip install --isolated --compile --no-index --find-links /tmp/vpython_bootstrap743910297/packages --requirement /tmp/vpython_bootstrap743910297/requirements.txt --no-binary=:none: --only-binary=:all: --use-wheel]
Process output:
Collecting appdirs==1.4.3 (from -r /tmp/vpython_bootstrap743910297/requirements.txt (line 1))
Collecting asn1crypto==0.22.0 (from -r /tmp/vpython_bootstrap743910297/requirements.txt (line 2))
Collecting boto==2.48.0 (from -r /tmp/vpython_bootstrap743910297/requirements.txt (line 3))
Collecting cffi==1.10.0 (from -r /tmp/vpython_bootstrap743910297/requirements.txt (line 4))
  Could not find a version that satisfies the requirement cffi==1.10.0 (from -r /tmp/vpython_bootstrap743910297/requirements.txt (line 4)) (from versions: )
No matching distribution found for cffi==1.10.0 (from -r /tmp/vpython_bootstrap743910297/requirements.txt (line 4))

Environment:
AWS_CREDENTIAL_FILE=
BOTO_CONFIG=
GYP_DEFINES=linux_use_bundled_binutils=0 linux_use_bundled_gold=0 clang=0 use_experimental_allocator_shim=0 use_allocator=none musl=1
HOME=/tmp/vpython_bootstrap743910297
HOSTNAME=d634cd4a67ca
OLDPWD=/
PATH=/root/.vpython-root/f72659/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/depot_tools
PIP_NO_BINARY=:none:
PIP_ONLY_BINARY=:all:
PIP_USE_WHEEL=1
PWD=/shaka-packager
PYTHONDONTWRITEBYTECODE=1
PYTHONNOUSERSITE=1
SHLVL=1
TEMP=/tmp
TERM=xterm
TMP=/tmp
TMPDIR=/tmp
VIRTUAL_ENV=/root/.vpython-root/f72659
[E2018-08-23T14:41:58.989146Z 804 0 annotate.go:242] original error: exit status 1
[E2018-08-23T14:41:58.989212Z 804 0 annotate.go:242]
[E2018-08-23T14:41:58.989241Z 804 0 annotate.go:242] goroutine 1:
[E2018-08-23T14:41:58.989270Z 804 0 annotate.go:242] #0 go.chromium.org/luci/vpython/venv/venv.go:730 - venv.(*Env).installWheels()
[E2018-08-23T14:41:58.989313Z 804 0 annotate.go:242]   reason: failed to install wheels
[E2018-08-23T14:41:58.989344Z 804 0 annotate.go:242]
[E2018-08-23T14:41:58.989393Z 804 0 annotate.go:242] #1 go.chromium.org/luci/vpython/venv/venv.go:527 - venv.(*Env).createLocked.func1()
[E2018-08-23T14:41:58.989439Z 804 0 annotate.go:242]   reason: failed to install wheels
[E2018-08-23T14:41:58.989469Z 804 0 annotate.go:242]
[E2018-08-23T14:41:58.989512Z 804 0 annotate.go:242] #2 go.chromium.org/luci/common/system/filesystem/tempdir.go:55 - filesystem.(*TempDir).With()
[E2018-08-23T14:41:58.989538Z 804 0 annotate.go:242] #3 go.chromium.org/luci/vpython/venv/venv.go:103 - venv.withTempDir()
[E2018-08-23T14:41:58.989646Z 804 0 annotate.go:242] #4 go.chromium.org/luci/vpython/venv/venv.go:497 - venv.(*Env).createLocked()
[E2018-08-23T14:41:58.989694Z 804 0 annotate.go:242] #5 go.chromium.org/luci/vpython/venv/venv.go:272 - venv.(*Env).ensure.func1()
[E2018-08-23T14:41:58.989724Z 804 0 annotate.go:242]   reason: failed to create new VirtualEnv
[E2018-08-23T14:41:58.989754Z 804 0 annotate.go:242]
[E2018-08-23T14:41:58.989798Z 804 0 annotate.go:242] #6 go.chromium.org/luci/vpython/venv/venv.go:957 - venv.mustReleaseLock()
[E2018-08-23T14:41:58.989828Z 804 0 annotate.go:242] #7 go.chromium.org/luci/vpython/venv/venv.go:258 - venv.(*Env).ensure()
[E2018-08-23T14:41:58.989870Z 804 0 annotate.go:242] #8 go.chromium.org/luci/vpython/venv/venv.go:328 - venv.(*Env).withImpl()
[E2018-08-23T14:41:58.989897Z 804 0 annotate.go:242] #9 go.chromium.org/luci/vpython/venv/venv.go:168 - venv.With()
[E2018-08-23T14:41:58.989941Z 804 0 annotate.go:242] #10 go.chromium.org/luci/vpython/run.go:63 - vpython.Run()
[E2018-08-23T14:41:58.989985Z 804 0 annotate.go:242] #11 go.chromium.org/luci/vpython/application/application.go:318 - application.(*application).mainImpl()
[E2018-08-23T14:41:58.990014Z 804 0 annotate.go:242] #12 go.chromium.org/luci/vpython/application/application.go:402 - application.(*Config).Main.func1()
[E2018-08-23T14:41:58.990057Z 804 0 annotate.go:242] #13 go.chromium.org/luci/vpython/application/support.go:47 - application.run()
[E2018-08-23T14:41:58.990102Z 804 0 annotate.go:242] #14 go.chromium.org/luci/vpython/application/application.go:401 - application.(*Config).Main()
[E2018-08-23T14:41:58.990147Z 804 0 annotate.go:242] #15 vpython/main.go:93 - main.mainImpl()
[E2018-08-23T14:41:58.990173Z 804 0 annotate.go:242] #16 vpython/main.go:99 - main.main()
[E2018-08-23T14:41:58.990218Z 804 0 annotate.go:242] #17 runtime/proc.go:198 - runtime.main()
[E2018-08-23T14:41:58.990263Z 804 0 annotate.go:242] #18 runtime/asm_amd64.s:2361 - runtime.goexit()
]

Downloading 1 files took 16.720923 second(s)
Traceback (most recent call last):
  File "src/packager/third_party/binutils/download.py", line 130, in <module>
    sys.exit(main(sys.argv[1:]))
  File "src/packager/third_party/binutils/download.py", line 117, in main
    return FetchAndExtract(arch)
  File "src/packager/third_party/binutils/download.py", line 82, in FetchAndExtract
    '-s', sha1file])
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['download_from_google_storage', '--no_resume', '--no_auth', '--bucket', 'chromium-binutils', '-s', '/shaka-packager/src/packager/third_party/binutils/Linux_x64/binutils.tar.bz2.sha1']' returned non-zero exit status 1
Error: Command '/usr/bin/python src/packager/third_party/binutils/download.py' returned non-zero exit status 1 in /shaka-packager
Hook '/usr/bin/python src/packager/third_party/binutils/download.py' took 16.97 secs
/shaka-packager #
kqyang commented 6 years ago

@wader Try downloading Shaka Packager source and its dependencies (i.e. run gclient sync) in the host machine instead of in the docker and then run the docker image on the downloaded source. It should resolve your issue.

$ gclient config https://www.github.com/google/shaka-packager.git --name=src --unmanaged
$ gclient sync -r 715ed939f1712a398281e56bbccaec25116828ea
$ docker run -v $PWD:/shaka-packager -w /shaka-packager/src test gclient runhooks
$ docker run -v $PWD:/shaka-packager -w /shaka-packager/src test ninja -C out/Release

You can also update DEPS file to remove the download of "binutils", which is not needed for compilation in Alpine Linux. Then you'll be able to run gclient sync in the docker image. (Unfortunately there does not seem to be a way to run DEPS hook conditionally on GYP parameters)

My goal is to build static binaries using the alpine build toolchain. My plan is to run ninja with GYP_GENERATORS="make" and then export CFLAGS/LDFLAGS etc to produce static binaries. Should that work? will all dependencies also use the flags?

I have never tried this but I think it should work.

dbpolito commented 6 years ago

@kqyang

Now that we have alpine builds, would be very cool having alpine tag at https://hub.docker.com/r/google/shaka-packager/ ❤️

kqyang commented 6 years ago

Yes, we are looking into adding a minimal docker image using Alpine :)

wader commented 6 years ago

@kqyang Hi, thanks for the tips! I would really like to do everything in docker as i'm building this in an internal managed build system at work so installing things on the host would not work. So i tried to changed the DEPS file but for some reason i can't get it to use it. Does it have to be commit and then point gclient sync to it? tried that to but still failed.

But i figured out a hacky way to make it build static binaries using a fixed ubuntu/alpine multi-stage build. It would be nice to do everything in only alpine containers so if you know what im doing wrong with the DEPS files please let me know 😄 

# This is a bit special:
# We first use the "offical" way to get the source code (ubuntu image) and generate ninja
# files (the "src" stage). Then we use a alpine image and copy the source from src and
# then build the static binary.
# This is due to current shaka-packager DEPS file having some issues on alpine.

FROM ubuntu:18.04 as src

ARG SHAKA_PACKAGER_VERSION=715ed939f1712a398281e56bbccaec25116828ea

RUN \
    apt-get update && \
    apt-get install -y \
    build-essential \
    curl \
    git \
    python

# install depot_tools http://www.chromium.org/developers/how-tos/install-depot-tools
RUN git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
ENV PATH /depot_tools:$PATH

# gpy ninja generator will look at these
ENV CFLAGS="-O3 -static-libgcc -fno-strict-overflow -fstack-protector-all -fPIE"
ENV CXXFLAGS="-O3 -static-libgcc -fno-strict-overflow -fstack-protector-all -fPIE"
ENV LDFLAGS="-static -Wl,-z,relro -Wl,-z,now -fPIE -pie"

# alpine specific config
ENV GYP_DEFINES="linux_use_bundled_binutils=0 linux_use_bundled_gold=0 clang=0 use_experimental_allocator_shim=0 use_allocator=none musl=1"

RUN mkdir shaka_packager
WORKDIR shaka_packager
RUN gclient config https://www.github.com/google/shaka-packager.git --name=src
RUN gclient sync -r $SHAKA_PACKAGER_VERSION --no-history

FROM alpine:3.8 as builder
RUN \
    apk add --no-cache \
    bash \
    build-base \
    findutils \
    ninja \
    python \
    bsd-compat-headers \
    linux-headers \
    libexecinfo-dev

RUN sed -i \
    '/malloc_usable_size/a \\nstruct mallinfo {\n  int arena;\n  int hblkhd;\n  int uordblks;\n};' \
    /usr/include/malloc.h

COPY --from=src /shaka_packager/src /shaka_packager/src
WORKDIR /shaka_packager/src
RUN ninja -C out/Release

FROM scratch
COPY --from=builder /shaka_packager/src/out/Release/packager /
# sanity test that the binary work in scratch container
RUN ["/packager"]
ENTRYPOINT ["/packager"]
kqyang commented 6 years ago

@wader Wow, that is a clever approach!

There are other workarounds you can do, e.g. clone Shaka Packager repository first, edit DEPS then run gclient sync (this is actually what we do in Travis: https://github.com/google/shaka-packager/blob/master/.travis.yml).

I thought about the pros and cons of having bundled binutils / gold in Linux again. The improve is only about 8%. Given that Shaka Packager codebase is not really big, I think we can afford not to have these bundled binaries by default. I have disabled bundled binutils and gold in https://github.com/google/shaka-packager/commit/9b655189cfce4046785c5e75ca2baa521e7f37b1.

So now you should be able to build Shaka Packager directly on Alpine without the need to modify DEPTH.

wader commented 6 years ago

@kqyang Thanks, not sure if i should be proud or not 😄 but after your change i can now build without the multi-stage hack. Note that the ninja binary from the depot_tools repo does not seems to run on alpine because of missing library. Current workaround is to install alpine ninja package and give it priority.

New Dockerfile:

FROM alpine:3.8 as builder
RUN \
    apk add --no-cache \
    bash \
    build-base \
    findutils \
    curl \
    git \
    ninja \
    python \
    bsd-compat-headers \
    linux-headers \
    libexecinfo-dev

ARG SHAKA_PACKAGER_VERSION=40a3b42980efc19407b0ba4d550ebb71070754cf
ARG DEPOT_TOOLS_VERSION=2d0e03cb5ddc5faef4d7bbd54d0177885d3975bc

# install depot_tools http://www.chromium.org/developers/how-tos/install-depot-tools
RUN \
    git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git && \
    cd /depot_tools && \
    git checkout $DEPOT_TOOLS_VERSION
# depot_tools path last so that alpine ninjs is used (depot_tools ninja does not run on alpine atm)
ENV PATH $PATH:/depot_tools

RUN sed -i \
    '/malloc_usable_size/a \\nstruct mallinfo {\n  int arena;\n  int hblkhd;\n  int uordblks;\n};' \
    /usr/include/malloc.h

# gpy ninja generator will look at these
ENV CFLAGS="-O3 -static-libgcc -fno-strict-overflow -fstack-protector-all -fPIE"
ENV CXXFLAGS="-O3 -static-libgcc -fno-strict-overflow -fstack-protector-all -fPIE"
ENV LDFLAGS="-static -Wl,-z,relro -Wl,-z,now -fPIE -pie"
# alpine specific config
ENV GYP_DEFINES="clang=0 use_experimental_allocator_shim=0 use_allocator=none musl=1"

WORKDIR /shaka_packager
RUN gclient config https://www.github.com/google/shaka-packager.git --name=src
RUN gclient sync -r $SHAKA_PACKAGER_VERSION --no-history
RUN ninja -C src/out/Release

FROM scratch
COPY --from=builder /shaka_packager/src/out/Release/packager /
# sanity test that the binary work in scratch container
RUN ["/packager"]
ENTRYPOINT ["/packager"]
wader commented 6 years ago

@kqyang feel free to use parts of if you decide to build official "static" binary images