Closed giovannicandido closed 8 years ago
This is more of a musl libc issue. The Oracle Java binaries only run on glibc at the moment. We are looking at possibly rebuilding https://aur.archlinux.org/packages/jd/jdk/PKGBUILD and making it available as a standalone package. I'll keep this issue open for now as we have had requests for Java 8 multiple times already.
Relevant to this is that the java 7 package is slightly broken too- the certstore is broken causing very hard to diagnose errors. The alpine linux irc guys say to copy it in from a debian install
(curious) @cultureulterior could you providde more details in your findings plz.
@AntonioMeireles the java certstore apparently is not regenerated from the CA certs on the machine itself, causing the java ssl to fail to trust certificates that should be trusted
I think I may start supporting progrium/busybox again just for Java 8 ... can you guys try that as your base image?
+1
Can anyone point me to relevant information about these musl libc limitations? I have the similar issue with Anaconda prebuilt packages on Alpine Linux:
# ldd /usr/local/miniconda/pkgs/python-2.7.9-1/bin/python
/lib64/ld-linux-x86-64.so.2 (0x7f9d68e45000)
libpython2.7.so.1.0 => /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0 (0x7f9d68a63000)
libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f9d68e45000)
libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f9d68e45000)
libutil.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f9d68e45000)
libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f9d68e45000)
libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f9d68e45000)
Error relocating /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0: __finite: symbol not found
Error relocating /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0: __rawmemchr: symbol not found
Error relocating /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0: __isinff: symbol not found
Error relocating /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0: tmpnam_r: symbol not found
Error relocating /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0: __isnan: symbol not found
Error relocating /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0: __isinf: symbol not found
I do understand that such prebuilt packages are evil, but there is not much I can do in this case except moving to another image for now. :(
However, I would like to understand limits of musl libc better.
/cc @flaviof @albertocsm
@cultureulterior not directly related to this issue, but it is possible to manually import all ca certificates to java trusted keystore, as I did it here
We have a new glibc package if anyone would like to test this out. You can grab the package at https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-2.21-r2.apk and install with apk add --allow-untrusted glibc-2.21-r2.apk
. This should allow Oracle Java 8 and other glibc x86_64 native binaries to run on Alpine.
@andyshinn Awesome! It works for me with Miniconda, though I had to bring libz.so from my ubuntu to /usr/lib/libz.so.1 as Alpine Linux has libz.so compiled against musl...
@andyshinn
UPD: I didn't need to bring libz.so from ubuntu, but needed to create symlinks:
/usr/lib/libc.musl-x86_64.so.1
pointing at /lib/libc.musl-x86_64.so.1
/usr/lib/libz.so.1
pointing at /lib/libz.so.1
Note: For some reason, LD_LIBRARY_PATH doesn't work in this case for me.
Are you able to share a Dockerfile
or repository with this? I'd like to see how this is working to better support these edge cases.
@andyshinn Sure!
@andyshinn
FROM alpine
RUN apk update && \
apk add bash wget
WORKDIR /tmp
RUN wget --no-check-certificate "https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-2.21-r2.apk" && \
apk add --allow-untrusted glibc-2.21-r2.apk
RUN wget "http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh"
# This will fail with "ImportError: libz.so.1: cannot open shared object file: No such file or directory"
#RUN bash ./Miniconda-latest-Linux-x86_64.sh -b -p /usr/local/miniconda
# This will succeed
RUN ln -s /lib/libc.musl-x86_64.so.1 /usr/lib/libc.musl-x86_64.so.1 && \
ln -s /lib/libz.so.1 /usr/lib/libz.so.1
RUN bash ./Miniconda-latest-Linux-x86_64.sh -b -p /usr/local/miniconda
I removed the post-install ldconfig
but it might actually cover this case. Can you try adding package https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-bin-2.21-r2.apk in addition to the glibc
one and then run /usr/glibc/usr/bin/ldconfig /lib /usr/glibc/usr/lib
before installing and see if it works?
@andyshinn Now it works without symlinking! Great!
Just for the history (it works like a charm):
FROM alpine
RUN apk add --update bash wget ca-certificates
WORKDIR /tmp
RUN wget "https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-2.21-r2.apk" && \
apk add --allow-untrusted glibc-2.21-r2.apk && \
wget "https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-bin-2.21-r2.apk" && \
apk add --allow-untrusted glibc-bin-2.21-r2.apk && \
/usr/glibc/usr/bin/ldconfig /lib /usr/glibc/usr/lib
# Hotfix for glibc hack that fixes the order of DNS resolving (i.e. check /etc/hosts first and then lookup DNS-servers).
# To fix this we just create /etc/nsswitch.conf and add the following line:
RUN echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf
RUN wget "http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh"
RUN bash ./Miniconda-latest-Linux-x86_64.sh -b -p /usr/local/miniconda
In real world, I suggest you write something like:
FROM alpine
# Here we use several hacks collected from https://github.com/gliderlabs/docker-alpine/issues/11:
# 1. install GLibc (which is not the cleanest solution at all)
# 2. hotfix /etc/nsswitch.conf, which is apperently required by glibc and is not used in Alpine Linux
RUN apk add --update bash wget ca-certificates && \
wget "https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-2.21-r2.apk" \
"https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-bin-2.21-r2.apk" && \
apk add --allow-untrusted glibc-2.21-r2.apk glibc-bin-2.21-r2.apk && \
/usr/glibc/usr/bin/ldconfig /lib /usr/glibc/usr/lib && \
echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
wget "http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh" && \
bash ./Miniconda-latest-Linux-x86_64.sh -b -p /usr/local/miniconda && \
rm ./Miniconda-latest-Linux-x86_64.sh && \
apk del bash wget ca-certificates && \
rm /var/cache/apk/*
This way you avoid packaging unnecessary packages (bash, wget), giant installer (rm miniconda.sh
), and outdated apk repository lists.
A couple of things I want to point out:
pip
and then start building more Python extensions (ones requiring build tools such as build-base
) those will be linked against musl (be aware of DNS implications this has).libz.so
from Arch to break the dependency on musl. Soon, i'll have an example of what a package like this looks like@andyshinn I don't really use Miniconda it was just a project that I played with. My concern was mostly about the limits of musl libc than about running Miniconda on Alpine Linux. Anyway, thank you for your effort to get things easier in Alpine. I believe, Java people will appreciate this a lot.
Fair enough, it is something that applies to anyone trying to use glibc on Alpine so just wanted to get it out there. We are definitely more targeting the edge cases of one-off glibc binaries (think Consul) or Java 8 applications that might have all dependencies bundled in a war file already. But it is helpful to see where the limits are, so thank you for that!
@andyshinn Another example, which may be considered harmful, but nevertheless it made the shortcut possible, is my frolvlad/alpine-mono image, where I use mono package from Arch Linux.
Here is the Apline OracleJDK 8 image (352MB 167MB) - frolvlad/alpine-oraclejkd8
Thanks! I have successfully build alpine oracle JRE image — size 170MB (old, based on progrium/busybox — 162MB). https://registry.hub.docker.com/u/develar/java/
hi @andyshinn this works like a charm, please let us know once you have merged it into a released alpine version!
Thanks @frol , we're using it now.
This is fantastic news!
Edit: The solution in https://github.com/boot2docker/boot2docker/issues/451#issuecomment-68874404 works (when using VirtualBox, whether using boot2docker or not).
A slight obstacle is that I'm getting a lot of network issues out of the base "alpine" Docker image, though. APK is intermittently not able to read data. For example:
$ docker run --rm -ti alpine sh
/ # apk -U upgrade
fetch http://dl-4.alpinelinux.org/alpine/v3.1/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-4.alpinelinux.org/alpine/v3.1/main: IO ERROR
WARNING: Ignoring APKINDEX.689bb31a.tar.gz: No such file or directory
OK: 5 MiB in 15 packages
/ # apk -U upgrade
fetch http://dl-4.alpinelinux.org/alpine/v3.1/main/x86_64/APKINDEX.tar.gz
(1/7) Upgrading musl (1.1.5-r2 -> 1.1.5-r4)
(2/7) Upgrading busybox (1.22.1-r14 -> 1.22.1-r15)
ERROR: busybox-1.22.1-r15: Resource temporarily unavailable
(3/7) Upgrading alpine-conf (3.1.0-r2 -> 3.1.0-r4)
ERROR: alpine-conf-3.1.0-r4: Resource temporarily unavailable
(4/7) Upgrading libcrypto1.0 (1.0.1k-r0 -> 1.0.1m-r1)
ERROR: libcrypto1.0-1.0.1m-r1: Resource temporarily unavailable
(5/7) Upgrading libssl1.0 (1.0.1k-r0 -> 1.0.1m-r1)
(6/7) Upgrading musl-utils (1.1.5-r2 -> 1.1.5-r4)
(7/7) Upgrading alpine-base (3.1.2-r0 -> 3.1.3-r0)
ERROR: alpine-base-3.1.3-r0: Resource temporarily unavailable
4 errors; 5 MiB in 15 packages
/ # apk upgrade
(1/4) Upgrading busybox (1.22.1-r14 -> 1.22.1-r15)
Executing busybox-1.22.1-r15.post-upgrade
(2/4) Upgrading alpine-conf (3.1.0-r2 -> 3.1.0-r4)
ERROR: alpine-conf-3.1.0-r4: Resource temporarily unavailable
(3/4) Upgrading libcrypto1.0 (1.0.1k-r0 -> 1.0.1m-r1)
(4/4) Upgrading alpine-base (3.1.2-r0 -> 3.1.3-r0)
Executing busybox-1.22.1-r15.trigger
1 errors; 5 MiB in 15 packages
/ # apk upgrade
(1/1) Upgrading alpine-conf (3.1.0-r2 -> 3.1.0-r4)
Executing busybox-1.22.1-r15.trigger
OK: 5 MiB in 15 packages
/ #
When there are several dependencies (e.g. bash
or ca-certificates
), some of the dependencies make it, some don't. If I keep trying, though, it eventually works.
When I do eventually get curl
, it is also having problems getting data (from a completely different site).
This may be name resolution preferring AAAA records over A records? Here you see ping
failing until an IPv4 address is resolved. There is no route to IPv6 networks from the host I'm running on. The ping can subsequently fail if the IPv6 address is resolved again.
$ docker run --rm -ti canva/alpine sh
/ # ping google.com
PING google.com (2404:6800:4006:805::1009): 56 data bytes
ping: sendto: Network unreachable
/ # ping google.com
PING google.com (2404:6800:4006:805::1008): 56 data bytes
ping: sendto: Network unreachable
/ # ping google.com
PING google.com (2404:6800:4006:805::1008): 56 data bytes
ping: sendto: Network unreachable
/ # ping google.com
PING google.com (2404:6800:4006:805::1005): 56 data bytes
ping: sendto: Network unreachable
/ # ping google.com
PING google.com (74.125.237.160): 56 data bytes
64 bytes from 74.125.237.160: seq=0 ttl=61 time=6.023 ms
64 bytes from 74.125.237.160: seq=1 ttl=61 time=5.549 ms
64 bytes from 74.125.237.160: seq=2 ttl=61 time=6.203 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 5.549/5.925/6.203 ms
/ #
Even supposedly "disabling" IPv6 by updating /etc/sysctl.conf
seems to have no affect:
$ docker run --rm -ti canva/alpine sh
/ # cat /etc/sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
kernel.panic = 120
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1
/ # ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:AC:11:00:1B
inet addr:172.17.0.27 Bcast:0.0.0.0 Mask:255.255.0.0
inet6 addr: fe80::42:acff:fe11:1b/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:7 errors:0 dropped:0 overruns:0 frame:0
TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:578 (578.0 B) TX bytes:578 (578.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
/ # ping google.com
PING google.com (216.58.220.110): 56 data bytes
64 bytes from 216.58.220.110: seq=0 ttl=61 time=3.537 ms
^C
--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 3.537/3.537/3.537 ms
/ # ping google.com
PING google.com (2404:6800:4006:801::200e): 56 data bytes
ping: sendto: Network unreachable
/ #
When I do the same activity on @progrium's busybox image with the same busybox version as the Alpine image (1.22.1), no problems are experienced.
@delitescere Try to remove the image and pull it again. I've had the same issue with the base alpine image on one host and it helped.
@frol sadly, not for me. After removing and pulling again:
$ docker history alpine
IMAGE CREATED CREATED BY SIZE
b3dbab3810fc 8 days ago /bin/sh -c #(nop) ADD file:3cca9bd4719b968870 5.03 MB
FWIW, I think it's a resolver(3)
issue with gethostbyname
behaving like RES_USE_INET6
has been set by default, when it shouldn't be.
Edit: The solution in https://github.com/boot2docker/boot2docker/issues/451#issuecomment-68874404 works (when using VirtualBox, whether using boot2docker or not).
@delitescere looks like some helpful information here, but can you continue this conversation over on #28 so it doesn't get lost in this discussion about Java.
For folks that might come looking, heka also appears to have this glibc requirement (at least as of 0.9.2).
The steps from the above comment did the trick though!
@matschaffer do you mean that their binaries require glibc or sources don't compile against musl libc?
Musl libc claims limited binary compatibility with glibc, but I have never seen a project that could work with musl libc without recompilation.
BTW, I've updated my comment (that you linked to) with a recommended way of creating the final image.
I believe "their binaries require glibc" - though I'm a little fuzzy on the technicalities.
I haven't actually tried to build it myself so I can't speak to if the sources compile against musl libc or not.
I only used the RUN wget ... && apk ...
part which seems to be unchanged since I copied it. That was enough to get the released hekad binary happy.
Huh... although I'm noting that once it's running it doesn't seem to resolve against /etc/hosts
. Could be a side effect of this.
@marcusandre Try my OracleJDK8 fix:
# Alpine Linux doesn't use pam, which means that there is no /etc/nsswitch.conf,
# but Java relies on /etc/nsswitch.conf to check the order of DNS resolving
# (i.e. check /etc/hosts first and then lookup DNS-servers).
# To fix this we just create /etc/nsswitch.conf and add the following line:
RUN echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf
Looking at this, it seems that it wasn't a Java bug, but glibc hack side effect...
@andyshinn You know better about glibc packages. Do you think it is possible that this nsswitch.conf
issue is introduced because of them?
@frol hey thanks adding that nsswitch line did indeed get hosts resolution working again. So now with a pair of ssh tunnels I can end-to-end test from boot2docker against Amazon's email service :+1:
Where does JDK/JRE8 support stand? I see here some folks that managed to get libc + some hacks working. Is that the official way, or is more official support for glibc in the works?
Working great for me. See https://hub.docker.com/r/delitescere/jvm-prep/
@delitescere well you write in your own Dockerfile that this is a temporary workaround. so the question is: is it in fact a temporary workaround, and when will it be fixed in apline base image.
Well, I think the goal would be to simplify the installation of glibc, but I don't think it would be installed in the base image. It would definitely be nice to simply apk add glibc
but installing it by default would increase the size of the image, and might cause confusion or issues in cases where it wasn't actually required.
@mgood agreed. It should not be in the base image, @CodingFabian, but it could be easier to install.
In the first place, the issue of OracleJDK not working on Alpine is not the issue of Alpine, but the issue of OracleJDK, which is not compiled for musl libc. Thus, if you wish to get things fixed properly, you have to ask Oracle to support musl libc. Meanwhile, everything that we implement here would be a hack either you install glibc from a repository or wget
+ apk add <files>
. Nevertheless, there are two images that should save almost everybody from headaches: frolvlad/alpine-oraclejdk8
(167MB Oracle JDK image) and develar/java
(117MB Oracle JRE image). I try to keep my image up-to-date, and I use it in production in my current projects.
Mine, using Zulu OpenJDK, are 162MB (JDK https://hub.docker.com/r/delitescere/jdk/) and 86MB (JRE https://hub.docker.com/r/delitescere/java/) respectively.
Technically I understand this thought but it's not realistic to expect a huge company like oracle to support every lib combo out there especially one that isn't the "top" one in terms of raw adoption. (I do get the appeal of musl)
If you want to be able to support a major constituency like oraclejdk users the smaller ship not the larger needs to turn.
My thought on the matter is that busybox had this support and Alpine is positioning itself as the successor to busybox then conceptually there shouldn't be a major loss of capability.
Sent from my iPhone
On Nov 3, 2015, at 11:11 PM, Vlad Frolov notifications@github.com wrote:
In the first place, the issue of OracleJDK not working on Alpine is not the issue of Alpine, but the issue of OracleJDK, which is not compiled for musl libc. Thus, if you wish to get things fixed properly, you have to ask Oracle to support musl libc. Meanwhile, everything that we implement here would be a hack either you install glibc from a repository or wget + apk add
. Nevertheless, there are two images that should save almost everybody from headaches: frolvlad/alpine-oraclejdk8 (167MB Oracle JDK image) and develar/java (117MB Oracle JRE image). I try to keep my image up-to-date, and I use it in production in my current projects. — Reply to this email directly or view it on GitHub.
You may want to take a look at one of our production alpine/JDK8 install scripts for our alpinejava docker image: https://github.com/garywiz/chaperone-docker/blob/master/alpinejava/setup/install.sh
We put this together quite a while ago, and it incorporates many of the suggestions in this thread as well as other threads (including the circle-artifacts glibc, nsswitch patch, etc). We have about 10 clients using production derivatives from this and it might serve as a template, who knows.
Cool. I'll check it out.
Sent from my iPhone
On Nov 4, 2015, at 5:08 PM, Gary Wisniewski notifications@github.com wrote:
You may want to take a look at one of our production alpine/JDK8 install scripts for our alpinejava docker image: https://github.com/garywiz/chaperone-docker/blob/master/alpinejava/setup/install.sh
We put this together quite a while ago, and it incorporates many of the suggestions in this thread as well as other threads (including the circle-artifacts glibc, nsswitch patch, etc). We have about 10 clients using production derivatives from this and it might serve as a template, who knows.
— Reply to this email directly or view it on GitHub.
@gzoller Use frolvlad/alpine-glibc
and you will feel it just like busybox.
@garywiz It is just the same as my frolvlad/alpine-oraclejdk8
image. 3.9k pulls of my image and two of my production projects shows that the image is stable.
@frol Thanks you! Very excited it runs go scripts too, which my attempt to run a Alpine/glibc mashup did not. This is going to be a nice base for me.
@andyshinn
I read your comments on Apr 10, and I think I have the issue you mentioned.
Since python extensions are not built with glibc, I can't make tensorflow (https://github.com/tensorflow/tensorflow) work in alpine.
The issue i raised : https://github.com/tensorflow/tensorflow/issues/103
any suggestions?
Yes, actually! Check out frolvlad/alpine-glibc
This worked great for me.
On Nov 11, 2015, at 7:18 AM, Bill W notifications@github.com wrote:
@andyshinn
I read your comments on Apr 10, and I think I met the issue you mentioned.
Since python extensions is not built with glibc, I can't make tensorflow (https://github.com/tensorflow/tensorflow) work in alpine.
The issue i raised : tensorflow/tensorflow#103
any suggestions?
— Reply to this email directly or view it on GitHub.
Trying to install oracle JDK 8 and I get this error when run:
. /opt/jdk/bin/java -version /bin/ash: /opt/jdk/bin/java: : not found /bin/ash: /opt/jdk/bin/java: : not found @@@@@@@?@@@@ |?@@DDP?td??@?@1Q?t/lib64/ld-linux-x86-64.so.2GNU GNU?t?)N?ݓ??;r?Z???h nonexistent directory /bin/ash: /opt/jdk/bin/java: line 1: ELF: not found /bin/ash: /opt/jdk/bin/java: ?: not found /bin/ash: /opt/jdk/bin/java: line 3: ?: not found /bin/ash: /opt/jdk/bin/java: syntax error: unexpected end of file (expecting ")")
The ldd command give the result:
What could be done?