search

sgerrand / alpine-pkg-glibc

A glibc compatibility layer package for Alpine Linux
2.07k stars 280 forks source link

Error while trying to install it with armhf arch #113

Open marianopeck opened 5 years ago

marianopeck commented 5 years ago

Hi,

I am installing the latest version on a Raspberry Pi 3B+ this way:

# Install Dependencies
RUN apk --no-cache add ca-certificates wget \
    && wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub \
    && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.29-r0/glibc-2.29-r0.apk \
    && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.29-r0/glibc-bin-2.29-r0.apk \
    && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.29-r0/glibc-i18n-2.29-r0.apk \
    && apk add glibc-bin-2.29-r0.apk glibc-i18n-2.29-r0.apk glibc-2.29-r0.apk

and I am getting this error:

fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/armhf/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/armhf/APKINDEX.tar.gz
(1/3) Installing glibc (2.29-r0)
(2/3) Installing glibc-bin (2.29-r0)
(3/3) Installing glibc-i18n (2.29-r0)
Executing glibc-bin-2.29-r0.trigger
/usr/glibc-compat/sbin/ldconfig: line 1: syntax error: unexpected "("
ERROR: glibc-bin-2.29-r0.trigger: script exited with error 2
OK: 63 MiB in 61 packages
Removing intermediate container 5d82962062fa
 ---> 524153539aee
Step 6/12 : RUN /usr/glibc-compat/bin/localedef -i en_US -f UTF-8 en_US.UTF-8
 ---> Running in 6640abb3a3db
/usr/glibc-compat/bin/localedef: line 1: syntax error: unexpected ")"
The command '/bin/sh -c /usr/glibc-compat/bin/localedef -i en_US -f UTF-8 en_US.UTF-8' returned a non-zero code: 2

Any idea of what might be wrong? Maybe this is the same case as issue https://github.com/sgerrand/alpine-pkg-glibc/issues/54 or https://github.com/sgerrand/alpine-pkg-glibc/issues/81

Thanks

marianopeck commented 5 years ago

Looks like there is no pre-compiled binary files for ARM...which is strange considering this is for Alpine...Anyway, I will stick with Debian Slim for the moment. Thanks anyway.

gorquan commented 5 years ago

Now I am using the docker on the Raspberry Pi to compile glibc's aarch64 binary package. Later in the day I will try to build an alpine apk package using this binary package.

marianopeck commented 5 years ago

and armhf? :)

gorquan commented 5 years ago

Yes, I will spend some time building glibc for arm64 and armhf

marianopeck commented 5 years ago

Thanks! That would be awesome. If/whenever you reach to something testeable, share with me the instructions and I can give it a try.

SaturnusDJ commented 5 years ago

Same problem and error here. Using a Dockerfile that detects the architecture (uname -m) and downloads a closed source glibc application, and the glibc from this repo. Works on x86_64, not on armv7l.

@gorquan Any progress?

@sgerrand Thanks for this repo, at least x86_64 works, which was a hell until I found this. Any suggestions for arm?

marianopeck commented 5 years ago

Hi @SaturnusDJ Honestly, I was quite disappointed. I remember that I eventually found a link from someone that compiled this project for ARM. And I give it a try on Alpine with Docker. It did install correctly but the resulting Docker image size of Alpine + glibc was about the SAME size of an Debian Slime docker image. So...in my case, it was not worth. I was looking for Alpine for having a smaller image but ended up with the same. And not, I cannot update my code to use musl...its a huge effort and risky and we have already found differences in their behaviors..

SaturnusDJ commented 5 years ago

@marianopeck Really this project and then the same size as Debian Slim? I think it was just emulation then. Possibly the same as I have seen a few weeks ago: someone using qemu to run glibc applications on Alpine.

marianopeck commented 5 years ago

Hi @SaturnusDJ I hope you are right as I really wanted to have it working. I still have the example around in case someone could come with pre-build packages...

SaturnusDJ commented 5 years ago

@marianopeck My Docker image that runs a closed source glibc application is 13,5 MB excluding that application and including libstdc++ (0,5-1MB), glibc and glibc-bin and excluding libc-utils, musl-utils and scanelf which were removed upon finalizing the Docker image, because not needed for that application. And it is working on x86_64. So if there would be an ARM version of this glibc, well...game on.

marianopeck commented 5 years ago

Wow, that would be lovely :) I will re-open the case just to see if we can gain more interest.

SaturnusDJ commented 5 years ago

Yes, I don't know what @sgerrand did, but it sure was something good. I am going to have a look at https://github.com/sgerrand/docker-glibc-builder. Maybe building that on ARM results in a glibc for ARM.

marianopeck commented 5 years ago

If that's the case, I can try to help. I do have ARM and ARM64 devices and Docker already installed on them.. Let me know.

SaturnusDJ commented 5 years ago

I am trying on ARMv7l (RPi 3B+)

~# docker run --rm --env STDOUT=1 sgerrand/glibc-builder 2.29 /usr/glibc-compat > glibc-bin.tar.gz
Unable to find image 'sgerrand/glibc-builder:latest' locally
latest: Pulling from sgerrand/glibc-builder
6cf436f81810: Pulling fs layer
987088a85b96: Pulling fs layer
b4624b3efe06: Pulling fs layer
d42beb8ded59: Pulling fs layer
29d4e839ed6d: Pulling fs layer
2ccda1bca1a3: Pulling fs layer
028a7df9c282: Pulling fs layer
29d4e839ed6d: Waiting
2ccda1bca1a3: Waiting
028a7df9c282: Waiting
d42beb8ded59: Waiting
b4624b3efe06: Verifying Checksum
987088a85b96: Verifying Checksum
987088a85b96: Download complete
d42beb8ded59: Verifying Checksum
d42beb8ded59: Download complete
2ccda1bca1a3: Verifying Checksum
2ccda1bca1a3: Download complete
028a7df9c282: Verifying Checksum
028a7df9c282: Download complete
6cf436f81810: Verifying Checksum
6cf436f81810: Download complete
6cf436f81810: Pull complete
987088a85b96: Pull complete
b4624b3efe06: Pull complete
d42beb8ded59: Pull complete
29d4e839ed6d: Verifying Checksum
29d4e839ed6d: Download complete
29d4e839ed6d: Pull complete
2ccda1bca1a3: Pull complete
028a7df9c282: Pull complete
Digest: sha256:36cb24e76f751eafa4b40c39331964dff8ff91331ee7db2bbb02f1da7d6964a3
Status: Downloaded newer image for sgerrand/glibc-builder:latest
standard_init_linux.go:190: exec user process caused "exec format error"
marianopeck commented 5 years ago

Not that I can't help much but I have also a ARMv7l (RPi 3B+) for building/testing as well as other SD with Ubuntu 18.04 compiled for aarch64 that also runs on the same Rpi 3B+ So I have both flavors.

SaturnusDJ commented 5 years ago

I ran it on x86_64 and that results in a 50MB glibc-bin.tar.gz. Cloned the repo on the RPi 3B+, it's running now but the outcome will either be the same (50MB) or it will fail or it will produce a faulty result.

I don't know how @sgerrand managed to generate such small results as on the https://github.com/sgerrand/alpine-pkg-glibc/releases page. Maybe he is selectively packing the resulting files instead of packing all of them.

SaturnusDJ commented 5 years ago

@marianopeck The good news is...my guess above was at least kind of right. I am down to ~11MB for the glibc ARM package now by removing files that are also not in the glibc-bin.apk from the release page, and the closed source application is running on ARM. This is without the files that are in glibc.apk from the release page. My build result is version 2.28, you can change to 2.29 in the Dockerfile maybe. I did not try yet. The kind of bad news is that execl (whatever that may be) is missing, also when using the full ~50MB result execl is still missing. This gave an error with the application I am using but it seemed to be working nonetheless. As I only tested one application, much more might be missing or broken. I am not knowledgeable with this kind of stuff.

Anyways...if you dare to try:

git clone https://github.com/sgerrand/docker-glibc-builder.git
docker build --rm -t glibc-builder:1 docker-glibc-builder/ --no-cache
docker run -d --name="glibc-builder_1" glibc-builder:1

Or leave out the -d to get the output in the current command line. The text output will be a lot and it will take at least half an hour. After that the container shuts down. I was unable to extract the tar from the container so I looked for it on the host filesystem.

find / -iname "*glibc*"

Then likely you will find a glibc-bin-2.28.tar.gz somewhere.

Inside it is usr/glibc-compat/ I deleted everything except the bin and sbin directories. When mimicking glibc.apk, more files will have to stay.

marianopeck commented 5 years ago

Hi @SaturnusDJ Great, I just fired it up. Will let you know. In the meanwhile, let me ask: once you got the necessary shared libs in usr/glibc-compat/ in your HOST how did you pass it back to the Dockerfile to be run on the Pi? I will need to test that my application. In other words, in the Dockerfile of my app, when I was previously doing:

# Install Dependencies
RUN apk --no-cache add ca-certificates wget \
    && wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub \
    && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.29-r0/glibc-2.29-r0.apk \
    && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.29-r0/glibc-bin-2.29-r0.apk \
    && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.29-r0/glibc-i18n-2.29-r0.apk \
    && apk add glibc-bin-2.29-r0.apk glibc-i18n-2.29-r0.apk glibc-2.29-r0.apk

I guess you replaced that by mounting from your host (Pi too)? Could you share that? Finally, you also said:

excluding libc-utils, musl-utils and scanelf which were removed upon finalizing the Docker image

Could you share that as well? I would like to do the same thing.

Thanks!

marianopeck commented 5 years ago

BTW, the last statement did nothing :(

    pi@marianopi3:~/Instantiations/glibc $ docker run -d --name="glibc-builder_1" glibc-builder:1
    8cfc08630d9955421c6a1e7b265bacca320b6203611c1faa19c17ad683127f64
            pi@marianopi3:~/Instantiations/glibc $
SaturnusDJ commented 5 years ago

Oh yes I forgot to describe the final step. Just make a volume to share the files with a container. Then copy the generated files into the container root. Thus, move the prepared usr/glibc-compat/ that is on a volume into /.

It's pretty general. You could share the end result .tar.gz, extract it, remove all but bin and sbin and the copy it to /. But when all in error free it would make sense to generate a single .tar.gz that is fully prepared and only needs to be extracted onto /. But first let get it error free.

Why did it not work for you? Did the git clone work? Did the build work? (Takes about a minute or 2 to build.) Are you trying on ARMv7l?

Or maybe it did work, after the last command you should have a container running, it will be generating the file and could take an hour. Try without -d to see what is going on. But now first check if it is running with docker ps.

marianopeck commented 5 years ago

Hi @SaturnusDJ Grrr crap. Yeah, I am trying on ARMv7l Raspbian Stretch on a Rpi3B+. The docker build does work, but the run fails:

pi@marianopi3:~/Instantiations/glibc $ docker run  glibc-builder:1

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
pi@marianopi3:~/Instantiations/glibc $

Weird...

marianopeck commented 5 years ago

I did a docker system prune, run the build again and I still have the same issue...I cannot even run interactive with the image:

pi@marianopi3:~/Instantiations/glibc $ docker run -it glibc-builder:1

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
pi@marianopi3:~/Instantiations/glibc $

But the build finishes correct...so I don't know.

SaturnusDJ commented 5 years ago

That's weird. No hidden errors in the build log or git log? I saw errors here but not severe ones for me. My host OS is DietPi.

marianopeck commented 5 years ago

OK, I think the problem is on my end. If I got to my browser and paste "https://ftpmirror.gnu.org/libc/glibc-2.28.tar.gz" it redirects me to "https://espejito.fder.edu.uy/gnu/libc/glibc-2.28.tar.gz" which throws me a nice:

Access forbidden!
You don't have permission to access the requested object. It is either read-protected or not readable by the server.

If you think this is a server error, please contact the webmaster.

Error 403
espejito.fder.edu.uy
Apache

Yeah...I am in Argentina, next to Uruguay. I will change the URL to see if I can use it with another mirror...

marianopeck commented 5 years ago

OK...working now... I changed the url to "https://ftp.gnu.org/gnu/glibc/glibc-$version.tar.gz" and now it's compiling :)

SaturnusDJ commented 5 years ago

Good find. 👍

Tomorrow I will have a closer look at https://github.com/sgerrand/docker-glibc-builder/. For example https://github.com/sgerrand/docker-glibc-builder/blob/master/builder where it says --enable-multi-arch. Maybe leaving that out speeds up the process and results in smaller end result. No clue yet.

marianopeck commented 5 years ago

Yes. I will also check once it finishes compiling what is the result and how to use it from my app... I guess I will need to export LD_LIBRARY_PATH to the mounted volume...but I cannot do more until this thing finishes compiling. BTW... this is NOTHING. Last week I compiled TensorFlow on the Pi3B+. How long? 13 hours. Seriously. hahahah

SaturnusDJ commented 5 years ago

For what is export LD_LIBRARY_PATH?

SaturnusDJ commented 5 years ago

Btw my container ends with

test ! -x /glibc-build/elf/ldconfig || LC_ALL=C \
  /glibc-build/elf/ldconfig  \
                        /usr/glibc-compat/lib /usr/glibc-compat/lib
/glibc-build/elf/ldconfig: Warning: ignoring configuration file that cannot be opened: /usr/glibc-compat/etc/ld.so.conf: No such file or directory
make[1]: Leaving directory '/glibc-2.29'

So we might have to fix that.

SaturnusDJ commented 5 years ago

https://github.com/sgerrand/docker-glibc-builder/issues/9

marianopeck commented 5 years ago

I also see that warning. BTW, this worked to copy the zip from the container to the host: docker cp glibc-builder_1:/glibc-bin-2.28.tar.gz ./

marianopeck commented 5 years ago

Hmm it just finished.. but it's still huge... See pic. I do need /lib for example. That's already 85MB. This more or less shows my conclusion... Alpine + glibc is almost same size as debian slim...right ?

SaturnusDJ commented 5 years ago

In my case I only needed bin and sbin to run my application. Total = 11MB. Try if that works for you too. So extract glibc-bin-2.28.tar.gz. Delete everything except the first bin and sbin. Then copy the remaining tree to your container root.

AMD 64 Alpine + glibc = ~15MB. ARM...we have to fix it, and then I am sure we reach that low number also without problems.

I cannot yet think of a way to apply the fix at https://github.com/sgerrand/docker-glibc-builder/issues/9. We have to intercept somewhere then. https://github.com/sgerrand/docker-glibc-builder/issues/20 could help us too.

SaturnusDJ commented 5 years ago

Sorry I read wrong. You need lib. Maybe look if there is something big that can be removed without consequences. Compare to the releases in this repo what has not been put in them. Then mimic that.

marianopeck commented 5 years ago

Hi @SaturnusDJ The releases in the repo are as big as my compiling results. .tar.gz is 50MB and uncompressed the whole thing is around 157MB...

marianopeck commented 5 years ago

BTW, the one I said I tried long ago and that worked out of the box was this from @yangxuan8282. The sizes are better but it's a very old glibc version and I cannot find documentation on how he does it. For more info see this link. I also remember testing this one which is newer. You can try with something like this:

RUN apk --no-cache add ca-certificates wget \
    && wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub \
    && wget https://github.com/yangxuan8282/alpine-pkg-glibc/releases/download/2.27-r0/glibc-2.27-r0.apk \
    && wget https://github.com/yangxuan8282/alpine-pkg-glibc/releases/download/2.27-r0/glibc-bin-2.27-r0.apk \
    && wget https://github.com/yangxuan8282/alpine-pkg-glibc/releases/download/2.27-r0/glibc-i18n-2.27-r0.apk \
    && apk add glibc-bin-2.29-r0.apk glibc-i18n-2.29-r0.apk glibc-2.29-r0.apk
gorquan commented 5 years ago

Oh, I am very sorry. I have put the project on hold for a while because I am looking for a job.I have already created a docker that can compile glibc. It works on the arm64 platform and is compiled on my Raspberry Pi 3B. At the same time, I also tried to package the apk of glibc, but there was an error during the packaging process.So I dare not release it. I think you can try my docker, maybe I can help you. docker-glibc-builder

marianopeck commented 5 years ago

Hi @gorquan Good luck with your job hunting! I was able to get the .tar.gz out of the compilation but I don't know the instructions on how to generate the apk out from them. Do you know?

gorquan commented 5 years ago

Please wait.I uploaded my apk packaging container to github, you can try it.

gorquan commented 5 years ago

Hi@marianopeck This is my packaging environment container, you can try it.If you have any questions, please submit an issue.Rpi-alpine-glibc-package

SaturnusDJ commented 5 years ago

@marianopeck Ah you are using glibc-dev? Then it may be true that you will never reach small size with this method. But you can still try to delete things, it is quite likely your application does not require everything from the lib folder. A bit controversial of course, but no other choice at this moment.

The error just before the container finishes, can indeed be fixed at suggested in the topic. Connect to the running container and execute:

mkdir -p /usr/glibc-compat/etc
touch /usr/glibc-compat/etc/ld.so.conf
echo '/usr/local/lib' >> /usr/glibc-compat/etc/ld.so.conf
echo '/opt/lib' >> /usr/glibc-compat/etc/ld.so.conf

My execl problem is not fixed by this.

SaturnusDJ commented 5 years ago

...I just tested my application with user config in a 'real' environment and actually, while the logs are okish, the output is not. Simply said: it is not working, even with the full compiled 40MB .tar.gz. So, for me it also ends here. I have no idea how to get it working on ARM.

marianopeck commented 5 years ago

Hi @SaturnusDJ @gorquan Sorry, but for my it also ends up here. I have already spent quite some time on it. I was able to build a docker image with Alpine + glibc. However, my application (VA Smalltalk VM) would not link to glibc but to musl, no matter what I did and so my app would crash. In addition, the docker image of Alpine+glibc+myapp was 60.5MB while debian buster slim + myapp was 87.7MB. That is 27MB of different. But it's worth every byte if I can get debian + glibc.. So, no worth to me continue trying with Alpine. Here it is my last dockerfile attempt.. Thank you all of you anyway.

SaturnusDJ commented 5 years ago

In my case I found that a newer version of the application I am trying to run, even runs without glibc. Maybe only libc6 and libstdc++6 are required, and....bash! Installing bash fixed an error for me. This is on ARM. Tested it successfully.

x86_64 likely will continue to have errors etc.

SaturnusDJ commented 5 years ago

@marianopeck This issue became such a giant mess that I am not sure anymore what we tried and how everything works. As said above, I also did not need it anymore because all of a sudden the application started working with bash installed. But a few days ago I again needed glibc stuff to work in Docker on ARM......

I gave it another try and found: https://github.com/armhf-docker-library/alpine-pkg-glibc/releases It did what nothing else did before. Two applications that did not want to run at all....are running now. No idea yet about the performance, limitations, etc., but give them a try if you have not yet.

FROM alpine:3.10
RUN wget https://github.com/armhf-docker-library/alpine-pkg-glibc/releases/download/2.22/glibc-2.22-r8.apk https://github.com/armhf-docker-library/alpine-pkg-glibc/releases/download/2.22/glibc-bin-2.22-r8.apk https://github.com/armhf-docker-library/alpine-pkg-glibc/releases/download/2.22/glibc-i18n-2.22-r8.apk && \
apk add --no-cache --allow-untrusted glibc-2.22-r8.apk glibc-bin-2.22-r8.apk glibc-i18n-2.22-r8.apk && \
rm -rf glibc*
ENTRYPOINT ["tail", "-f", "/dev/null"]

Test image: 16,4MB here.

The version, 2.22, is outdated though. I really cannot understand why there is no active development in this. Not sure how that Drone stuff works in the repo linked. Hope to check it out later and hopefully dev into a more recent version of glibc.

marianopeck commented 5 years ago

Are you sure that's real GNU libc and not musl?

On Wed, Sep 11, 2019, 16:29 SaturnusDJ notifications@github.com wrote:

@marianopeck https://github.com/marianopeck This issue became such a giant mess that I am not sure anymore what we tried and how everything works. As said above, I also did not need it anymore because all of a sudden the application started working with bash installed. But a few days ago I again needed glibc stuff to work in Docker......

I gave it another try and found: https://github.com/armhf-docker-library/alpine-pkg-glibc/releases It did what nothing else did before. Two applications that did not want to run at all....are running now. No idea yet about the performance, limitations, etc., but give them a try if you have not yet.

FROM alpine:3.10 RUN wget https://github.com/armhf-docker-library/alpine-pkg-glibc/releases/download/2.22/glibc-2.22-r8.apk https://github.com/armhf-docker-library/alpine-pkg-glibc/releases/download/2.22/glibc-bin-2.22-r8.apk https://github.com/armhf-docker-library/alpine-pkg-glibc/releases/download/2.22/glibc-i18n-2.22-r8.apk && \ apk add --no-cache --allow-untrusted glibc-2.22-r8.apk glibc-bin-2.22-r8.apk glibc-i18n-2.22-r8.apk && \ rm -rf glibc* ENTRYPOINT ["tail", "-f", "/dev/null"]

Test image: 16,4MB here.

The version, 2.22, is outdated though. I really cannot understand why there is no active development in this. Not sure how that Drone stuff works in the repo linked. Hope to check it out later and hopefully dev into a more recent version of glibc.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sgerrand/alpine-pkg-glibc/issues/113?email_source=notifications&email_token=AAH4FAVHNAGQGD5SSEJIRLLQJFBIHA5CNFSM4G36NEFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6PT7WA#issuecomment-530530264, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH4FARMH5ORHIQT4WXCXGTQJFBIHANCNFSM4G36NEFA .

SaturnusDJ commented 5 years ago

Hmm not sure. How to find out? When installing the first five of https://pkgs.alpinelinux.org/packages?name=*musl*&branch=edge&arch=armhf, the application is not working.

"pkgdesc = GNU C Library compatibility layer"

SaturnusDJ commented 5 years ago

@marianopeck Have you tried it?

Rjerk commented 4 years ago

I build for arm64 version for my own usage:

https://github.com/Rjerk/alpine-pkg-glibc/releases/tag/2.30-r0-arm64

jvasileff commented 4 years ago

For armhf builds, see the forks: