Bootstrapping without a copy of asmc?

[voltagex]

Found this project via 7-Zip which seems to require it.

I then tried to compile asmc on Alpine, which doesn't have glibc.

It seems like you store a compiled copy of asmc64 in source control which is used during the build.

Is it possible to bootstrap asmc without this? Is it possible to compile entirely from source?

[nidud]

I then tried to compile asmc on Alpine, which doesn't have glibc.

It seems like you store a compiled copy of asmc64 in source control which is used during the build.

quote

There's a mixed bag of Linux distributions with a variety of dependencies. Some are also exclusively 64-bit so it's difficult to provide binaries that works for all.

In most cases the default setting using a dynamically linked 64-bit binary works. However, there is a statically linked 32-bit binary provided for incompatibility issues like this, so you may try to see if that works.

To test this change the line in the makefile acc = asmc64 to acc = asmc

This will still build a 64-bit version thought.

Is it possible to compile entirely from source?

Yes, and that's what you trying to do.

[voltagex]

Hang on.

I mean: can you compile asmc without already having a copy of asmc64 or asmc in ./bin?

It's weird to have an open source project that relies on binary blobs.

I even looked to the earliest released version of asmc and it looks like the initial commit contained a whole lot of Windows binaries - how were those built initially?

Think about something like Linux from Scratch where everything is built from source. To take it one step further, have a look at the https://bootstrappable.org/ although this is the extreme case.

I am more interested in things like the Debian build process - it's expected there would not be third party binaries involved in the package build process.

[nidud]

Hang on.

I mean: can you compile asmc without already having a copy of asmc64 or asmc in ./bin?

Asmc is assembled so you need a capable assembler to assemble the code.

It's weird to have an open source project that relies on binary blobs.

All open source relies on binary blobs.

I even looked to the earliest released version of asmc and it looks like the initial commit contained a whole lot of Windows binaries - how were those built initially?

Asmc is a fork of JWasm so it was originally written in C.

Think about something like Linux from Scratch where everything is built from source. To take it one step further, have a look at the https://bootstrappable.org/ although this is the extreme case.

It is the chicken-and-egg problem, yes.

I am more interested in things like the Debian build process - it's expected there would not be third party binaries involved in the package build process.

I will assume it's possible to build 7-zip without the assembler part, and there are also other assemblers written in C (like Uasm and Jwasm) that probably would work.

[teoberi]

Is it possible to bootstrap asmc without this?

No!

[teoberi]

I will assume it's possible to build 7-zip without the assembler part, and there are also other assemblers written in C (like Uasm and Jwasm) that probably would work.

UASM 2.57 can be used to build 7-Zip. You can test also with this: https://github.com/Terraspace/UASM/pull/216#issue-2564336684

[voltagex]

Thanks @teoberi. Do I need to worry about any of the warnings that are mentioned in that thread?

[teoberi]

The author built UASM 2.57 with GCC version 13.2.0. I built 7-Zip with that UASM and it works. I tried to build UASM with GCC version 14.2.0 but here all the warnings are now errors. That thread tries to solve the errors, but the solutions are not validated by the UASM maintainer, nor by other maintainers of the UASM package from other Linux distributions. Even if with those patches there are no error/warning messages, this does not mean that everything is OK, additional testing is still needed for validation. I will also build 7-Zip with the UASM containing those patches in the next few days to test it.

[teoberi]

However, it should be noted that the development of UASM is not going as well as what @nidud is doing with ASMC.

[voltagex]

Doesn't that then make the reproducibility of asmc more of an issue?

[teoberi]

I have built 7-Zip 24.08 using GCC version 14.2.0 + UASM with all the patches from here without any errors.

uasm -nologo -elf64 -DABI_LINUX -Fob/g_x64/ ../../../../Asm/x86/LzmaDecOpt.asm LzmaDecOpt.asm: 1339 lines, 3 passes, 6623 ms, 0 warnings, 0 errors

uasm -nologo -elf64 -DABI_LINUX -Fob/g_x64/ ../../../../Asm/x86/7zCrcOpt.asm 7zCrcOpt.asm: 258 lines, 3 passes, 2173 ms, 0 warnings, 0 errors

uasm -nologo -elf64 -DABI_LINUX -Fob/g_x64/ ../../../../Asm/x86/AesOpt.asm "++ VAES 256" "x86-64" "ABI : LINUX" AesOpt.asm: 742 lines, 2 passes, 4322 ms, 0 warnings, 0 errors

uasm -nologo -elf64 -DABI_LINUX -Fob/g_x64/ ../../../../Asm/x86/Sha1Opt.asm Sha1Opt.asm: 263 lines, 2 passes, 2357 ms, 0 warnings, 0 errors

uasm -nologo -elf64 -DABI_LINUX -Fob/g_x64/ ../../../../Asm/x86/Sha256Opt.asm Sha256Opt.asm: 275 lines, 2 passes, 2256 ms, 0 warnings, 0 errors

uasm -nologo -elf64 -DABI_LINUX -Fob/g_x64/ ../../../../Asm/x86/XzCrc64Opt.asm XzCrc64Opt.asm: 523 lines, 3 passes, 1920 ms, 0 warnings, 0 errors

uasm -nologo -elf64 -DABI_LINUX -Fob/g_x64/ ../../../../Asm/x86/LzFindOpt.asm LzFindOpt.asm: 540 lines, 3 passes, 2356 ms, 0 warnings, 0 errors

[nidud]

I then tried to compile asmc on Alpine, which doesn't have glibc.

I built a static version of asmc64 that should remove this dependency.

~/asmc/source/asmc$ file ./asmc64

./asmc64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=4ef1fc563adafd0712aba2469ba90c9b457f800b, stripped

It was built using libasmc.a and briefly tested in Ubuntu. It seems to work for building Asmc and the libraries so if you could test this on your end that will be good.

It wont fix the underlying problem with the binaries but at least fix something.

[nidud]

Is it possible to bootstrap asmc without this?

No!

There is a solution to this by using @AgnerF's objconv to produce GNU Assembler output from the object files.

objconv -fgasm asmc.o src\asmc.s

A few problems:

• There's a bug in objconv. It produce intel syntax but use cmp byte [rbx] should be cmp byte ptr [rbx]
• It doesn't handle jump tables very well so it ends up producing random code from the content.
• Asmc produce @C???? labels and GAS do not handle '@' very well.

The bug in objconv and the Asmc issue are fixable but the random code issue need some handling.

[voltagex]

I'm still working on this out of interest to see if it's possible.

At the moment I think I can build asmc 2.30 from c32537d using OpenWatcom 2 and then rebuild from there.

[voltagex]

I am currently stuck on

#17 0.301 Error! E2028: qerrno is an undefined reference
#17 0.301 Error! E2028: atoquad is an undefined reference
#17 0.301 Error! E2028: quadtold is an undefined reference
#17 0.301 Error! E2028: quadtoh is an undefined reference
#17 0.301 Error! E2028: quadtof is an undefined reference
#17 0.301 Error! E2028: quadtod is an undefined reference
#17 0.302 Error! E2028: quadadd is an undefined reference
#17 0.302 Error! E2028: quadsub is an undefined reference
#17 0.302 Error! E2028: quaddiv is an undefined reference
#17 0.302 Error! E2028: quadmul is an undefined reference
#17 0.303 file atofloat.o(atofloat): undefined symbol qerrno
#17 0.303 file atofloat.o(atofloat): undefined symbol atoquad
#17 0.303 file atofloat.o(atofloat): undefined symbol quadtold
#17 0.303 file atofloat.o(atofloat): undefined symbol quadtoh
#17 0.303 file atofloat.o(atofloat): undefined symbol quadtof
#17 0.303 file atofloat.o(atofloat): undefined symbol quadtod
#17 0.303 file atoquad.o(atoquad): undefined symbol qerrno
#17 0.303 file atoquad.o(atoquad): undefined symbol quadtod
#17 0.303 file atoquad.o(atoquad): undefined symbol quadtold
#17 0.303 file atoquad.o(atoquad): undefined symbol quadtof
#17 0.303 file atoquad.o(atoquad): undefined symbol quadtoh
#17 0.303 file expreval.o(expreval): undefined symbol quadadd
#17 0.303 file expreval.o(expreval): undefined symbol quadsub
#17 0.303 file expreval.o(expreval): undefined symbol quaddiv
#17 0.303 file expreval.o(expreval): undefined symbol quadmul
#17 0.303 file fastcall.o(fastcall): undefined symbol atoquad

while linking 2.30 using wlink, even after compiling atoquad.c

[nidud]

I've managed to produce a somewhat functional source for the GNU assembler. It builds and works for small single-file tests but there are a few issues to work out, so it currently fails to build Asmc.

However, this is produced from the default output which is a dynamically linked:

$ file asmc64
asmc64: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=7b1604a36f31ef6d66fd290653f08d4bfb24bbe9, for GNU/Linux 3.2.0, stripped

This should probably be static.

The idea is then to remove the Linux binaries and build it from source using the GNU assembler. This will be the minimum version needed to build the current version of Asmc.

[voltagex]

@nidud thanks, I really appreciate the hard work here. I'll halt my experiments with the Watcom toolchain in that case.

A dynamically linked binary from source wouldn't necessarily be an issue as it'd be linked against whatever the host or toolchain libc is.

[nidud]

A dynamically linked binary from source wouldn't necessarily be an issue as it'd be linked against whatever the host or toolchain libc is.

Okey, so I will just use the modified source then.

The Linux binaries are now removed and the makefile updated.