search

dslm4515 / CMLFS

Clang-Built Musl Linux From Scratch
MIT License
105 stars 18 forks source link

Could we also have Heirloom NG as an alternative to GNU Coreutils in the future? #80

Open takusuman opened 1 year ago

takusuman commented 1 year ago

I was taking a look at https://github.com/dslm4515/CMLFS/tree/bsd-userland and thought that it would be interesting to also have Heirloom NG as an alternative to GNU Coreutils. These tools were derived from UNIX v7, 4BSD and OpenSolaris, while the majority were rewritten from scratch based on the payloads defined for original UNIX tools. I've been maintaining them in the last two years, and also added a new chroot implementation based on the Solaris 2.5 one. It's the main toolset at Copacabana Linux, so it was also patched to build on musl without any problems. I know we have many problems to solve before, but I just wanted to show it as an alternative.

dslm4515 commented 1 year ago

Yeah, I was dropping coreutils in favor for bsdutils. I'll check out Heirloom NG as well.

I just need a small set of tools to get a chroot running.

Chimera Linux is VERY interesting. Their patches for LLVM is geared towards musl+libc++, unlike Alpine & Void!

takusuman commented 1 year ago

I just need a small set of tools to get a chroot running.

Heirloom NG can do the job well, I hope it can help as an alternative. 😄

dslm4515 commented 1 year ago

Looks like Chimera Linux dropped bsdutils and developed chimerautils.

Chimerautils requires:

With optional deps:

Looks like Heirloom NG has less dependencies ... and chimerautils lacks a chroot binary. It provides a chroot. I just didnt see it in sbin

dslm4515 commented 1 year ago

Having trouble building heirloom-ng for llvmtools:

clang -z muldefs cpio.o unshrink.o explode.o expand.o inflate.o crc32.o blast.o flags.o nonpax.o version.o -Wl,-Bstatic -lz -Wl,-Bstatic -lbz2 -L../lib
common -lcommon   -o cpio                                                                                                                              
ld.lld: error: relocation R_X86_64_32S cannot be used against local symbol; recompile with -fPIC                                                       
>>> defined in /llvmtools/lib/libz.a(deflate.o)                                                                                                        
>>> referenced by deflate.c                                                                                                                            
>>>               deflate.o:(deflateReset) in archive /llvmtools/lib/libz.a

This is the mk.config I used:

SHELL = /llvmtools/bin/sh
POSIX_SHELL = /llvmtools/bin/sh
ROOT ?=
DEFBIN = /llvmtools/bin
SV3BIN = /llvmtools/bin
S42BIN = /llvmtools/bin
SUSBIN = /llvmtools/bin
UCBBIN = //llvmtools/ucbbin
CCSBIN = /llvmtools/ccs/bin
DEFLIB = /llvmtools/lib
DEFSBIN = /llvmtools/bin
MANDIR = /llvmtools/share/man/5man
DFLDIR = /llvmtools/etc/default
SPELLHIST = /dev/null
SULOG = /llvmtools/var/log/sulog
MAGIC = $(DEFLIB)/magic
TTYGRP = -g utmp
LIBPATH = -L/llvmtools/lib -L/llvmtools/ccs/lib
LCURS = -lcurses
LIBZ = -Wl,-Bstatic -lz
USE_ZLIB = 1
LIBBZ2 = -Wl,-Bstatic -lbz2
USE_BZLIB = 1
CC ?= clang
HOSTCC = clang
WARN=
LD = clang
CPPFLAGS = -D_GNU_SOURCE
CFLAGS = -O -fomit-frame-pointer $(WARN)
CFLAGS2 = -O2 -fomit-frame-pointer $(WARN)
CFLAGSS = -Os -fomit-frame-pointer $(WARN)
CFLAGSU = -O2 -fomit-frame-pointer -funroll-loops $(WARN)
STRIP = llvm-strip -s -R .comment -R .note
LARGEF = -D_FILE_OFFSET_BITS=64L
LNS = ln -s
YACC = yacc
LEX = flex
LCRYPT=-lcrypt
RANLIB=(hash ranlib) >/dev/null 2>&1 || exit 0; ranlib
UCBINST = $(ROOT)$(UCBBIN)/install

ICOMMON = -I../libcommon
LCOMMON = -L../libcommon -lcommon

IUXRE = -I../libuxre -DUXRE
LUXRE = -L../libuxre -luxre

MANINST = $(SHELL) ../build/maninst
dslm4515 commented 1 year ago

If it matters, i rebuilt bzip2 with CFLAGS="-fPIC' ... no change.

takusuman commented 1 year ago

If it matters, i rebuilt bzip2 with CFLAGS="-fPIC' ... no change.

In any case, you can disable bzip2 with USE_BZLIB=0, or make them build dynamically by commenting the current definition and enabling the other (as the file instructs).

takusuman commented 1 year ago

Looks like Heirloom NG has less dependencies

In fact, way less. You would just need yacc, lex and curses to build it, just disable support for zlib and bzip2. ... And it also provides some utilities that, in GNU Coreutils, would be provided as individual packages, such as cpio and tar.

takusuman commented 1 year ago

If it matters, i rebuilt bzip2 with CFLAGS="-fPIC' ... no change.

In any case, you can disable bzip2 with USE_BZLIB=0, or make them build dynamically by commenting the current definition and enabling the other (as the file instructs).

It's strange to me seeing this error, since it can build fine on Copacabana with musl and GCC... Maybe libbz2 static files aren't installed? @dslm4515

dslm4515 commented 1 year ago

I am using zlib-ng instead of zlib

But still, even when i disable both, similar error:

make[1]: Entering directory '/mnt/cmlfs/sources/heirloom-ng-230120/cpio'                                                                      [70/1871]
clang -z muldefs cpio.o unshrink.o explode.o expand.o inflate.o crc32.o blast.o flags.o nonpax.o version.o -Wl,-static -lz -Wl,-static -lbz2 -L../libco
mmon -lcommon   -o cpio                                                                                                                                
ld.lld: error: relocation R_X86_64_32S cannot be used against local symbol; recompile with -fPIC                                                       
>>> defined in /llvmtools/lib/libc.a(sysconf.o)                                                                                                        
>>> referenced by sysconf.c                                                                                                                            
>>>               sysconf.o:(sysconf) in archive /llvmtools/lib/libc.a                                                                                 

ld.lld: error: relocation R_X86_64_32S cannot be used against local symbol; recompile with -fPIC                                                       
>>> defined in /llvmtools/lib/libc.a(sysconf.o)                                                                                                        
>>> referenced by sysconf.c                                                                                                                            
>>>               sysconf.o:(sysconf) in archive /llvmtools/lib/libc.a                                                                                 

ld.lld: error: relocation R_X86_64_64 cannot be used against local symbol; recompile with -fPIC                                                        
>>> defined in /llvmtools/lib/libc.a(sysconf.o)                                                                                                        
>>> referenced by sysconf.c                                                                                                                            
>>>               sysconf.o:(.rodata.sysconf+0x0) in archive /llvmtools/lib/libc.a                                                                     

ld.lld: error: relocation R_X86_64_64 cannot be used against local symbol; recompile with -fPIC                                                        
>>> defined in /llvmtools/lib/libc.a(sysconf.o)                                                                                                        
>>> referenced by sysconf.c
>>>               sysconf.o:(.rodata.sysconf+0x8) in archive /llvmtools/lib/libc.a

ld.lld: error: relocation R_X86_64_64 cannot be used against local symbol; recompile with -fPIC
>>> defined in /llvmtools/lib/libc.a(sysconf.o)
>>> referenced by sysconf.c
>>>               sysconf.o:(.rodata.sysconf+0x10) in archive /llvmtools/lib/libc.a

ld.lld: error: relocation R_X86_64_64 cannot be used against local symbol; recompile with -fPIC
>>> defined in /llvmtools/lib/libc.a(sysconf.o)
>>> referenced by sysconf.c
>>>               sysconf.o:(.rodata.sysconf+0x18) in archive /llvmtools/lib/libc.a

ld.lld: error: relocation R_X86_64_64 cannot be used against local symbol; recompile with -fPIC
>>> defined in /llvmtools/lib/libc.a(sysconf.o)
>>> referenced by sysconf.c
>>>               sysconf.o:(.rodata.sysconf+0x20) in archive /llvmtools/lib/libc.a

ld.lld: error: relocation R_X86_64_64 cannot be used against local symbol; recompile with -fPIC
>>> defined in /llvmtools/lib/libc.a(sysconf.o)
>>> referenced by sysconf.c
>>>               sysconf.o:(.rodata.sysconf+0x28) in archive /llvmtools/lib/libc.a

ld.lld: error: relocation R_X86_64_64 cannot be used against local symbol; recompile with -fPIC
>>> defined in /llvmtools/lib/libc.a(sysconf.o)
>>> referenced by sysconf.c
>>>               sysconf.o:(.rodata.sysconf+0x30) in archive /llvmtools/lib/libc.a

ld.lld: error: relocation R_X86_64_64 cannot be used against local symbol; recompile with -fPIC
>>> defined in /llvmtools/lib/libc.a(sysconf.o)
>>> referenced by sysconf.c
>>>               sysconf.o:(.rodata.sysconf+0x38) in archive /llvmtools/lib/libc.a

ld.lld: error: relocation R_X86_64_64 cannot be used against local symbol; recompile with -fPIC
>>> defined in /llvmtools/lib/libc.a(sysconf.o)
>>> referenced by sysconf.c
>>>               sysconf.o:(.rodata.sysconf+0x40) in archive /llvmtools/lib/libc.a

ld.lld: error: relocation R_X86_64_64 cannot be used against local symbol; recompile with -fPIC
>>> defined in /llvmtools/lib/libc.a(sysconf.o)
>>> referenced by sysconf.c
>>>               sysconf.o:(.rodata.sysconf+0x48) in archive /llvmtools/lib/libc.a

ld.lld: error: relocation R_X86_64_64 cannot be used against local symbol; recompile with -fPIC
>>> defined in /llvmtools/lib/libc.a(sysconf.o)
>>> referenced by sysconf.c
>>>               sysconf.o:(.rodata.sysconf+0x50) in archive /llvmtools/lib/libc.a

ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /llvmtools/lib/libc.a(__libc_start_main.o)
>>> referenced by __libc_start_main.c 
>>>               __libc_start_main.o:(__init_libc) in archive /llvmtools/lib/libc.a
ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /llvmtools/lib/libc.a(__libc_start_main.o)
>>> referenced by __libc_start_main.c 
>>>               __libc_start_main.o:(__init_libc) in archive /llvmtools/lib/libc.a

ld.lld: error: cannot preempt symbol: __init_array_start
>>> referenced by __libc_start_main.c 
>>>               __libc_start_main.o:(libc_start_init) in archive /llvmtools/lib/libc.a

ld.lld: error: cannot preempt symbol: __init_array_end
>>> referenced by __libc_start_main.c 
>>>               __libc_start_main.o:(libc_start_init) in archive /llvmtools/lib/libc.a

ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /llvmtools/lib/libc.a(__libc_start_main.o)
>>> referenced by __libc_start_main.c 
>>>               __libc_start_main.o:(__libc_start_main) in archive /llvmtools/lib/libc.a

ld.lld: error: cannot preempt symbol: __thread_list_lock
>>> defined in /llvmtools/lib/libc.a(__init_tls.o)
>>> referenced by __init_tls.c
>>>               __init_tls.o:(__init_tp) in archive /llvmtools/lib/libc.a 

ld.lld: error: cannot preempt symbol: __libc
>>> defined in /llvmtools/lib/libc.a(libc.o)
>>> referenced by __init_tls.c
>>>               __init_tls.o:(__init_tp) in archive /llvmtools/lib/libc.a 

ld.lld: error: too many errors emitted, stopping now (use --error-limit=0 to see all errors)
clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:60: cpio] Error 1
make[1]: Leaving directory '/mnt/cmlfs/sources/heirloom-ng-230120/cpio'
make: *** [makefile:23: all] Error 2
takusuman commented 1 year ago

I am using zlib-ng instead of zlib

That seems to be ok. I'm building it on Copacabana with sortix-zlib and on Ubuntu with the default zlib (dynamically) and both work. I don't think the zlib implementation is a problem.

But still, even when i disable both, similar error

Did you ran make mrproper after disabling them? It's needed to apply the new configuration at build/mk.config. In the worst case scenario, apparently (at least, for now), LLVM can't build Heirloom NG statically linked, so you will need to build it dynamically (just comment out LDFLAGS= -static).

dslm4515 commented 1 year ago

stage1 clang static compiling works. I can build a static binary of libtree:

$ CC=clang make LDFLAGS="-static" 
clang -O2 -std=c99 -Wall -Wextra -Wshadow -pedantic -D_FILE_OFFSET_BITS=64 -c libtree.c
libtree.c:309:40: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
static inline int host_is_little_endian() {
                                       ^
                                        void
1 warning generated.
clang -static -o libtree libtree.o
$
dslm4515 commented 1 year ago

Did you ran make mrproper

Yes. I'll comment out LDFLAGS= -static next

takusuman commented 1 year ago

Did you ran make mrproper after disabling them? It's needed to apply the new configuration at build/mk.config.

I think I need to cite it in the README some time soon...

In the worst case scenario, apparently (at least, for now), LLVM can't build Heirloom NG statically linked, so you will need to build it dynamically (just comment out LDFLAGS= -static).

stage1 clang static compiling works.

So maybe I can cross this out?

takusuman commented 1 year ago

Did you ran make mrproper

Yes.

So maybe I can cross this out?

Eh, maybe not...

dslm4515 commented 1 year ago

oops. my mk.config was missing LDFLAGS

cpio builds fine

takusuman commented 1 year ago

oops. my mk.config was missing LDFLAGS

So maybe this is/was the problem. Try building it again like it was originally: both libbz2 and zlib-ng statically linked plus LDFLAGS= -static. If we can get it working, I can put LLVM in Heirloom NG's compatibility list.

takusuman commented 1 year ago

oops. my mk.config was missing LDFLAGS

cpio builds fine

Dynamically? Great!

dslm4515 commented 1 year ago

I went back to the default: static.

So cpio builds fine. Next roadblock: nawk:

make[1]: Entering directory '/mnt/cmlfs/sources/heirloom-ng-230120/nawk'
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c awk.g.c
flex  -t awk.lx.l > awk.lx.c
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c awk.lx.c
awk.lx.l:277:9: warning: using the result of an assignment as a condition without parentheses [-Wparentheses]
                if (c = *lexprog & 0377)
                    ~~^~~~~~~~~~~~~~~~~
awk.lx.l:277:9: note: place parentheses around the assignment to silence this warning
                if (c = *lexprog & 0377)
                      ^
                    (                  )
awk.lx.l:277:9: note: use '==' to turn this assignment into an equality comparison
                if (c = *lexprog & 0377)
                      ^
                      ==
1 warning generated.
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c b.c
In file included from b.c:19:
./y.tab.h:105:2: error: unknown type name 'intptr_t'
        intptr_t        i;
        ^
1 error generated.
make[1]: *** [Makefile:109: b.o] Error 1
make[1]: Leaving directory '/mnt/cmlfs/sources/heirloom-ng-230120/nawk'
make: *** [makefile:23: all] Error 2
dslm4515 commented 1 year ago

btw, bmake doesnt work:

$ bmake
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 4: Unassociated shell command "banner basename bc bdiff bfs \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 5: Unassociated shell command "cal calendar cat chmod chown chroot \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 6: Unassociated shell command "cksum cmp col comm copy cp cpio csplit cut \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 7: Unassociated shell command "date dc dd deroff diff diff3 dircmp dirname df du \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 8: Unassociated shell command "echo ed env expand expr \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 9: Unassociated shell command "factor file find fmt fmtmsg fold \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 10: Unassociated shell command "getconf getopt grep groups hd head hostname id join \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 11: Unassociated shell command "kill line listusers ln logins logname ls \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 12: Unassociated shell command "mail man mesg mkdir mkfifo mknod more mvdir \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 13: Unassociated shell command "nawk news nice nl nohup oawk od \
bmake: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 14: Unassociated shell command "paste pathchk pg pgrep pr printenv printf priocntl ps psri
"fo pwd \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 15: Unassociated shell command "random renice rm rmdir \
bmake: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 16: Unassociated shell command "sdiff sed setpgrp shl sleep sort spell split stty su sum s
"nc \
bmake: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 17: Unassociated shell command "tabs tail tapecntl tar tcopy tee test time touch tr true t
"ort tty \
bmake: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 18: Unassociated shell command "ul uname uniq units users wc what who whoami whodo xargs yes"
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 81: Unassociated shell command "CFLAGS="-Os -fomit-frame-pointer" \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 82: Unassociated shell command "CFLAGSS="-Os -fomit-frame-pointer" \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 83: Unassociated shell command "CFLAGS2="-Os -fomit-frame-pointer" \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 84: Unassociated shell command "CFLAGSU="-Os -fomit-frame-pointer" \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 85: Unassociated shell command "LCRYPT= \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 86: Unassociated shell command "IWCHAR=-I../libwchar LWCHAR="-L../libwchar -lwchar" \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 87: Unassociated shell command "DEFBIN=/5bin SV3BIN=/5bin S42BIN=/5bin/s42 \
bmake: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 88: Unassociated shell command "SUSBIN=/5bin/posix SU3BIN=/5bin/posix2001 UCBBIN=/5bin/ucb
"\
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 89: Unassociated shell command "CCSBIN=/5bin/ccs \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 90: Unassociated shell command "DEFLIB=/5bin/lib DEFSBIN=/5bin MANDIR=/tmp/__man__ \
"make: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 91: Unassociated shell command "DFLDIR=/etc/default SPELLHIST=/var/adm/spellhist \
bmake: "/mnt/cmlfs/sources/heirloom-ng-230120/makefile" line 92: Unassociated shell command "SULOG=/var/log/sulog MAGIC=/5bin/lib/magic"
bmake: Fatal errors encountered -- cannot continue
bmake: stopped in /mnt/cmlfs/sources/heirloom-ng-230120
$
takusuman commented 1 year ago

I went back to the default: static.

So cpio builds fine. Next roadblock: nawk:

make[1]: Entering directory '/mnt/cmlfs/sources/heirloom-ng-230120/nawk'
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c awk.g.c
flex  -t awk.lx.l > awk.lx.c
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c awk.lx.c
awk.lx.l:277:9: warning: using the result of an assignment as a condition without parentheses [-Wparentheses]
                if (c = *lexprog & 0377)
                    ~~^~~~~~~~~~~~~~~~~
awk.lx.l:277:9: note: place parentheses around the assignment to silence this warning
                if (c = *lexprog & 0377)
                      ^
                    (                  )
awk.lx.l:277:9: note: use '==' to turn this assignment into an equality comparison
                if (c = *lexprog & 0377)
                      ^
                      ==
1 warning generated.
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c b.c
In file included from b.c:19:
./y.tab.h:105:2: error: unknown type name 'intptr_t'
        intptr_t        i;
        ^
1 error generated.
make[1]: *** [Makefile:109: b.o] Error 1
make[1]: Leaving directory '/mnt/cmlfs/sources/heirloom-ng-230120/nawk'
make: *** [makefile:23: all] Error 2

Which version of Heirloom NG are you using? I think this was fixed back in this commit: https://github.com/Projeto-Pindorama/heirloom-ng/commit/63793000d599c8902b1e39ca02229e71f93cd9cf

dslm4515 commented 1 year ago

i used:

https://github.com/Projeto-Pindorama/heirloom-ng/archive/refs/tags/230120.tar.gz
takusuman commented 1 year ago

btw, bmake doesnt work:

I think this isn't exactly a bug, but a question of standards. Heirloom build system was written almost 21 years ago using the "pure" POSIX make syntax, and, if I'm not mistaken, the POSIX syntax isn't compatible with BSD's (and vice-versa).

takusuman commented 1 year ago

i used:

https://github.com/Projeto-Pindorama/heirloom-ng/archive/refs/tags/230120.tar.gz

It's the most recent, you're right. Let me see from where the y.tab.h file is generated and then add the #include <stdint.h> directive to it.

dslm4515 commented 1 year ago

I think this isn't exactly a bug, but a question of standards.

I was hoping to replace GNU's make with bmake to avoid building texinfo for llvmtools

takusuman commented 1 year ago

For now, a quick-n-dirty fix would be to add #include <stdint.h> to y.tab.h (without running make clean or make mrproper) and then running make again.

takusuman commented 1 year ago

I think this isn't exactly a bug, but a question of standards.

I was hoping to replace GNU's make with bmake to avoid building texinfo for llvmtools

Maybe you could use a POSIX make implementation, there's some out there. I don't think porting an entire build system would be possible right now.

dslm4515 commented 1 year ago

Maybe you could use a POSIX make implementation, there's some out there. I don't think porting an entire build system would be possible right now.

Yeah. I don't want to use chimerautils...it requires musl-compat-headers, musl-fts, musl-rpmatch, libxo, and meson [which need python... not sure if i should just make python host requirement OR add to cgnutools]

takusuman commented 1 year ago

By the way, isn't GNU's make needed for building the Linux kernel? I think it's a monster that all of us using Linux will need to have installed in our systems at some point.

takusuman commented 1 year ago

Maybe you could use a POSIX make implementation, there's some out there. I don't think porting an entire build system would be possible right now.

Yeah. I don't want to use chimerautils...it requires musl-compat-headers, musl-fts, musl-rpmatch, libxo, and meson [which need python... not sure if i should just make python host requirement OR add to cgnutools]

Well, if it helps, there's this make implementation: https://frippery.org/make/ In theory it can build Heirloom NG just fine.

dslm4515 commented 1 year ago

By the way, isn't GNU's make needed for building the Linux kernel?

No wonder make mrproper was so familiar!

I guess i can't ditch GNU make then. Which reminds me... some individuals want to port the linux kernel to ...RUST! Rust is VERY tricky to build...once i just used a prebuilt that the rust developers released for musl...

takusuman commented 1 year ago

No wonder make mrproper was so familiar!

Well, the only thing in common between Heirloom's build system (talking both about original Heirloom, Ryan Woodsmall's fork and Heirloom NG) and Linux's Kbuild is mrproper itself --- probably this was a personal choice by Ritter when he was writing Heirloom, honestly I don't know --- besides that, Heirloom's build system don't depend on GNU make, only on a make implementation that follows the POSIX standard.

takusuman commented 1 year ago

I guess i can't ditch GNU make then. Which reminds me... some individuals want to port the linux kernel to ...RUST! Rust is VERY tricky to build...once i just used a prebuilt that the rust developers released for musl...

Well, that will be kind of fun to build...

takusuman commented 1 year ago

I went back to the default: static.

So cpio builds fine. Next roadblock: nawk:

make[1]: Entering directory '/mnt/cmlfs/sources/heirloom-ng-230120/nawk'
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c awk.g.c
flex  -t awk.lx.l > awk.lx.c
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c awk.lx.c
awk.lx.l:277:9: warning: using the result of an assignment as a condition without parentheses [-Wparentheses]
                if (c = *lexprog & 0377)
                    ~~^~~~~~~~~~~~~~~~~
awk.lx.l:277:9: note: place parentheses around the assignment to silence this warning
                if (c = *lexprog & 0377)
                      ^
                    (                  )
awk.lx.l:277:9: note: use '==' to turn this assignment into an equality comparison
                if (c = *lexprog & 0377)
                      ^
                      ==
1 warning generated.
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c b.c
In file included from b.c:19:
./y.tab.h:105:2: error: unknown type name 'intptr_t'
        intptr_t        i;
        ^
1 error generated.
make[1]: *** [Makefile:109: b.o] Error 1
make[1]: Leaving directory '/mnt/cmlfs/sources/heirloom-ng-230120/nawk'
make: *** [makefile:23: all] Error 2

Oh, you could also try adding -std=c99 to the Makefile.mk at nawk/ directory, more specifically at this line which caused the error:

b.o: b.c
        $(CC) $(CFLAGSS) $(CPPFLAGS) $(XO5FL) $(LARGEF) $(IWCHAR) $(ICOMMON) $(IUXRE) -c b.c -std=c99
takusuman commented 1 year ago

About y.tab.h, the most interesting part is that it is generated in a one-liner inside the Makefile itself using ed!

awk.g.c: awk.g.y
        $(YACC) -d awk.g.y
        mv -f y.tab.c awk.g.c
        (echo '1i'; echo '#include <inttypes.h>'; echo '.'; echo 'w';) | \
                ed -s y.tab.h
dslm4515 commented 1 year ago

Just to make sure i didnt forget any variables, I started over and this time copied mk.config and removed most of the comments for easier editing. Then i edited nawk/Makefile.mk to add -std=c99 :

make[1]: Entering directory '/mnt/cmlfs/sources/heirloom-ng-230120/nawk'
yacc -d awk.g.y
yacc: 46 shift/reduce conflicts, 178 reduce/reduce conflicts.
mv -f y.tab.c awk.g.c
(echo '1i'; echo '#include <inttypes.h>'; echo '.'; echo 'w';) | \
        ed -s y.tab.h
/llvmtools/bin/sh: 2: ed: not found
make[1]: *** [Makefile:114: awk.g.c] Error 127
make[1]: Leaving directory '/mnt/cmlfs/sources/heirloom-ng-230120/nawk'
make: *** [makefile:23: all] Error 2

Does this mean i need ed installed? I don't have it on my host or in llvmtools.

About y.tab.h...

This might be why the generated y.tab.h is missing that header line

takusuman commented 1 year ago

Does this mean i need ed installed? I don't have it on my host or in llvmtools.

ed is part of the POSIX standard, in theory every UNIX-compatible (Linux, *BSD etc) need to have it installed. oed (OpenEd, from OpenBSD) do the job well; I haven't needed it in Copacabana, through, since I wasn't using Heirloom's Nawk. It was used to do more sophisticated changes on files, since POSIX sed doesn't (didn't?) support inserting lines by defaut.

By the way, you would only need it in the host, since Heirloom will provide an ed implementation after being installed.

dslm4515 commented 1 year ago

Well, if it helps, there's this make implementation: https://frippery.org/make/

It wont build dash shell [ POSIX shell implementation i chose] :

 config.status: executing depfiles commands
cmlfs:/mnt/cmlfs/sources/dash-0.5.12
$ pdpmake 
pdpmake  all-recursive
Making all in src
  CC       builtins.def
clang-15: error: no input files
pdpmake: failed to build 'builtins.def'
pdpmake: nothing to be done for all-am

But GNU's make builds dash fine... perhaps, i'll try another POSIX make implementation or patch dash. pdpmake was able to build flex and byacc fine

dslm4515 commented 1 year ago

By the way, you would only need it in the host, since Heirloom will provide an ed implementation after being installed.

I'll just build ed for cgnutools in case a host does not have ed

eli-schwartz commented 1 year ago

Yeah. I don't want to use chimerautils...it requires musl-compat-headers, musl-fts, musl-rpmatch, libxo, and meson [which need python... not sure if i should just make python host requirement OR add to cgnutools]

It builds okay with https://sr.ht/~lattis/muon/ , the c99 reimplementation of Meson. No need for python, at least.

takusuman commented 1 year ago

Yeah. I don't want to use chimerautils...it requires musl-compat-headers, musl-fts, musl-rpmatch, libxo, and meson [which need python... not sure if i should just make python host requirement OR add to cgnutools]

It builds okay with https://sr.ht/~lattis/muon/ , the c99 reimplementation of Meson. No need for python, at least.

Interesting, I'll take a look at this too.

dslm4515 commented 1 year ago

No need for python, at least.

YES. muon is good...as in ... Muon for meson, samurai for ninja!

takusuman commented 1 year ago

By the way, you would only need it in the host, since Heirloom will provide an ed implementation after being installed.

I'll just build ed for cgnutools in case a host does not have ed

Were you able to build it in the end? If yes, which are the versions of the musl libc, Linux kernel and LLVM?

dslm4515 commented 1 year ago

I'm currently testing chimerautils... definitely will not be the default "coreutils" for llvmtools as it lacks chroot... I rather not expect the host to provide a chroot binary.

When I test heirloom-ng, i'll post the versions

dslm4515 commented 1 year ago

And I hit a snag with chimerautils: no install binary.

When compiling bmake under chroot, it uses install to install:

bmake: exec(install) failed (No such file or directory)
*** Error code 1

@takusuman, hopefully heirloom-ng has install ?

eli-schwartz commented 1 year ago

chimerautils does have install, not sure what you mean?

dslm4515 commented 1 year ago

I configured chimerautils with this meson invocation:

CC=clang CXX=clang++ \
AR=llvm-ar \
AS=llvm-as \
RANLIB=llvm-ranlib \
LD=ld.lld \
STRIP=llvm-strip \
LDFLAGS="-lterminfo " \
meson --prefix=/llvmtools \
      --pkg-config-path /llvmtools/lib/pkgconfig \
      -Dstatic_fts=true \
      -Dstatic_rpmatch=true \
      -Dcolor_ls=true \
      -Dlibcrypto=disabled \
      -Dlibedit=disabled BUILD

Which gave this output:

The Meson build system
Version: 0.63.1
Source dir: /mnt/cmlfs/sources/chimerautils-f5e7bc7ba541b46ff6ff8fe73b3b27a599e898b1
Build dir: /mnt/cmlfs/sources/chimerautils-f5e7bc7ba541b46ff6ff8fe73b3b27a599e898b1/BUILD
Build type: native build
Project name: chimerautils
Project version: 1.0
C compiler for the host machine: clang (clang 3.0.0 "cgnutools 3.0.0 clang version 15.0.6")
C linker for the host machine: clang ld.lld 3.0.0
C++ compiler for the host machine: clang++ (clang 3.0.0 "cgnutools 3.0.0 clang version 15.0.6")
C++ linker for the host machine: clang++ ld.lld 3.0.0
Host machine cpu family: x86_64
Host machine cpu: x86_64
Compiler for C supports arguments -Wstring-plus-int: YES 
Program flex found: YES (/llvmtools/bin/flex)
Program byacc found: YES (/llvmtools/bin/byacc)
Checking for function "strtoq" : NO 
Checking for function "strlcpy" : YES 
Checking for function "strlcat" : YES 
Checking for function "strtonum" : NO 
Library tinfo found: YES
Dependency libedit skipped: feature libedit disabled
Dependency libcrypto skipped: feature libcrypto disabled
Library m found: YES
Found pkg-config: /bin/pkg-config (1.9.3)
Run-time dependency libxo found: YES 1.6.0
Run-time dependency zlib found: YES 1.2.11.zlib-ng
Library bz2 found: YES
Run-time dependency liblzma found: YES 5.4.0
Checking for function "fts_open" : NO 
Library fts found: YES
Checking for function "rpmatch" : NO 
Library rpmatch found: YES
Run-time dependency threads found: YES
Compiler for C supports arguments -Wclobbered: NO
Program flex found: YES (/llvmtools/bin/flex)
Program byacc found: YES (/llvmtools/bin/byacc)
Checking for function "strtoq" : NO 
Checking for function "strlcpy" : YES 
Checking for function "strlcat" : YES 
Checking for function "strtonum" : NO 
Library tinfo found: YES
Dependency libedit skipped: feature libedit disabled
Dependency libcrypto skipped: feature libcrypto disabled
Library m found: YES
Found pkg-config: /bin/pkg-config (1.9.3)
Run-time dependency libxo found: YES 1.6.0
Run-time dependency zlib found: YES 1.2.11.zlib-ng
Library bz2 found: YES
Run-time dependency liblzma found: YES 5.4.0
Checking for function "fts_open" : NO 
Library fts found: YES
Checking for function "rpmatch" : NO 
Library rpmatch found: YES
Run-time dependency threads found: YES
Compiler for C supports arguments -Wclobbered: NO 
Configuring config.h using configuration
Build targets in project: 115

chimerautils 1.0

  User defined options
    pkg_config_path: /llvmtools/lib/pkgconfig
    prefix         : /llvmtools
    color_ls       : true
    libcrypto      : disabled
    libedit        : disabled
    static_fts     : true
    static_rpmatch : true

Found ninja-1.9.0 at /llvmtools/bin/ninja

Then i installed it to my toolchain in /llvmtools. Only these binaries were installed:

'['         cat      csplit    echo     fmt      head       look      mkfifo   pathchk    rgrep   stat      true       vdir     xzegrep   zgrep
 base32     chgrp    cut       ed       fold     hexdump    ls        mktemp   pr         rm      stdbuf    truncate   wall     xzfgrep   zless
 base64     chmod    date      egrep    getopt   hostid     lzcmp     mv       printenv   rmdir   stty      tsort      wc       xzgrep    zmore
 basename   cksum    dd        env      grep     hostname   lzdiff    nice     printf     sdiff   sum       tty        which    yes       znew
 basenc     cmp      df        expand   groups   id         lzegrep   nl       pwd        sed     tail      ul         who      zcat      zstdegrep
 bzcmp      col      diff      expr     gunzip   join       lzfgrep   nohup    readlink   seq     tee       uname      whoami   zcmp      zstdfgrep
 bzdiff     colrm    diff3     factor   gzcat    kill       lzgrep    nproc    realpath   sh      test      unexpand   write    zdiff     zstdgrep
 bzegrep    column   dir       false    gzexe    link       m4        od       red        sleep   timeout   uniq       xargs    zegrep
 bzfgrep    comm     dirname   fgrep    gzip     ln         mesg      paste    renice     sort    touch     unlink     xzcmp    zfgrep
 bzgrep     cp       du        find     hd       logname    mkdir     patch    rev        split   tr        users      xzdiff   zforce
eli-schwartz commented 1 year ago

-Dlibcrypto=disabled

There's your problem, chimerautils is based on FreeBSD and includes the -h <hash> -M <metalog> options for the install command -- this needs libcrypto, which you have chosen to disable.

takusuman commented 1 year ago

And I hit a snag with chimerautils: no install binary.

When compiling bmake under chroot, it uses install to install:

bmake: exec(install) failed (No such file or directory)
*** Error code 1

@takusuman, hopefully heirloom-ng has install ?

G'day, Derrick. Sorry for the delay. I'm employed now and I was without time for checking my e-mails until now. Yes, it have install. If you want a custom destination directory, use ROOT (eg.: ROOT=/tmp/heirloom gmake install)

dslm4515 commented 1 year ago

Damn. I accidentally executed rm -rf /* instead of rm -rf ./* [while in chroot] and destroyed my CMLFS build.

I guess its good time to test heirloom-ng as the coreutils for chroot.

dslm4515 commented 1 year ago

Rebuilt cgnutools and llvmtools far enough to reach heirloom-ng. I made sure to copy the build/mk.config and just removed the comments for easy editing. Build still fails at nawk:

make[1]: Entering directory '/mnt/cmlfs/sources/heirloom-ng-230120/nawk'
byacc -d awk.g.y
byacc: 46 shift/reduce conflicts, 178 reduce/reduce conflicts.
mv -f y.tab.c awk.g.c
(echo '1i'; echo '#include <inttypes.h>'; echo '.'; echo 'w';) | \
    ed -s y.tab.h
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c awk.g.c
flex  -t awk.lx.l > awk.lx.c
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c awk.lx.c
awk.lx.l:277:9: warning: using the result of an assignment as a condition without parentheses [-Wparentheses]
                if (c = *lexprog & 0377)
                    ~~^~~~~~~~~~~~~~~~~
awk.lx.l:277:9: note: place parentheses around the assignment to silence this warning
                if (c = *lexprog & 0377)
                      ^
                    (                  )
awk.lx.l:277:9: note: use '==' to turn this assignment into an equality comparison
                if (c = *lexprog & 0377)
                      ^
                      ==
1 warning generated.
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c b.c -std=c99
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c lib.c
cc -Os -fomit-frame-pointer  -D_GNU_SOURCE -DUSE_TERMCAP  -D_FILE_OFFSET_BITS=64L  -I../libcommon -I../libuxre -DUXRE -c main.c
main.c:56:5: error: second parameter of 'main' (argument array) must be of type 'char **'
int main(int argc, unsigned char *argv[], unsigned char *envp[])
    ^
main.c:56:5: error: third parameter of 'main' (environment) must be of type 'char **'
2 errors generated.
make[1]: *** [Makefile:136: main.o] Error 1
make[1]: Leaving directory '/mnt/cmlfs/sources/heirloom-ng-230120/nawk'
make: *** [makefile:23: all] Error 2

I'm not worried about the warning, but there is a problem with nawk/main.c ?

My build/mk.config:

SHELL = /llvmtools/bin/sh
POSIX_SHELL = /llvmtools/bin/sh
ROOT ?=
DEFBIN = /llvmtools/bin
SV3BIN = /llvmtools/bin
S42BIN = /llvmtools/bin
SUSBIN = /llvmtools/bin
SU3BIN = /llvmtools/sbin
UCBBIN = /llvmtools/ucbbin
CCSBIN = /llvmtools/ccs/bin
DEFLIB = /llvmtools/lib
DEFSBIN = /llvmtools/bin
MANDIR = /llvmtools/share/man/5man
DFLDIR = /llvmtools/etc/default
SPELLHIST = /dev/null
SULOG = /llvmtools/var/log/sulog
MAGIC = $(DEFLIB)/magic
TTYGRP = -g utmp
LIBPATH = -L/llvmtools/lib -L/llvmtools/ccs/lib
# Curses library. Change to -lncurses if necessary. Caution: Some gcc
# setups on Solaris are broken so that ncurses headers are used during
# compilation, but the system libcurses is used for linking. This will
# usually lead to segmentation violations. The easiest fix is to write
# something like -L/path/to/ncurses/lib/directory -lncurses here.
#
# The 4.4 BSD curses library, which is still supplied with NetBSD 2.0,
# does not supply our needs. Use -ltermcap instead and add -DUSE_TERMCAP
# to CPPFLAGS.
#
LCURS = -ltermcap

LIBZ = -Wl,-Bstatic -lz
USE_ZLIB = 1
LIBBZ2 = -Wl,-Bstatic -lbz2
USE_BZLIB = 1

CC ?= clang
HOSTCC = clang
WARN=

LD = clang
LDFLAGS = -static $(LIBPATH)
CPPFLAGS = -D_GNU_SOURCE -DUSE_TERMCAP
CFLAGS = -O -fomit-frame-pointer $(WARN)
CFLAGS2 = -O2 -fomit-frame-pointer $(WARN)
CFLAGSS = -Os -fomit-frame-pointer $(WARN)
CFLAGSU = -O2 -fomit-frame-pointer -funroll-loops $(WARN)
STRIP = llvm-strip -s -R .comment -R .note
LARGEF = -D_FILE_OFFSET_BITS=64L

LNS = ln -s
YACC = byacc
LEX = flex

LCRYPT=-lcrypt
RANLIB=(hash llvm-ranlib) >/dev/null 2>&1 || exit 0; ranlib

########################################################################

UCBINST = $(ROOT)$(UCBBIN)/install

ICOMMON = -I../libcommon
LCOMMON = -L../libcommon -lcommon

IUXRE = -I../libuxre -DUXRE
LUXRE = -L../libuxre -luxre

MANINST = $(SHELL) ../build/maninst