Closed andrew-m-leonard closed 2 years ago
Hi I've been working on this and followed https://github.com/adoptium/temurin-build/blob/master/docker/dockerfile-generator.sh for building.
I've seen the following output from my terminal, but there're no images output in the /termurin-build/workspace/target. May I ask if the above docker guide should produce such result? Thanks!
Your final archive was created at /openjdk/build/src/build/linux-x86_64-normal-server-release/images/OpenJDK.tar.gz
Moving the artifact to /openjdk//target/
build.sh : 18:03:49 : All done!
The finished image can be found in /<my local>/temurin-build/workspace/target on the host system
Removing container jdk8-hotspot
4a84f8f8141f
hi @JeromeJu - Andrew added this wiki page https://github.com/adoptium/temurin-build/wiki/Building-OpenJDK-using-temurin-build-scripts-within-the-adopt-build-docker-container to give guidance.
hi @JeromeJu the dockerfile-generator.sh script is used to produce docker images on dockerhub, not for building openjdk. As Shelley pointed you to, I wrote the above wiki guide.
@JeromeJu however i've just thought of something relevant to this, as we want to run strace, I have a feeling strace won't work within a docker container. @sxa is that correct?
@JeromeJu i've just tried using strace in a docker container and it does seem to work, eg:
[root@e6637d98787e /]# strace -ff -e trace=openat,execve ls
execve("/usr/bin/ls", ["ls"], 0xffffdcb1dc28 /* 17 vars */) = 0
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libacl.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/proc/filesystems", O_RDONLY) = 3
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/locale/en_US.UTF-8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/locale/en_US.utf8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/locale/en_US/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/locale/en.UTF-8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/locale/en.utf8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/locale/en/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
anaconda-post.log boot etc lib media opt root sbin strace.150 tmp var
bin dev home lib64 mnt proc run srv sys usr
+++ exited with 0 +++
[root@e6637d98787e /]#
Thanks for the wiki and the pointers @andrew-m-leonard ! Finaly got through the longest local build in my life :)
Following on the steps, I'd like to check my understanding regarding touch libnet source: touch src/java.base/share/native/libnet/*
shall be under workspace/temurin-build/workspace/build/src
? And then with strace we'd rebuild with the same workflow as the build-farm script?
Thanks!
hi @JeromeJu the dockerfile-generator.sh script is used to produce docker images on dockerhub, not for building openjdk. As Shelley pointed you to, I wrote the above wiki guide.
To be clear, the above statement is not correct - those dockerfiles are for producing a build environment suitable for building OpenJDK - you can see that from the prerequisites that it puts in :-)
strace
Yeah that's correct as you've confirmed. I think it's possible to work around it but I don't have the process to hand. I've only ever used it running on a native machine. If you can create a VM and set it up with the playbooks that would allow the use of strace
easily.
running it on my local machine: RHEL8 amd64
[wenzhou@wenzhou jdk18u]$ uname -a
Linux wenzhou.remote.csb 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
[wenzhou@wenzhou jdk18u]$ git log -1
commit 0e73b3911f20169607e8836d38f532fe51d577f9 (HEAD -> master, tag: jdk-18.0.2-ga, tag: jdk-18.0.2+9, origin/master, origin/jdk18.0.2, origin/HEAD)
Author: Ravi Reddy <rreddy@openjdk.org>
Date: Tue May 31 17:15:22 2022 +0000
8287175: Backout 8270480: Better path to expressing Xpaths
Reviewed-by: joehw
Backport-of: 84c8701632ac02657bc6c65257df19870b3fe67e
[wenzhou@wenzhou jdk18u]$ git remote -v
origin git@github.com:openjdk/jdk18u.git (fetch)
[wenzhou@wenzhou jdk18u]$ bash configure --with-source-date=version
[wenzhou@wenzhou jdk18u]$ make images
...done in about 10 mins....
[wenzhou@wenzhou linux-x86_64-server-release]$ ls
bootcycle-spec.gmk buildjdk-spec.gmk build.log buildtools compare.sh configure.log configure.log.old configure-support hotspot images jdk Makefile make-support spec.gmk support
[wenzhou@wenzhou jdk18u]$ ls -lat src/java.base/share/native/libnet
total 40
drwxrwxr-x. 2 wenzhou wenzhou 147 Jun 16 16:51 .
drwxrwxr-x. 12 wenzhou wenzhou 159 Jun 16 16:51 ..
-rw-rw-r--. 1 wenzhou wenzhou 3968 Jun 16 16:51 proxy_util.c
-rw-rw-r--. 1 wenzhou wenzhou 1584 Jun 16 16:51 proxy_util.h
-rw-rw-r--. 1 wenzhou wenzhou 1964 Jun 16 16:51 Inet4Address.c
-rw-rw-r--. 1 wenzhou wenzhou 2901 Jun 16 16:51 Inet6Address.c
-rw-rw-r--. 1 wenzhou wenzhou 2993 Jun 16 16:51 InetAddress.c
-rw-rw-r--. 1 wenzhou wenzhou 12280 Jun 16 16:51 net_util.c
-rw-rw-r--. 1 wenzhou wenzhou 6734 Jun 16 16:51 net_util.h
[wenzhou@wenzhou jdk18u]$ touch src/java.base/share/native/libnet/*
[wenzhou@wenzhou jdk18u]$ ls -lat src/java.base/share/native/libnet
total 40
-rw-rw-r--. 1 wenzhou wenzhou 1964 Aug 15 11:13 Inet4Address.c
-rw-rw-r--. 1 wenzhou wenzhou 2901 Aug 15 11:13 Inet6Address.c
-rw-rw-r--. 1 wenzhou wenzhou 2993 Aug 15 11:13 InetAddress.c
-rw-rw-r--. 1 wenzhou wenzhou 12280 Aug 15 11:13 net_util.c
-rw-rw-r--. 1 wenzhou wenzhou 6734 Aug 15 11:13 net_util.h
-rw-rw-r--. 1 wenzhou wenzhou 3968 Aug 15 11:13 proxy_util.c
-rw-rw-r--. 1 wenzhou wenzhou 1584 Aug 15 11:13 proxy_util.h
drwxrwxr-x. 2 wenzhou wenzhou 147 Jun 16 16:51 .
drwxrwxr-x. 12 wenzhou wenzhou 159 Jun 16 16:51 ..
[wenzhou@wenzhou jdk18u]$ strace -o strace -ff -e openat,execve make images
Building target 'images' in configuration 'linux-x86_64-server-release'
Updating support/modules_libs/java.base/libnet.so due to 5 file(s)
Creating jdk.sctp.jmod
Creating interim java.base.jmod
Creating interim jimage
Creating jdk.jlink.jmod
Creating java.base.jmod
Creating jdk image
Creating CDS archive for jdk image
Creating CDS-NOCOOPS archive for jdk image
Finished building target 'images' in configuration 'linux-x86_64-server-release'
... done in about 20 seconds....
[wenzhou@wenzhou jdk18u]$ ls -lat strace.* | wc
3120 28080 205920
[wenzhou@wenzhou jdk18u]$ grep -v ENOENT strace.* | cut -d'"' -f2 | grep -v "/jdk/" | grep "^/" | grep -v "^/proc/" | grep -v "^/tmp/" | grep -v "^/home/wenzhou" | grep -v "^/dev" | sort | uniq > result.txt
[wenzhou@wenzhou jdk18u]$ cat result.txt
/bin/sh
/etc/ld.so.cache
/etc/localtime
/etc/nsswitch.conf
/lib64/ld-linux-x86-64.so.2
/lib64/libacl.so.1
/lib64/libattr.so.1
/lib64/libbfd-2.30-113.el8.so
/lib64/libcap.so.2
/lib64/libcrypto.so.1.1
/lib64/libc.so.6
/lib64/libdl.so.2
/lib64/libgcc_s.so.1
/lib64/libgmp.so.10
/lib64/libmpc.so.3
/lib64/libmpfr.so.4
/lib64/libm.so.6
/lib64/libnss_sss.so.2
/lib64/libopcodes-2.30-113.el8.so
/lib64/libpcre2-8.so.0
/lib64/libpcre.so.1
/lib64/libpthread.so.0
/lib64/libreadline.so.7
/lib64/librt.so.1
/lib64/libselinux.so.1
/lib64/libsigsegv.so.2
/lib64/libstdc++.so.6
/lib64/libtinfo.so.6
/lib64/libz.so.1
/sys/devices/system/cpu
/sys/devices/system/cpu/online
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us
/sys/fs/cgroup/memory/user.slice/user-4208579.slice/user@4208579.service/memory.limit_in_bytes
/sys/fs/cgroup/memory/user.slice/user-4208579.slice/user@4208579.service/memory.stat
/sys/fs/cgroup/memory/user.slice/user-4208579.slice/user@4208579.service/memory.use_hierarchy
/usr/bin/as
/usr/bin/bash
/usr/bin/cat
/usr/bin/chmod
/usr/bin/cp
/usr/bin/cut
/usr/bin/date
/usr/bin/find
/usr/bin/gawk
/usr/bin/gcc
/usr/bin/git
/usr/bin/gmake
/usr/bin/grep
/usr/bin/ld
/usr/bin/ln
/usr/bin/ls
/usr/bin/make
/usr/bin/mkdir
/usr/bin/mv
/usr/bin/nice
/usr/bin/objcopy
/usr/bin/rm
/usr/bin/sed
/usr/bin/sort
/usr/bin/strip
/usr/bin/tee
/usr/bin/touch
/usr/bin/tr
/usr/bin/xargs
/usr/include/alloca.h
/usr/include/arpa/inet.h
/usr/include/asm/bitsperlong.h
/usr/include/asm/errno.h
/usr/include/asm-generic/bitsperlong.h
/usr/include/asm-generic/errno-base.h
/usr/include/asm-generic/errno.h
/usr/include/asm-generic/int-ll64.h
/usr/include/asm-generic/ioctl.h
/usr/include/asm-generic/ioctls.h
/usr/include/asm-generic/param.h
/usr/include/asm-generic/posix_types.h
/usr/include/asm-generic/socket.h
/usr/include/asm-generic/sockios.h
/usr/include/asm-generic/types.h
/usr/include/asm/ioctl.h
/usr/include/asm/ioctls.h
/usr/include/asm/param.h
/usr/include/asm/posix_types_64.h
/usr/include/asm/posix_types.h
/usr/include/asm/socket.h
/usr/include/asm/sockios.h
/usr/include/asm/types.h
/usr/include/assert.h
/usr/include/bits/byteswap.h
/usr/include/bits/confname.h
/usr/include/bits/cpu-set.h
/usr/include/bits/dirent.h
/usr/include/bits/dlfcn.h
/usr/include/bits/endian.h
/usr/include/bits/environments.h
/usr/include/bits/errno.h
/usr/include/bits/fcntl.h
/usr/include/bits/fcntl-linux.h
/usr/include/bits/floatn-common.h
/usr/include/bits/floatn.h
/usr/include/bits/getopt_core.h
/usr/include/bits/getopt_posix.h
/usr/include/bits/in.h
/usr/include/bits/ioctls.h
/usr/include/bits/ioctl-types.h
/usr/include/bits/libc-header-start.h
/usr/include/bits/local_lim.h
/usr/include/bits/long-double.h
/usr/include/bits/netdb.h
/usr/include/bits/param.h
/usr/include/bits/poll.h
/usr/include/bits/posix1_lim.h
/usr/include/bits/posix2_lim.h
/usr/include/bits/posix_opt.h
/usr/include/bits/pthreadtypes-arch.h
/usr/include/bits/pthreadtypes.h
/usr/include/bits/resource.h
/usr/include/bits/sched.h
/usr/include/bits/select.h
/usr/include/bits/setjmp.h
/usr/include/bits/sigaction.h
/usr/include/bits/sigcontext.h
/usr/include/bits/sigevent-consts.h
/usr/include/bits/siginfo-arch.h
/usr/include/bits/siginfo-consts-arch.h
/usr/include/bits/siginfo-consts.h
/usr/include/bits/signum-generic.h
/usr/include/bits/signum.h
/usr/include/bits/sigstack.h
/usr/include/bits/sigthread.h
/usr/include/bits/sockaddr.h
/usr/include/bits/socket.h
/usr/include/bits/socket_type.h
/usr/include/bits/ss_flags.h
/usr/include/bits/stat.h
/usr/include/bits/statx-generic.h
/usr/include/bits/statx.h
/usr/include/bits/stdint-intn.h
/usr/include/bits/stdint-uintn.h
/usr/include/bits/stdio.h
/usr/include/bits/stdio_lim.h
/usr/include/bits/stdlib-bsearch.h
/usr/include/bits/stdlib-float.h
/usr/include/bits/sys_errlist.h
/usr/include/bits/thread-shared-types.h
/usr/include/bits/time.h
/usr/include/bits/timex.h
/usr/include/bits/types/clockid_t.h
/usr/include/bits/types/clock_t.h
/usr/include/bits/types/cookie_io_functions_t.h
/usr/include/bits/types/error_t.h
/usr/include/bits/types/__FILE.h
/usr/include/bits/types/FILE.h
/usr/include/bits/types/__fpos64_t.h
/usr/include/bits/types/__fpos_t.h
/usr/include/bits/types.h
/usr/include/bits/typesizes.h
/usr/include/bits/types/__locale_t.h
/usr/include/bits/types/locale_t.h
/usr/include/bits/types/__mbstate_t.h
/usr/include/bits/types/sig_atomic_t.h
/usr/include/bits/types/sigevent_t.h
/usr/include/bits/types/siginfo_t.h
/usr/include/bits/types/__sigset_t.h
/usr/include/bits/types/sigset_t.h
/usr/include/bits/types/__sigval_t.h
/usr/include/bits/types/sigval_t.h
/usr/include/bits/types/stack_t.h
/usr/include/bits/types/struct_FILE.h
/usr/include/bits/types/struct_iovec.h
/usr/include/bits/types/struct_itimerspec.h
/usr/include/bits/types/struct_osockaddr.h
/usr/include/bits/types/struct_rusage.h
/usr/include/bits/types/struct_sched_param.h
/usr/include/bits/types/struct_sigstack.h
/usr/include/bits/types/struct_statx.h
/usr/include/bits/types/struct_statx_timestamp.h
/usr/include/bits/types/struct_timespec.h
/usr/include/bits/types/struct_timeval.h
/usr/include/bits/types/struct_tm.h
/usr/include/bits/types/timer_t.h
/usr/include/bits/types/time_t.h
/usr/include/bits/uintn-identity.h
/usr/include/bits/uio-ext.h
/usr/include/bits/uio_lim.h
/usr/include/bits/utsname.h
/usr/include/bits/waitflags.h
/usr/include/bits/waitstatus.h
/usr/include/bits/wchar.h
/usr/include/bits/wordsize.h
/usr/include/bits/xopen_lim.h
/usr/include/ctype.h
/usr/include/dirent.h
/usr/include/dlfcn.h
/usr/include/endian.h
/usr/include/errno.h
/usr/include/fcntl.h
/usr/include/features.h
/usr/include/gnu/stubs-64.h
/usr/include/gnu/stubs.h
/usr/include/inttypes.h
/usr/include/limits.h
/usr/include/linux/errno.h
/usr/include/linux/falloc.h
/usr/include/linux/ioctl.h
/usr/include/linux/limits.h
/usr/include/linux/param.h
/usr/include/linux/posix_types.h
/usr/include/linux/stat.h
/usr/include/linux/stddef.h
/usr/include/linux/types.h
/usr/include/netdb.h
/usr/include/net/if_arp.h
/usr/include/net/if.h
/usr/include/netinet/icmp6.h
/usr/include/netinet/in.h
/usr/include/netinet/in_systm.h
/usr/include/netinet/ip.h
/usr/include/netinet/ip_icmp.h
/usr/include/netinet/tcp.h
/usr/include/net/route.h
/usr/include/poll.h
/usr/include/pthread.h
/usr/include/rpc/netdb.h
/usr/include/sched.h
/usr/include/signal.h
/usr/include/stdc-predef.h
/usr/include/stdint.h
/usr/include/stdio.h
/usr/include/stdlib.h
/usr/include/string.h
/usr/include/strings.h
/usr/include/sys/cdefs.h
/usr/include/sys/ioctl.h
/usr/include/sys/param.h
/usr/include/sys/poll.h
/usr/include/sys/resource.h
/usr/include/sys/select.h
/usr/include/sys/socket.h
/usr/include/sys/stat.h
/usr/include/sys/time.h
/usr/include/sys/ttydefaults.h
/usr/include/sys/types.h
/usr/include/sys/ucontext.h
/usr/include/sys/uio.h
/usr/include/sys/un.h
/usr/include/sys/utsname.h
/usr/include/time.h
/usr/include/unistd.h
/usr/lib64/gconv/gconv-modules.cache
/usr/lib64/libbfd-2.30-113.el8.so
/usr/lib64/libc_nonshared.a
/usr/lib64/libc.so.6
/usr/lib64/libdl.so.2
/usr/libexec/gcc/x86_64-redhat-linux/8/cc1
/usr/libexec/gcc/x86_64-redhat-linux/8/collect2
/usr/libexec/gcc/x86_64-redhat-linux/8/liblto_plugin.so
/usr/lib/gcc/x86_64-redhat-linux/8/crtbeginS.o
/usr/lib/gcc/x86_64-redhat-linux/8/crtendS.o
/usr/lib/gcc/x86_64-redhat-linux/8/include/limits.h
/usr/lib/gcc/x86_64-redhat-linux/8/include/stdarg.h
/usr/lib/gcc/x86_64-redhat-linux/8/include/stddef.h
/usr/lib/gcc/x86_64-redhat-linux/8/include/stdint.h
/usr/lib/gcc/x86_64-redhat-linux/8/include/syslimits.h
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crti.o
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crtn.o
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libc.so
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libdl.so
/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libpthread.so
/usr/lib/gcc/x86_64-redhat-linux/8/libgcc.a
/usr/lib/gcc/x86_64-redhat-linux/8/libgcc_s.so
/usr/lib/locale/locale-archive
/usr/share/locale/locale.alias
/var/lib/sss/mc/passwd
Excellent. So we need to categorize all those as to what "pkg" or component they are part of ? that would need defining in an sbom build dependency...
Excellent. So we need to categorize all those as to what "pkg" or component they are part of ? that would need defining in an sbom build dependency...
I am thinking to do a test in the docker image (889fd9bd7658 docker.io/adoptopenjdk/alpine3_build_image:latest) as well ( so we are certain both my local machine and container can run the same command)
e.g
[wenzhou@wenzhou jdk18u]$ rpm -q bash
bash-4.4.20-3.el8.x86_64
[wenzhou@wenzhou jdk18u]$ which bash
/usr/bin/bash
[wenzhou@wenzhou jdk18u]$ grep bash result.txt
/usr/bin/bash
[wenzhou@wenzhou jdk18u]$ rpm -qf /usr/bin/bash
bash-4.4.20-3.el8.x86_64
[wenzhou@wenzhou jdk18u]$ cat result.txt | while read F; do rpm -qf "$F"; done | sort | uniq
bash-4.4.20-3.el8.x86_64
binutils-2.30-113.el8.x86_64
coreutils-8.30-12.el8.x86_64
cpp-8.5.0-10.1.el8_6.x86_64
file /sys/devices/system/cpu is not owned by any package
file /sys/devices/system/cpu/online is not owned by any package
file /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us is not owned by any package
file /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us is not owned by any package
file /sys/fs/cgroup/memory/user.slice/user-4208579.slice/user@4208579.service/memory.limit_in_bytes is not owned by any package
file /sys/fs/cgroup/memory/user.slice/user-4208579.slice/user@4208579.service/memory.stat is not owned by any package
file /sys/fs/cgroup/memory/user.slice/user-4208579.slice/user@4208579.service/memory.use_hierarchy is not owned by any package
findutils-4.6.0-20.el8.x86_64
gawk-4.2.1-4.el8.x86_64
gcc-8.5.0-10.1.el8_6.x86_64
git-core-2.31.1-2.el8.x86_64
glibc-2.28-189.5.el8_6.x86_64
glibc-all-langpacks-2.28-189.5.el8_6.x86_64
glibc-common-2.28-189.5.el8_6.x86_64
glibc-devel-2.28-189.5.el8_6.x86_64
glibc-headers-2.28-189.5.el8_6.x86_64
gmp-6.1.2-10.el8.x86_64
grep-3.1-6.el8.x86_64
kernel-headers-4.18.0-372.16.1.el8_6.x86_64
libacl-2.2.53-1.el8.x86_64
libattr-2.4.48-3.el8.x86_64
libcap-2.48-2.el8.x86_64
libgcc-8.5.0-10.1.el8_6.x86_64
libmpc-1.1.0-9.1.el8.x86_64
libselinux-2.9-5.el8.x86_64
libsigsegv-2.11-5.el8.x86_64
libstdc++-8.5.0-10.1.el8_6.x86_64
make-4.2.1-11.el8.x86_64
mpfr-3.1.6-1.el8.x86_64
ncurses-libs-6.1-9.20180224.el8.x86_64
openssl-libs-1.1.1k-6.el8_5.x86_64
pcre2-10.32-2.el8.x86_64
pcre-8.42-6.el8.x86_64
readline-7.0-10.el8.x86_64
sed-4.5-5.el8.x86_64
sssd-client-2.6.2-4.el8_6.1.x86_64
sssd-common-2.6.2-4.el8_6.1.x86_64
systemd-239-58.el8.x86_64
zlib-1.2.11-18.el8_5.x86_64
image docker.io/adoptopenjdk/centos6_build_image does not work with jdk18 with error that gcc ld version is too old
using docker.io/adoptopenjdk/alpine3_build_image.
in order to make strace
work need to add capacity for running container
podman run -it --cap-add=SYS_PTRACE docker.io/adoptopenjdk/alpine3_build_image:latest sh
or
docker run -it --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --security-opt apparmor=unconfined docker.io/adoptopenjdk/alpine3_build_image:latest sh
/ # apk add strace fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz (1/1) Installing strace (5.14-r0) Executing busybox-1.34.1-r7.trigger OK: 408 MiB in 216 packages
/ # uname -a Linux 5138db736ed1 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 x86_64 Linux
/workspace/jdk18u # bash configure --with-source-date=version .... /workspace/jdk18u # make images ... in about 10 mins... /workspace/jdk18u # ls -atl build/linux-x86_64-server-release/ total 108 -rw-r--r-- 1 root root 13627 Aug 15 14:00 build.log drwxr-xr-x 6 root root 81 Aug 15 14:00 images drwxr-xr-x 14 root root 4096 Aug 15 14:00 buildtools drwxr-xr-x 24 root root 4096 Aug 15 14:00 support drwxr-xr-x 8 root root 199 Aug 15 14:00 jdk drwxr-xr-x 9 root root 263 Aug 15 13:58 . drwxr-xr-x 3 root root 28 Aug 15 13:49 hotspot drwxr-xr-x 6 root root 177 Aug 15 13:49 make-support -rw-r--r-- 1 root root 1239 Aug 15 13:48 Makefile -rw-r--r-- 1 root root 2251 Aug 15 13:48 bootcycle-spec.gmk -rw-r--r-- 1 root root 8468 Aug 15 13:48 buildjdk-spec.gmk -rwxr-xr-x 1 root root 3805 Aug 15 13:48 compare.sh drwxr-xr-x 2 root root 64 Aug 15 13:48 configure-support -rw-r--r-- 1 root root 20905 Aug 15 13:48 configure.log -rw-r--r-- 1 root root 34221 Aug 15 13:48 spec.gmk drwxr-xr-x 4 root root 67 Aug 15 13:48 .. /workspace/jdk18u # ls -lat src/java.base/share/native/libnet total 40 drwxr-xr-x 2 root root 147 Aug 15 13:45 . drwxr-xr-x 12 root root 159 Aug 15 13:45 .. -rw-r--r-- 1 root root 1964 Aug 15 13:45 Inet4Address.c -rw-r--r-- 1 root root 2901 Aug 15 13:45 Inet6Address.c -rw-r--r-- 1 root root 2993 Aug 15 13:45 InetAddress.c -rw-r--r-- 1 root root 12280 Aug 15 13:45 net_util.c -rw-r--r-- 1 root root 6734 Aug 15 13:45 net_util.h -rw-r--r-- 1 root root 3968 Aug 15 13:45 proxy_util.c -rw-r--r-- 1 root root 1584 Aug 15 13:45 proxy_util.h /workspace/jdk18u # touch src/java.base/share/native/libnet/* /workspace/jdk18u # ls -lat src/java.base/share/native/libnet total 40 -rw-r--r-- 1 root root 1964 Aug 15 14:02 Inet4Address.c -rw-r--r-- 1 root root 2901 Aug 15 14:02 Inet6Address.c -rw-r--r-- 1 root root 2993 Aug 15 14:02 InetAddress.c -rw-r--r-- 1 root root 12280 Aug 15 14:02 net_util.c -rw-r--r-- 1 root root 6734 Aug 15 14:02 net_util.h -rw-r--r-- 1 root root 3968 Aug 15 14:02 proxy_util.c -rw-r--r-- 1 root root 1584 Aug 15 14:02 proxy_util.h drwxr-xr-x 2 root root 147 Aug 15 13:45 . drwxr-xr-x 12 root root 159 Aug 15 13:45 ..
/workspace/jdk18u # strace -o strace -ff -e openat,execve make images Building target 'images' in configuration 'linux-x86_64-server-release' Updating support/modules_libs/java.base/libnet.so due to 5 file(s) Creating jdk.sctp.jmod Creating interim java.base.jmod Creating interim jimage Creating jdk.jlink.jmod Creating java.base.jmod Creating jdk image Creating CDS archive for jdk image Creating CDS-NOCOOPS archive for jdk image Finished building target 'images' in configuration 'linux-x86_64-server-release' /workspace/jdk18u # ls -lat strace.* | wc 3109 27981 217630
/workspace/jdk18u # grep -v ENOENT strace.* | cut -d'"' -f2 | grep -v "/jdk/" | grep "^/" | grep -v "^/proc/" | grep -v "^/tmp/" | grep -v "^/workspace/jdk18u" | grep -v "^/dev" | sort | uniq > result.txt /workspace/jdk18u # cat result.txt /bin/bash /bin/cat /bin/chmod /bin/cp /bin/date /bin/grep /bin/ln /bin/ls /bin/mkdir /bin/mv /bin/nice /bin/rm /bin/sed /bin/sh /bin/touch /usr/bin/awk /usr/bin/cut /usr/bin/find /usr/bin/gcc /usr/bin/git /usr/bin/gmake /usr/bin/make /usr/bin/objcopy /usr/bin/sort /usr/bin/strip /usr/bin/tee /usr/bin/tr /usr/bin/xargs /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/as /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld /usr/libexec/gcc/x86_64-alpine-linux-musl/10.3.1/cc1 /usr/libexec/gcc/x86_64-alpine-linux-musl/10.3.1/collect2
/workspace/jdk18u # cat result.txt | while read F; do apk info --who-owns "$F"; done | sort | uniq ERROR: /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/as: Could not find owner package ERROR: /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: Could not find owner package /bin/bash is owned by bash-5.1.16-r0 /bin/cat symlink target is owned by busybox-1.34.1-r7 /bin/chmod symlink target is owned by busybox-1.34.1-r7 /bin/cp symlink target is owned by busybox-1.34.1-r7 /bin/date symlink target is owned by busybox-1.34.1-r7 /bin/grep is owned by grep-3.7-r0 /bin/ln symlink target is owned by busybox-1.34.1-r7 /bin/ls symlink target is owned by busybox-1.34.1-r7 /bin/mkdir symlink target is owned by busybox-1.34.1-r7 /bin/mv symlink target is owned by busybox-1.34.1-r7 /bin/nice symlink target is owned by busybox-1.34.1-r7 /bin/rm symlink target is owned by busybox-1.34.1-r7 /bin/sed is owned by sed-4.8-r0 /bin/sh is owned by busybox-1.34.1-r7 /bin/touch symlink target is owned by busybox-1.34.1-r7 /usr/bin/awk symlink target is owned by busybox-1.34.1-r7 /usr/bin/cut symlink target is owned by busybox-1.34.1-r7 /usr/bin/find symlink target is owned by busybox-1.34.1-r7 /usr/bin/gcc is owned by gcc-10.3.1_git20211027-r0 /usr/bin/git is owned by git-2.34.4-r0 /usr/bin/gmake symlink target is owned by make-4.3-r0 /usr/bin/make is owned by make-4.3-r0 /usr/bin/objcopy is owned by binutils-2.37-r3 /usr/bin/sort symlink target is owned by busybox-1.34.1-r7 /usr/bin/strip is owned by binutils-2.37-r3 /usr/bin/tee symlink target is owned by busybox-1.34.1-r7 /usr/bin/tr symlink target is owned by busybox-1.34.1-r7 /usr/bin/xargs symlink target is owned by busybox-1.34.1-r7 /usr/libexec/gcc/x86_64-alpine-linux-musl/10.3.1/cc1 is owned by gcc-10.3.1_git20211027-r0 /usr/libexec/gcc/x86_64-alpine-linux-musl/10.3.1/collect2 is owned by gcc-10.3.1_git20211027-r0
so the packages we need is
/workspace/jdk18u # cat result.txt | while read F; do apk info --who-owns "$F" 2>> filesnotinpackage.txt; done | sort | uniq | awk -F'by' '{print $2}' | sort |uniq bash-5.1.16-r0 binutils-2.37-r3 busybox-1.34.1-r7 gcc-10.3.1_git20211027-r0 git-2.34.4-r0 grep-3.7-r0 make-4.3-r0 sed-4.8-r0
a full build on my local RHEL generates 59637 strace files. with 750 lines in result.txt
cat result.txt | while read F; do rpm -qf "$F"; done | sort | uniq | grep -v "is not owned"
alsa-lib-devel-1.2.6.1-3.el8.x86_64
bash-4.4.20-3.el8.x86_64
binutils-2.30-113.el8.x86_64
bzip2-libs-1.0.6-26.el8.x86_64
coreutils-8.30-12.el8.x86_64
cpp-8.5.0-10.1.el8_6.x86_64
cups-devel-2.2.6-45.el8_6.2.x86_64
filesystem-3.8-6.el8.x86_64
findutils-4.6.0-20.el8.x86_64
fontconfig-devel-2.13.1-4.el8.x86_64
freetype-devel-2.9.1-4.el8_3.1.x86_64
gawk-4.2.1-4.el8.x86_64
gcc-8.5.0-10.1.el8_6.x86_64
gcc-c++-8.5.0-10.1.el8_6.x86_64
git-core-2.31.1-2.el8.x86_64
glibc-2.28-189.5.el8_6.x86_64
glibc-all-langpacks-2.28-189.5.el8_6.x86_64
glibc-common-2.28-189.5.el8_6.x86_64
glibc-devel-2.28-189.5.el8_6.x86_64
glibc-headers-2.28-189.5.el8_6.x86_64
gmp-6.1.2-10.el8.x86_64
grep-3.1-6.el8.x86_64
kernel-headers-4.18.0-372.16.1.el8_6.x86_64
libacl-2.2.53-1.el8.x86_64
libattr-2.4.48-3.el8.x86_64
libcap-2.48-2.el8.x86_64
libgcc-8.5.0-10.1.el8_6.x86_64
libmpc-1.1.0-9.1.el8.x86_64
libselinux-2.9-5.el8.x86_64
libsigsegv-2.11-5.el8.x86_64
libstdc++-8.5.0-10.1.el8_6.x86_64
libstdc++-devel-8.5.0-10.1.el8_6.x86_64
libX11-devel-1.6.8-5.el8.x86_64
libXext-devel-1.3.4-1.el8.x86_64
libXi-devel-1.7.10-1.el8.x86_64
libXrandr-devel-1.5.2-1.el8.x86_64
libXrender-devel-0.9.10-7.el8.x86_64
libXtst-devel-1.2.3-7.el8.x86_64
make-4.2.1-11.el8.x86_64
mpfr-3.1.6-1.el8.x86_64
ncurses-libs-6.1-9.20180224.el8.x86_64
openssl-libs-1.1.1k-6.el8_5.x86_64
pcre2-10.32-2.el8.x86_64
pcre-8.42-6.el8.x86_64
python3-abrt-addon-2.10.9-21.el8.x86_64
python3-libs-3.6.8-45.el8.x86_64
python3-pyparsing-2.1.10-7.el8.noarch
readline-7.0-10.el8.x86_64
sed-4.5-5.el8.x86_64
setup-2.12.2-6.el8.noarch
sssd-client-2.6.2-4.el8_6.1.x86_64
sssd-common-2.6.2-4.el8_6.1.x86_64
systemd-239-58.el8.x86_64
systemtap-sdt-devel-4.6-4.el8.x86_64
temurin-18-jdk-18.0.1.0.0.10-1.x86_64
xorg-x11-proto-devel-2020.1-3.el8.noarch
xz-libs-5.2.4-4.el8_6.x86_64
zip-3.0-23.el8.x86_64
zlib-1.2.11-18.el8_5.x86_64
zlib-devel-1.2.11-18.el8_5.x86_64
cat result.txt | while read F; do rpm -qf "$F"; done | sort | uniq | grep "is not owned"
file /home/wenzhou/.gitconfig is not owned by any package
file /home/wenzhou/.local/lib/python3.6/site-packages is not owned by any package
file /sys/devices/system/cpu is not owned by any package
file /sys/devices/system/cpu/online is not owned by any package
file /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us is not owned by any package
file /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us is not owned by any package
file /sys/fs/cgroup/cpu,cpuacct/cpu.shares is not owned by any package
file /sys/fs/cgroup/memory/user.slice/user-4208579.slice/user@4208579.service/memory.limit_in_bytes is not owned by any package
file /sys/fs/cgroup/memory/user.slice/user-4208579.slice/user@4208579.service/memory.stat is not owned by any package
file /sys/fs/cgroup/memory/user.slice/user-4208579.slice/user@4208579.service/memory.use_hierarchy is not owned by any package
file /usr/local/lib64/python3.6/site-packages is not owned by any package
file /usr/local/lib/python3.6/site-packages is not owned by any package
file /usr/local/lib/python3.6/site-packages/ruamel.yaml-0.17.21-py3.9-nspkg.pth is not owned by any package
what if some tools are not installed by rpm? e.g compile from source code, npm install, go install, pip install etc
Looks like the only one in that last which is unaccounted for is /usr/local/lib/python3.6/site-packages/ruamel.yaml-0.17.21-py3.9-nspkg.pth
which is interesting. I wonder where that came from? I don't recognise ruamel
, but it seems quite possible that it came from pip
somewhere along the lines, potentially as a dependency of something else (Although if it was a dependency I'd expect to see it show more "not owned by" things with a dependency I recognise in there. Would be interesting if we could identify which part of the process was trying to read that file.
3 strace files marked with ruamel.yaml. one example strace.txt
In a new container with both strace run on "configure" and "make" commands:
/worksapce/jdk18u/strace-build # cat result-build.txt | while read F; do apk info --who-owns "$F"; done | sort | uniq
ERROR: /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/as: Could not find owner package
ERROR: /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: Could not find owner package
ERROR: /usr/lib/jvm/jdk-17.0.4+8/bin/jar: Could not find owner package
ERROR: /usr/lib/jvm/jdk-17.0.4+8/bin/java: Could not find owner package
ERROR: /usr/lib/jvm/jdk-17.0.4+8/bin/javac: Could not find owner package
ERROR: /usr/lib/jvm/jdk-17.0.4+8/lib/jspawnhelper: Could not find owner package
/bin/bash is owned by bash-5.1.16-r0
/bin/cat symlink target is owned by busybox-1.34.1-r7
/bin/chmod symlink target is owned by busybox-1.34.1-r7
/bin/cp symlink target is owned by busybox-1.34.1-r7
/bin/date symlink target is owned by busybox-1.34.1-r7
/bin/grep is owned by grep-3.7-r0
/bin/ln symlink target is owned by busybox-1.34.1-r7
/bin/ls symlink target is owned by busybox-1.34.1-r7
/bin/mkdir symlink target is owned by busybox-1.34.1-r7
/bin/mv symlink target is owned by busybox-1.34.1-r7
/bin/nice symlink target is owned by busybox-1.34.1-r7
/bin/rm symlink target is owned by busybox-1.34.1-r7
/bin/sed is owned by sed-4.8-r0
/bin/sh is owned by busybox-1.34.1-r7
/bin/touch symlink target is owned by busybox-1.34.1-r7
/usr/bin/ar is owned by binutils-2.37-r3
/usr/bin/awk symlink target is owned by busybox-1.34.1-r7
/usr/bin/cut symlink target is owned by busybox-1.34.1-r7
/usr/bin/dirname symlink target is owned by busybox-1.34.1-r7
/usr/bin/find symlink target is owned by busybox-1.34.1-r7
/usr/bin/g++ is owned by g++-10.3.1_git20211027-r0
/usr/bin/gcc is owned by gcc-10.3.1_git20211027-r0
/usr/bin/git is owned by git-2.34.4-r0
/usr/bin/gmake symlink target is owned by make-4.3-r0
/usr/bin/make is owned by make-4.3-r0
/usr/bin/nm is owned by binutils-2.37-r3
/usr/bin/objcopy is owned by binutils-2.37-r3
/usr/bin/sort symlink target is owned by busybox-1.34.1-r7
/usr/bin/strip is owned by binutils-2.37-r3
/usr/bin/tee symlink target is owned by busybox-1.34.1-r7
/usr/bin/tr symlink target is owned by busybox-1.34.1-r7
/usr/bin/xargs symlink target is owned by busybox-1.34.1-r7
/usr/bin/zip is owned by zip-3.0-r9
/usr/libexec/gcc/x86_64-alpine-linux-musl/10.3.1/cc1 is owned by gcc-10.3.1_git20211027-r0
/usr/libexec/gcc/x86_64-alpine-linux-musl/10.3.1/cc1plus is owned by g++-10.3.1_git20211027-r0
/usr/libexec/gcc/x86_64-alpine-linux-musl/10.3.1/collect2 is owned by gcc-10.3.1_git20211027-r0
/worksapce/jdk18u/strace-config # cat result-config.txt | while read F; do apk info --who-owns "$F"; done | sort | uniq
ERROR: /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/as: Could not find owner package
ERROR: /usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: Could not find owner package
ERROR: /usr/lib/jvm/jdk-17.0.4+8/bin/java: Could not find owner package
ERROR: /usr/lib/jvm/jdk8/bin/java: Could not find owner package
ERROR: /usr/lib/jvm/jdk8u342-b07/bin/java: Could not find owner package
ERROR: /usr/lib/jvm/zulu14.29.23-ca-jdk14.0.2-linux_musl_x64/bin/java: Could not find owner package
ERROR: /usr/lib/jvm/zulu14/bin/java: Could not find owner package
ERROR: /usr/lib/jvm/zulu15.38.17-ca-jdk15.0.6-linux_musl_x64/bin/java: Could not find owner package
ERROR: /usr/lib/jvm/zulu15/bin/java: Could not find owner package
/bin/arch symlink target is owned by busybox-1.34.1-r7
/bin/bash is owned by bash-5.1.16-r0
/bin/cat symlink target is owned by busybox-1.34.1-r7
/bin/chmod symlink target is owned by busybox-1.34.1-r7
/bin/date symlink target is owned by busybox-1.34.1-r7
/bin/df symlink target is owned by busybox-1.34.1-r7
/bin/grep is owned by grep-3.7-r0
/bin/hostname symlink target is owned by busybox-1.34.1-r7
/bin/ln symlink target is owned by busybox-1.34.1-r7
/bin/ls symlink target is owned by busybox-1.34.1-r7
/bin/mkdir symlink target is owned by busybox-1.34.1-r7
/bin/mktemp symlink target is owned by busybox-1.34.1-r7
/bin/mv symlink target is owned by busybox-1.34.1-r7
/bin/rm symlink target is owned by busybox-1.34.1-r7
/bin/rmdir symlink target is owned by busybox-1.34.1-r7
/bin/sed is owned by sed-4.8-r0
/bin/sh is owned by busybox-1.34.1-r7
/bin/uname symlink target is owned by busybox-1.34.1-r7
/lib/ld-musl-x86_64.so.1 is owned by musl-1.2.2-r7
/usr/bin/autoconf is owned by autoconf-2.71-r0
/usr/bin/autom4te is owned by autoconf-2.71-r0
/usr/bin/awk symlink target is owned by busybox-1.34.1-r7
/usr/bin/basename symlink target is owned by busybox-1.34.1-r7
/usr/bin/cut symlink target is owned by busybox-1.34.1-r7
/usr/bin/diff symlink target is owned by busybox-1.34.1-r7
/usr/bin/dirname symlink target is owned by busybox-1.34.1-r7
/usr/bin/env symlink target is owned by busybox-1.34.1-r7
/usr/bin/expr symlink target is owned by busybox-1.34.1-r7
/usr/bin/find symlink target is owned by busybox-1.34.1-r7
/usr/bin/g++ is owned by g++-10.3.1_git20211027-r0
/usr/bin/gcc is owned by gcc-10.3.1_git20211027-r0
/usr/bin/gmake symlink target is owned by make-4.3-r0
/usr/bin/head symlink target is owned by busybox-1.34.1-r7
/usr/bin/ldd is owned by musl-utils-1.2.2-r7
/usr/bin/m4 is owned by m4-1.4.18-r2
/usr/bin/pkg-config is owned by pkgconf-1.8.0-r0
/usr/bin/readlink symlink target is owned by busybox-1.34.1-r7
/usr/bin/sort symlink target is owned by busybox-1.34.1-r7
/usr/bin/tar is owned by tar-1.34-r0
/usr/bin/tee symlink target is owned by busybox-1.34.1-r7
/usr/bin/time symlink target is owned by busybox-1.34.1-r7
/usr/bin/tr symlink target is owned by busybox-1.34.1-r7
/usr/libexec/gcc/x86_64-alpine-linux-musl/10.3.1/cc1 is owned by gcc-10.3.1_git20211027-r0
/usr/libexec/gcc/x86_64-alpine-linux-musl/10.3.1/cc1plus is owned by g++-10.3.1_git20211027-r0
/usr/libexec/gcc/x86_64-alpine-linux-musl/10.3.1/collect2 is owned by gcc-10.3.1_git20211027-r0
packages from both result files
autoconf-2.71-r0
bash-5.1.16-r0
binutils-2.37-r3
busybox-1.34.1-r7
g++-10.3.1_git20211027-r0
gcc-10.3.1_git20211027-r0
git-2.34.4-r0
grep-3.7-r0
m4-1.4.18-r2
make-4.3-r0
musl-1.2.2-r7
musl-utils-1.2.2-r7
pkgconf-1.8.0-r0
sed-4.8-r0
tar-1.34-r0
zip-3.0-r9
files not found by "apk" from both result files:
/usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/as
/usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld
/usr/lib/jvm/jdk-17.0.4+8/bin/jar
/usr/lib/jvm/jdk-17.0.4+8/bin/java
/usr/lib/jvm/jdk-17.0.4+8/bin/javac
/usr/lib/jvm/jdk-17.0.4+8/lib/jspawnhelper
/usr/lib/jvm/jdk8/bin/java
/usr/lib/jvm/jdk8u342-b07/bin/java
/usr/lib/jvm/zulu14.29.23-ca-jdk14.0.2-linux_musl_x64/bin/java
/usr/lib/jvm/zulu14/bin/java
/usr/lib/jvm/zulu15.38.17-ca-jdk15.0.6-linux_musl_x64/bin/java
/usr/lib/jvm/zulu15/bin/java
We could add to the generate SBOM script on Linux a function to generate the SBOM "build tools" for this set:
autoconf-2.71-r0
bash-5.1.16-r0
binutils-2.37-r3
busybox-1.34.1-r7
g++-10.3.1_git20211027-r0
gcc-10.3.1_git20211027-r0
git-2.34.4-r0
grep-3.7-r0
m4-1.4.18-r2
make-4.3-r0
musl-1.2.2-r7
musl-utils-1.2.2-r7
pkgconf-1.8.0-r0
sed-4.8-r0
tar-1.34-r0
zip-3.0-r9
strace dpkg scripting: https://github.com/adoptium/temurin-build/issues/3104#issuecomment-1270272405
The above scripting provide the necessary detail. https://github.com/adoptium/temurin-build/issues/3104#issuecomment-1270272405
A reproducible build investigation task to determine the what the complete input bill of materials are for building just a single native library.
For this task I suggest using a base library, eg. libnet.so, and determine the complete source, tooling input to re-build just that library. The task would involve something along the lines of: