"make bootstrap" expects object files in ./bootstrap directory, but they seem to be built in ./lib/freebasic

[ericherman]

gcc -o bootstrap/fbc lib/freebasic/linux-x86_64/fbrt0.o bootstrap/linux-x86_64/*.o lib/freebasic/linux-x86_64/libfb.a -lncurses -lm -pthread
gcc: error: bootstrap/linux-x86_64/*.o: No such file or directory
make: *** [makefile:1084: bootstrap/fbc] Error 1
eric@vanth:~/src/misc-sources/fbc(1)$ find . -iname linux-x86_64
./lib/freebasic/linux-x86_64
./src/gfxlib2/obj/linux-x86_64
./src/rtlib/obj/linux-x86_64
./tests/warnings/r/linux-x86_64
eric@vanth:~/src/misc-sources/fbc$ ls -l ./lib/freebasic/linux-x86_64/*.o
-rw-r--r-- 1 eric eric 1736 Sep 22 11:13 ./lib/freebasic/linux-x86_64/fbrt0.o
-rw-r--r-- 1 eric eric 1736 Sep 22 11:13 ./lib/freebasic/linux-x86_64/fbrt0pic.o
eric@vanth:~/src/misc-sources/fbc$

[jayrm]

See: Bootstrapping fbc on a new system

What is it that you are trying to do?

If you are using 'make bootstrap-dist' to create a package and 'make bootstrap' to build it, then files are expected to be in ./bootstrap.

If you are following the manual steps, then files are expected to be in ./lib/freebasic.

I added an example to the wiki showing the how to create and use a bootstrap package in more detail.

[jayrm]

Eric, weird, I received your response notification with the option to view it here (but it is not here).

Interesting paper, thanks for the link. https://www.acsa-admin.org/countering-trusting-trust-through-diverse-double-compiling/

Yes, fbc is a self-hosting compiler. The process for building a new fbc has always been, since the beginning of fbc, to build the new version with an installation of the previous official release, and using cross-compiling to other systems where fbc is not yet available.

Makefile targets 'bootstrap-dist' and 'bootstrap' are a means to create a bootstrap compiler for a few popular host/targets by taking the emitted .c or .asm from the build system to the host/target system. We don't have a separately maintained version of fbc written in some other language (like C or assembler) to bootstrap from independently.

For our Travis-CI builds, we use the previous official release of fbc to build the next release. You can see our scripts here: https://github.com/freebasic/fbc/tree/master/contrib/travis

Does this information answer the issue report?

[ericherman]

Hello Jeff,

On 09/23/2018 04:32 PM, coderjeff wrote:

Eric, weird, I received your response notification with the option to view it here (but it is not here).

I deleted the comment in part because I felt that if I wanted to raise the issue of having a bootstrap-from-source method that it should be a separate issue ... but more because I don't feel that github issues make a particularly good medium for discussion of vision or architecture or philosophy. But I'll try to capture some ideas here.

Interesting paper, thanks for the link. https://www.acsa-admin.org/countering-trusting-trust-through-diverse-double-compiling/

The essential notion here for me as a distro maintainer is one of whether or not my ancestor fbc has been compromised, and I look for a way to answer that question.

Yes, fbc is a self-hosting compiler. The process for building a new fbc has always been, since the beginning of fbc, to build the new version with an installation of the previous official release, and using cross-compiling to other systems where fbc is not yet available.

Yes. This is common, and not at all bad as the fast-track. Yet, it obviously leaves open a question with regards to being able to verify the integrity of the software, as Ken Thompson demonstrated in the mid-1980s.

Makefile targets 'bootstrap-dist' and 'bootstrap' are a means to create a bootstrap compiler for a few popular host/targets by taking the emitted .c or .asm from the build system to the host/target system. We don't have a separately maintained version of fbc written in some other language (like C or assembler) to bootstrap from independently.

A separate bootstrap-from-trusted toolchain need not be easy, or fast, but it should be documented and hopefully scriptable.

For instance, the golang boot-strap-from-C plan is to build an early version of golang which can then compile a later version, then it can walk the version tree forward in time until it gets to modern day.

(As it happens, there is also a short-cut: gcc has a golang front-end, which can be used to bootstrap, so we have options there.)

To create a process for verifying fbc compiler integrity, one approach I can imagine is that if the emitted .c and .asm files are roughly human-readable, then it might be reasonable to check those in, and use those as a starting point which can be examined and audited buy many eyes. Then the bootstrap-from-source story becomes 1) grab the bootstrap sources, 2) use those to build a bootstrap-fbc, then 3) use the bootstrap-fbc to build fbc.

I recognize, however, that this -- or another approach -- would be inviting more work. As I'm not excited about signing up for the job, I am not about to insist that "someone" prioritize doing it.

For our Travis-CI builds, we use the previous official release of fbc to build the next release. You can see our scripts here: https://github.com/freebasic/fbc/tree/master/contrib/travis

Travis answers the question, "have we broken existing tests?" - The "fast track" approach of using the fbc we elected to trust last time makes perfect sense for CI, because it will answer that question.

I'd probably want a higher level of scrutiny if I were building for a safety critical purpose -- where I might be asked what my level of certainty is regarding the question, "does the toolchain have a trojan?"

Does this information answer the issue report?

I'd have to look closely at the emitted .c and .asm files, which I have not yet done.

Thank you for following up with me.

Cheers, -Eric

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/freebasic/fbc/issues/102#issuecomment-423820851, or mute the thread https://github.com/notifications/unsubscribe-auth/ABHq7mvgx8TgV04_a5eAddCUcWLGLTQRks5ud5t6gaJpZM4W1ML8.

-- Eric Herman mobile:+31 620719662 skype:eric_herman jabber:eric.herman@gmail.com

[jayrm]

Eric, I appreciate you giving your (free) time to provide more information (I get the immediate impression that you have much more knowledge on this subject). If there's nothing else to add to this issue, can probably just close it out.

---

yea, it would be difficult to answer, with confidence, the question of "does the toolchain have a trojan?" given our current practices.

Then the bootstrap-from-source story becomes 1) grab the bootstrap sources, 2) use those to build a bootstrap-fbc, then 3) use the bootstrap-fbc to build fbc.

There are bootstrap source versions available last few fbc releases. So, I wrote the following script, 1) to provide another example related to the issue report, and 2) at least minimally, answer for myself, the question "is the bootstrap package self consistent?"

I only tried with FreeBASIC-1.05.0-source-bootstrap.tar.xz so far. (Sorry about the '~/' paths, but I'm not sure how the process will handle relative paths.)

#!/bin/sh
set -ex

# bootstrap source versions available are:
#   1.02.0, 1.02.1, 1.03.0, 1.04.0, 1.05.0

bootstrap_version='1.05.0'
bootstrap_package='FreeBASIC-1.05.0-source-bootstrap'
download_url='https://github.com/freebasic/fbc/releases/download'

# download source for the bootstrap compiler
cd ~
wget -O $bootstrap_package.tar.xz $download_url/$bootstrap_version/$bootstrap_package.tar.xz
tar xf $bootstrap_package.tar.xz

# build the bootstrap complier (c/asm sources)
cd ~/$bootstrap_package
make bootstrap

# rebuild the compiler (bas sources) using bootstrap compiler
cd ~/$bootstrap_package
make "FBC=~/$bootstrap_package/bin/fbc -i ~/$bootstrap_package/inc"

# At this point we'd like to also rebuild the bootstrap package to compare, but
# currently, '$ make bootstrap-dist' expects to be made in a clone of the git repo, so,
# we would need to make some modifications to the makefile first to have this work:
#   $ cd ~/$bootstrap_package
#   $ make bootstrap-dist "FBC=~/$bootstrap_package/bin/fbc -i ~/$bootstrap_package/inc"

# instead, remove the ./bootstrap directory, and create a fake git repo:
cd ~/$bootstrap_package
rm -rf bootstrap
git init
git add .
git -c user.name="none" -c user.email="none" commit -m "bootstrap"

# make a new bootstrap package using newly built compiler
make bootstrap-dist "FBC=~/$bootstrap_package/bin/fbc -i ~/$bootstrap_package/inc"

# compare the contents of the 2 bootstrap packages:
tardiff --modified ~/$bootstrap_package.tar.xz ~/$bootstrap_package/$bootstrap_package.tar.xz

Turns out, for me, there is a difference by one file: `./contrib/swig/swig.exe' that probably wasn't meant to be in the original bootstrap package.

[ericherman]

Yes. Thank you for your time and thoughtful treatment of the subject. Perhaps I will make some time next weekend to look at this again. Regardless, I think it makes sense to close this issue. Cheers!