Open Earnestly opened 5 years ago
oridb
commented
5 years ago Hm. What platform? (The change you cited only affects plan 9).
It's likely that the bootstrap just wasn't regenerated correctly. If you install, and then run make genbootstrap, do you get differences? If you do, can you send a pull request?
Earnestly
commented
5 years ago Ah, I thought you may remember me. This is on Linux, the linked commit is the current commit I was using, and yes running make genbootstrap seems to fix the issues.
This is the diff I get afterwards:
diff --git a/mk/bootstrap/bootstrap+Linux-x86_64.sh b/mk/bootstrap/bootstrap+Linux-x86_64.sh
index 8bc30cc3..fcaa108c 100755
--- a/mk/bootstrap/bootstrap+Linux-x86_64.sh
+++ b/mk/bootstrap/bootstrap+Linux-x86_64.sh
@@ -1,6 +1,24 @@
#!/bin/sh
# This script is generated by genbootstrap.sh
# to regenerate, run "make bootstrap"
+mkdir -p obj/lib/bio
+mkdir -p obj/lib/bld.sub
+mkdir -p obj/lib/crypto
+mkdir -p obj/lib/date
+mkdir -p obj/lib/escfmt
+mkdir -p obj/lib/fileutil
+mkdir -p obj/lib/flate
+mkdir -p obj/lib/http
+mkdir -p obj/lib/inifile
+mkdir -p obj/lib/iter
+mkdir -p obj/lib/json
+mkdir -p obj/lib/math
+mkdir -p obj/lib/regex
+mkdir -p obj/lib/std
+mkdir -p obj/lib/sys
+mkdir -p obj/lib/testr
+mkdir -p obj/lib/thread
+mkdir -p obj/mbld
pwd=`pwd`
set -x
$pwd/6/6m -O obj -I obj/lib/sys -I obj/lib/std -I obj/lib/bio -I obj/lib/regex -I obj/lib/thread mbld/config.myr
refi64
commented
5 years ago Ah, I thought you may remember me.
Something about this is just so sad...
Earnestly
commented
5 years ago I've decided to just include make genbootstrap in the usual build as it avoids this issue and any future issues that might originate from improperly generated bootstraps.
oridb
commented
5 years ago Keep in mind that genbootstrap assumes you have a working compiler already available. Bootstrap is there only to get you to a working compiler and build system.
At some point, I'm thinking of just having the bootstrap pull down prebuilt/presigned tarballs.
Earnestly
commented
5 years ago Oh, that's no good then. Ideally the bootstrap should not download anything. It should be possible to download everything necessary for a build and then not require network access beyond that point.
edit: some packaging systems will enforce this by using a separate network namespace.
oridb
commented
5 years ago The root of the problem is that the libraries are built with mbld; Mbld depends on the libraries. The bootstrap script hard codes a line by line replay of what the mbld that generated the bootstrap script did.
We need some way of getting a working mbld to build things. That mbld can come from anywhere, but it needs to come from somewhere. Options include a -bootstrap tarball, be generated via the mk/bootstrap/* script, be committed into the repo for each architecture, or be baked into a release tarball that's used to build from git.
I'm not sure what's the least painful. The bootstrap script keeps getting broken when the libraries change, though, and I'm becoming less of a fan of keeping it up to date.
oridb
commented
5 years ago (Also, I definitely remember you, but I don't memorize what platform everyone uses -- and even if I did, people sometimes molt, and come back with a new computer.)
Earnestly
commented
5 years ago Heh, fair enough. Not sure what to suggests here but keeping tarballs in the git repo sounds like a bad call. The chezscheme guys did this and eventually ended up with a repo over 1G in size.
oridb
commented
5 years ago What I'm starting to lean towards is that when I do make release, it should pack a prebuilt copy of mbld for each supported architecture/os combo; that'll use a few megs in the release, but internet is fast these days. Then, I'll require a previously installed version to build git. I can put those up online and have a convenience script for bootstrapping that downloads off the internet.
That's a bit inconvenient for someone to build git and demand no internet accesss, but it's probably workable, and a lot easier to maintain. Including bootstrap scripts with the releases would probably be a solution.
oridb
commented
5 years ago ...huh. that's another thought: Adding a commit id to the bootstrap script; when I bootstrap from git, I check out the commit and build that code, which should be in sync with the script. Once I do that, I'd have a working mbld, and can check out the previous branch and rebuild using the bootstrapped mbld.
refi64
commented
5 years ago What about just doing a pre-commit hook that makes sure the bootstrap is up to date?
akkartik
commented
5 years ago https://github.com/oridb/mc#build
If you are building from development then issue
make bootstrapafter the call to./configure.
When running from git, you may need to first run
make bootstrapon the first installation, or upon significant changes tombld.
Probably a stupid question, but what is the alternative to building from git that doesn't require running make bootstrap?
oridb
commented
5 years ago Have an old version available.
On Fri, Dec 14, 2018, 20:43 Kartik Agaram <notifications@github.com wrote:
https://github.com/oridb/mc#build
If you are building from development then issue make bootstrap after the call to ./configure.
When running from git, you may need to first run make bootstrap on the first installation, or upon significant changes to mbld.
Probably a stupid question, but what is the alternative to building from git that doesn't require running make bootstrap?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/oridb/mc/issues/184#issuecomment-447537083, or mute the thread https://github.com/notifications/unsubscribe-auth/AFIphDMvmnF-Ea_kEGkb6i6VAGqo5nA_ks5u5H4LgaJpZM4Y6V7K .
akkartik
commented
5 years ago You don't distribute binaries of old versions, right? So there's no way for someone coming in now to build it? Is this just not a priority?
refi64
commented
5 years ago @akkartik AFAIK the bootstrap scripts were already fixed/regenerated on master; this bug is more about the long-term solving of the issue of the build scripts constantly going out of sync.
Of course, even if they weren't, the stable releases still work. Building the last release and then Git isn't really that much harder than in languages that are self-hosted.
oridb
commented
5 years ago Correct: make bootstrap should work on all platforms except NetBSD at the moment. I haven't had a chance to correct that one yet.
For robustness, I'll probably figure out some sort of continuous integration + commit hook that rewrites the bootstrap soon. The annoyance is that it's going to have to be an external system, because there are a number of platforms that need this. I'll try to get to that in the next week or to.
akkartik
commented
5 years ago Ah, that makes sense thanks. I found this issue because I ran into that cp: cannot stat 'obj/mbld/mbld': No such file or directory error message. I'm on Mac OS X:
$ uname -a
Darwin 4133 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64 i386 MacBookPro14,1 DarwinThe steps I ran were:
$ git clone https://github.com/oridb/mc
$ cd mc
$ ./configure
$ make bootstrap
$ make bootstrapAs you can see, rerunning make bootstrap didn't fix the problem like it seems to have done for @Earnestly above. Let me know if you'd like me to create a fresh issue for this.
akkartik
commented
5 years ago The same commit hash (261918774d46c50c96f729f4e92b01323c56c398) bootstraps fine on Ubuntu 18.04.1. I hadn't tried that before, thanks.
Earnestly
commented
5 years ago The issue has occurred again. I like the suggestion by @kirbyfan64 of using commit hooks or something of that nature because it would be a great shame to loose such an effortless bootstrap, one better than any other language and runtime I've had to compile.
As a workaround I've decided to make genbootstrap conditional on the presence of mbld in the system. This is only temporary as each package should be self-contained.
oridb
commented
5 years ago I think that in the next few days, I'll set up a few machines to do the build and regeneration on.
oridb
commented
5 years ago Should be 90% done now -- the bootstrap script to regenerate across all platforms is committed, but it needs a build farm to run the bootstrap. Some of the machines are local -- partly because I haven't yet set them up on a vultr node, but the OSX machine will probably stay where it is because, well, there are no good hosting providers for mac infrastructure.
It may end up in a colo at some point.
lojikil
commented
4 years ago I'm seeing a related issue on OS X:
+ ld -pagezero_size 0x100000000 -macosx_version_min 10.6 -o obj/mbld/mbld /Users/lojikil/Build/mc/rt/_myrrt.o obj/mbld/deps.o obj/mbld/main.o obj/mbld/util.o obj/mbld/cpufeatures.o obj/mbld/libs.o obj/mbld/syssel.o obj/mbld/config.o obj/mbld/opts.o obj/mbld/subtest.o obj/mbld/types.o obj/mbld/test.o obj/mbld/install.o obj/mbld/parse.o obj/mbld/build.o -Lobj/lib/thread -lthread -Lobj/lib/bio -lbio -Lobj/lib/regex -lregex -Lobj/lib/std -lstd -Lobj/lib/sys -lsys -macosx_version_min 10.6
ld: warning: passed two min versions (10.6.0, 10.6) for platform macOS. Using 10.6.
Undefined symbols for architecture x86_64:
"_std$intfmt", referenced from:
.L163 in libstd.a(fmt.o)
.L164 in libstd.a(fmt.o)
.L165 in libstd.a(fmt.o)
.L166 in libstd.a(fmt.o)
.L167 in libstd.a(fmt.o)
.L168 in libstd.a(fmt.o)
.L169 in libstd.a(fmt.o)
...
ld: symbol(s) not found for architecture x86_64
+ true
cp obj/mbld/mbld xmbld
cp: obj/mbld/mbld: No such file or directory
make: *** [bootstrap] Error 1
lojikil@malum [~/Build/mc]
% uname -a
Darwin malum.local 19.2.0 Darwin Kernel Version 19.2.0: Sat Nov 9 03:47:04 PST 2019; root:xnu-6153.61.1~20/RELEASE_X86_64 x86_64The build sequence was:
git clone ...
./configure --prefix=/usr/local
make bootstrapwhich then failed with the above
Earnestly
commented
4 years ago @oridb Perhaps consider using https://man.sr.ht/builds.sr.ht/ for generating bootstraps on multiple platforms?
oridb
commented
4 years ago Yeah, that's a good idea. I'll probably try to get to it next weekend.
Building after a clean clone with the following:
Yields a new error:
Interestingly, if I run
make bootstrapagain, it completes without issue.