search

TritonDataCenter / rfd

Requests for Discussion
Mozilla Public License 2.0
261 stars 77 forks source link

RFD 141 Platform Image Build v2 (PIBv2) #101

Open mgerdts opened 6 years ago

mgerdts commented 6 years ago

This is for discussion of RFD 141

https://github.com/joyent/rfd/blob/master/rfd/0141/README.md

cburroughs commented 6 years ago

Short of running builds (very slowly) in VMware, there is no way for Joyent engineers to build a PI on the legendary PI from 2015.

You could run this on https://github.com/joyent/joyent-retro in one of the staging DCs.

As you have outlined it sounds like the oral tradition around min_platform has become deeply tangled. Do you know if it is written anywhere that the min_platform for application code is the same as the minimum platform to build a new PI? I am not aware of this ever being the case. Regardless of the outcome of this RFD we should correct anything that says that. The Jenkins docs are explicit that agents building PIs "don't care about the host's platform version", but that is buried in the middle of a lot of text.

How would the smartos-extra-<hash>.tar.bz2 tarball be managed? Is it just cached locally? Something we publish and maintained along with releases? Something else?

Other:

mgerdts commented 6 years ago

Recent conversation in ~os had these highlights:

mgerdts commented 6 years ago

I tried a smartos-live build with the 20151126 image and it failed while building proto.strap.

+ /home/mgerdts/smartos/proto.strap/usr/bin/gcc -fident -finline -fno-inline-functions -fno-bu
iltin -fno-asm -fdiagnostics-show-option -nodefaultlibs -D__sun -O -fno-inline-small-functions
 -fno-inline-functions-called-once -m32 -Wall -Wextra -Werror -Wno-missing-braces -Wno-sign-co
mpare -Wno-unknown-pragmas -Wno-unused-parameter -Wno-missing-field-initializers -Wno-array-bo
unds -Wno-unused -Wno-empty-body -std=gnu99 -fno-inline-small-functions -fno-inline-functions-
called-once -fno-ipa-cp -D_TS_ERRNO -DOSNAME="illumos" -D_FILE_OFFSET_BITS=64 -nostdinc -I/usr
/include -I/opt/local/include -D_FILE_OFFSET_BITS=64 -nostdinc -I. -include fts.h -I/opt/local
/include -c -o dba.o /home/mgerdts/smartos/projects/illumos/usr/src/cmd/mandoc/dba.c
/home/mgerdts/smartos/projects/illumos/usr/src/cmd/mandoc/dba.c:24:20: error: endian.h: No suc
h file or directory
cc1: warnings being treated as errors
/home/mgerdts/smartos/projects/illumos/usr/src/cmd/mandoc/dba.c: In function 'dba_macro_new':
/home/mgerdts/smartos/projects/illumos/usr/src/cmd/mandoc/dba.c:378: error: implicit declarati
on of function 'ntohl' [-Wimplicit-function-declaration]
*** Error code 1
make: Fatal error: Command failed for target `dba.o'
Current working directory /home/mgerdts/smartos/projects/illumos/usr/src/tools/mandoc

It turns out that /usr/include/endian.h was added later.

$ git blame manifest | grep usr/include/endian.h
b22cc2b5 (Jerry Jelinek      2016-05-20 11:11:36 +0000  2705) f usr/include/endian.h 0644 root bin

The use of <endian.h> by mandoc was introduced with:

$ git log -n 1 a40ea1a7
commit a40ea1a7d80eee1b409e9dcc2e48c730988147ea
Author: Yuri Pankov <yuri.pankov@nexenta.com>
Date:   Sun May 28 20:36:38 2017 +0300

    8297 update mdocml to 1.14.1
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Approved by: Richard Lowe <richlowe@richlowe.net>
mgerdts commented 6 years ago

Given the troubles mentioned above, I jumped forward to after /usr/include/endian.h was introduced. With joyent_20160527T033529Z, the build almost completes. While building the live target, it fails because the PI's ucodeadm doesn't work.

$ NO_CLEANUP=1 make live
(cd tools/mancheck && gmake mancheck CC=/opt/local/bin/gcc CTFMERGE=/home/mgerdts/smartos/projects/illumos/usr/src/tools/proto/*/opt/onbld/bin/i386/ctfmerge CTFCONVERT=/home/mgerdts/smartos/projects/illumos/usr/src/tools/proto/*/opt/onbld/bin/i386/ctfconvert ALTCTFCONVERT=/home/mgerdts/smartos/projects/illumos/usr/src/tools/proto/*/opt/onbld/bin/i386/ctfconvert-altexec MAX_JOBS=8)
...
Importing GZ SMF manifests                                                ... done (25s)
Building SMF seeds                                                        ... done (6s)
Generating microcode files                                                ...
ucodeadm: /tmp/build_live-1000.14523/a/platform/i86pc/ucode/intel-ucode.txt: File header is invalid

BUILD FAILURE:
  [  2] bi_expand_ucode
        (file "./tools/build_live" line 352)
  [  3] main
        (file "./tools/build_live" line 960)

CLEANING UP ON FAILURE ...
| WARNING: skipping cleaning; tmpdir = /tmp/build_live-1000.14523
... DONE

build_live: ERROR: failed to generate Intel microcode files
Makefile:124: recipe for target 'live' failed
make: *** [live] Error 1

The command that failed is found in the log file:

$ grep ucodeadm log/build_live.20180515T162632Z.1526414136.log  | tail -1
[2018-05-15T19:56:49Z] ./tools/build_live:351: bi_expand_ucode(): pfexec /usr/sbin/ucodeadm -i -R /tmp/build_live-1000.14523/a/platform/i86pc/ucode/GenuineIntel /tmp/build_live-1000.14523/a/platform/i86pc/ucode/intel-ucode.txt

$ pfexec /usr/sbin/ucodeadm -i -R /tmp/build_live-1000.14523/a/platform/i86pc/ucode/GenuineIntel /tmp/build_live-1000.14523/a/platform/i86pc/ucode/intel-ucode.txt
ucodeadm: /tmp/build_live-1000.14523/a/platform/i86pc/ucode/intel-ucode.txt: File header is invalid

Running the just-built version of ucodeadm has better results.

$ pfexec proto/usr/sbin/ucodeadm -i -R /tmp/build_live-1000.14523/a/platform/i86pc/ucode/GenuineIntel /tmp/build_live-1000.14523/a/platform/i86pc/ucode/intel-ucode.txt
$ echo $?
0

This suggests that joyent_20160527T033529Z is also too old to be the CBE's PI.

pfmooney commented 6 years ago

The current use of the platform ucodeadm, rather than one built from the sources for use in the live image generation is a known issue. Robert was intending to address soon.

rmustacc commented 6 years ago

Hi Mike,

First, thanks for putting this together. I think it captures a lot of the challenges that we have with the build process today and a lot of the things that you bring up here overlap with things that we've talked about historically, but not done a good job of writing down and clearly communicating.

In particular, things like having pre-build proto.strap directory trees available and a separate proto area for different components are both important things that we should adopt. I have a lot of thoughts on some of the methods proposed for how we might do this and in particular, the challenges with illumos-extra after the normal build. Before I get to those, I'd like to share a lot of background and history. I know that there will be a lot here, so I appreciate your time with it all.

Background

I'd like to give a bit of background and try to share some of the thoughts behind some of the design decisions and how things ended up the way that they have. One of the biggest challenges that we had at Joyent as we were bootstrapping SmartOS and dealing with the illumos build was just the amount of hard coded assumption on the build machine and its contents. That things were hardcoded to look in /usr and have packages installed, making it quite challenging to build not just illumos but the general platform image.

When we first had generalized building, we had to have a process we called the 'fake-subset' which basically copied over /usr, added a bunch of things the build required, and then would lofs mount over /usr. If reading the description doesn't give you a visceral feeling of grossness, then I hope you'll take my word that it was a pretty terrible thing and we had to work pretty hard to destroy it. This was done because our /usr wasn't writable in the vein of spare zones.

One of the major directional things was to change the notion of what a 'CBE' was and basically make it possible to build illumos in a world where the required items at the right versions needed to build were independent of the version that ran the build system.

To summarize this there were several major goals of what you call PIBv1, though it's really been a much more evolutionary process. These goals were:

Goal 1: Be able to do a full a build from a minimal system with a C compiler and some other basic packages.

This is a noble goal and is an important part of understanding the history and rationale of the current build process. The idea is that we should be able to build, from source, all of the dependents and components that we need. This was designed to ensure that we had the right versions of dependent artifacts even if the other packaging methods on the system (for us generally pkgsrc, but the same is true for any other) did not have the software that we wanted or in the method that made sense.

That said, there are some challenges and we never really achieved it. Some of these challenges are:

Goal 2: Correctness over build performance, think of the build like a cross-compilation.

One of the important goals was that the build always be as correct as possible. Part of the reason that illumos-extra is rebuilt after illumos (and we should do better incremental building here) is to ensure that things were always correct. While it's in true that most of this software should only be relying on public ABIs, it still does need to have been built against the proto area that we are actually shipping against. This is probably where one of the biggest frustration and views of time loss are.

Historically, building illumos or Solaris was never thought of as a cross-compilation. However, when we're building SmartOS, we're building the entire image and there the mentality that we've had is that this is intended to be more like a cross-compilation process. This is partly why we'll rebuild the pieces that we're dependent on in the context of the proto area (and only the proto area). We have a rather different release story than Solaris does, especially with respect to when new symbols are allowed to appear.

I realize that this may be a weird way to think about the platform build, but I believe it is an important way to think about it and probably helps inform some of the decisions that we've had over the years.

There are a number of tickets and projects that were done and still have to be done to make this clearer. For example, the work to add support for LD_TOXIC_PATH and its original motivation in the form of bugs like OS-2215 and OS-2724.

There are also a number of things that have fallen out that have been useful properties that weren't intentional. These are:

Incidental 1: Networkless Builds

A nice property of today's build that's almost completely true is that after running ./configure, you can completely run the entire build offline. I've used this a lot while developing while traveling or in other environments without network access.

That said, this has slightly regressed due to some aspects of mdb_v8, but it's still a nice property to try and maintain while practical.

Incidental 2: Helping deal with the QDS

One of the rather important incidental aspects of the current SmartOS build is that when building illumos-extra's second pass (software included in the platform image) is that it has readily helped us with the QDS. Because third-party software that's in illumos-extra will see all of the new symbols and headers that have been exposed in illumos, it allows us to always be testing changes there in the context of -extra.

This has helped us find at runtime, issues with recent changes that were missed during testing, or that someone didn't discover from upstream based on the way they had changed things.

It's also allowed us to take advantage of things that have been fixed in the build and software. For example, we can take immediate advantage of fixes to the CRT, rtld, etc. when building software in extra, whether at build time or build considerations for runtime. Another example of this is support for preadv/pwritev helping us find issues with QEMU IOV_MAX (HVM-815 and HVM-846).

The other way that things help with the QDS is that by having everything always be built the same way we build it for customers, we get slightly closer to reducing errors that wouldn't be found. That said, I know that this isn't perfect and that there are a lot of cases that it misses.

Incidental 3: Builds are faster if you're willing to take on the risk

Note, this may really just be a different view on the correctness problem.

The general design is that if you remove 0-illumos-stamp (or the stamp for the corresponding component) and rerun gmake live at the top-level of smartos-live, you always have a correct build.

That said, if a developer wants to do an incremental build of a component to speed things up and take on the risks associated with it, it has always been possible to do so using bldenv and co. One can run nightly as well. It is more challenging to do a full nightly where you want to remove the proto area due to the issues with -extra as you point out, which definitely motivates separate proto areas.

Affirmation of Problems

While I've written all of that background and I will have a discussion of some of the different takes I have on the proposals, I want to agree with what I see as some of the problems that folks have:

Discussion on Proposals

Separate Proto Areas

I agree that it makes sense to establish separate proto areas for the illumos build and everything else. This will make the image construction a little more complicated, but it is probably a worthwhile change as it will make it easier to do illumos iteration and it will also enable something I've long wanted to be able to do: use nightly's ability to create both the debug and non-debug proto areas. Often times I've had to hack around that with just having two workspaces and trying to sync diffs there, but MULTI_PROTO is much more elegant.

One question I have here is: is the intent going to be to have a proto area per-project? In other words is there one proto area for each local project, illumos, illumos-extra, and smartos-live? Perhaps we should just stick with one for illumos and one for everything else for now?

Bootstrapping proto.strap

Today the proto.strap contains three different sets of software:

  1. The software required to build illumos, but not required at run-time (binutils).

  2. The software required at build-time to build illumos because it is a run-time requirement (openssl). A separate copy of this software is built for run time against the illumos bits.

  3. The software which is required at build-time to build illumos, but is not part of what we ship at run-time because the run-time components are not shipped (glib, d-bus, etc.)

I 100% agree that we need to have the ability to allow developers to opt in to being able to get access to this software. Especially as we want to allow for more compiler warnings and the like to be usable by folks in a development environment, forcing people to build multiple compilers is not a good decision.

I'm not sure I agree with the proposed methods for this.

Where software comes from for the build

When we look at how the build works, there are a couple of different places that software comes from that is used to build. It's a bit of a mess, but before we discuss smartos-extra or pkgsrc, it's worth talking about what that is:

  1. We use tools from pkgsrc (gmake, bootstrap compiler, java, python, etc.)
  2. We use tools that someone previously built from illumos and turned into pacakges (sgstools, dmake, etc.).
  3. We use tools from the build system (ld.so.1, mcs, elfdump)
  4. We use binaries we download (Studio, the adjuncts tgz)
  5. We build a subset of illumos-extra for proto.strap

smartos-extra

To me, I think it is important that the smartos build is able to always build everything. If a developer wants to take the risks associated with wanting a faster build, they do that themselves, and that we do not opt into it for at least the CI and release builds.

The same way that someone uses bldenv and its up to them to know the dependencies (which I'm not advocating as a permanent thing, but more so a reflection of the current reality), I think the same should be true with how we view the smartos-extra bits. Now, the risk for say proto.strap is pretty low. However, I do not believe we should give up the QDS-avoidance properties that we get from illumos-extra today.

I do appreciate that the current situation isn't perfect either. Maybe we can find a compromise position for all this? I really don't like the idea that illumos-extra is going to be snapshot against some specific point in time as far as builds go and that our official builds will just use something built elsewhere. When do we roll that snapshot forward? Every six months, every two years? How do we make sure we can actually always still build? It is a tempting position to take and I can understand, especially from the context of only caring about one component, why it's what someone wants, but I'm not sure it's going to be ultimately the most healthy.

I'm not sure if pkgsrc is the right solution for us or not. Is the vision with pkgsrc that we'll end up tracking specific branches? Are we going to stay on an LTS for all this software or do something else? pkgsrc isn't designed as a cross-compiler. It's designed to build everything and be installed into that specific prefix. It does a lot of things that we don't want with prefixes being encoded as rpaths, etc. It's not generally designed to be relocatable. So it's not clear to me exactly how that will work. It may also be that pkgsrc has changed in its infrastructure for dealing with this, but it's not clear to me that pkgsrc is going to be the right fit here.

Importantly, I'm not sure how we convince the equivalent of 1-extra-stamp to build in a pkgsrc setting and build against the current proto area that someone is building. pkgsrc build artifacts are generally not relocatable. They are built for a specific prefix, the compiler run time paths are embedded in all binaries, etc. Building against the illumos proto area is also important because we don't always ship all of the items in the platform live image that we would want to use at build time.

I understand why that is important for helping developers be able to build faster, but I'm not sure the corresponding correctness hit is good. The RFD makes the important point that if private interfaces change and software relies on them, then we need to trigger a rebuild, but it doesn't indicate how folks will figure that out. I'd argue that most folks don't realize that they're depending on private interfaces today and one minor advantage of the current design of the build is that it doesn't matter.

That said, I would like to find some way to opt in to it, even if it's not the default as I think a lot of people really want to be able to deal with this and complain about it. Maybe we should have a tarball that we build periodically with these components as a snapshot? Though we should be careful about what that means for integration for a number of changes.

Conclusions

I realize this is a very long comment at this point. I just want to make sure it's clear that I do understand the system today isn't ideal, and it can be improved. There are a lot of good, concrete details in the RFD for improvement. I hope we can figure out some different approaches to move forward.

I'm also happy to start looking at some of the things like multiple proto areas and dealing with the some of the mechanics of how to deal with proto.strap issues.