Consider making Build Reproducible

[sensslen]

In order to use Raspbian for industrial purposes (e.g. using the Compute Module) following requirements are there:

1. Ability to reproduce same image at a later time
2. Be able to distribute the source code and also instructions on how to build to anyone requesting it (as stated by the GPL Licence)

There is a tool called elbe (https://github.com/Linutronix/elbe) that does exactly this - maybe it would be possible to integrate with this.

[MichaelTiernan]

Just for the record, I added #435 which dovetails with this situation too.

I have yet to be able to produce a build twice in a row. :(

I'm sure it can be done but I'm not there yet.

[starbasessd]

I would think, since this uses the current apt depositories, downloading and installing the latest updates as it builds, without some sort of log of which version of each file installed was, it would be nigh impossible?

[sensslen]

Well this is exactly what the suggested tool does. In addition it also stores all installed packages as well as it‘s sources. This allows to rebuild the exact image by using the downloaded sources instead of the repositories.

[MichaelTiernan]

Yes, this is what the tool is intended to do.

The reality is that it isn't providing a consistent reproducibility. It's only recently that, after too much effort, I find that there's a known problem building a version that is beyond the core CLI since there's a broken (not the fault/responsibility of pi-gen) package that fails to install without indication. Breaking the build.

I'm now forced to build a purpose specific image by taking a stock image and opening it up to manipulate it then roll it back up.

[starbasessd]

Seriously curious: Wouldn't it be 'better' to manipulate your own image generated by pi-gen, and archived, than hope that a previous image generated by RPiFoundation has archived? What package fails to install because it's broken? I build lots of custom images, and have archives of those that were successful back a year or 2. They save me so much time when I am deploying my images, instead of having always to sudo raspi-config or edit /etc/something.conf every time I deploy an image. (I do Custom Tech Support / Help Desk and reproduce environments of my clients)

On Sun, Sep 20, 2020 at 7:01 AM Michael Tiernan notifications@github.com wrote:

Yes, this is what the tool is intended to do.

The reality is that it isn't providing a consistent reproducibility. It's only recently that, after too much effort, I find that there's a known problem building a version that is beyond the core CLI since there's a broken (not the fault/responsibility of pi-gen) package that fails to install without indication. Breaking the build.

I'm now forced to build a purpose specific image by taking a stock image and opening it up to manipulate it then roll it back up.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/RPi-Distro/pi-gen/issues/427#issuecomment-695773817, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEZTUHJQ2L25QZ2Q3TB5ZL3SGXOI7ANCNFSM4OZI6TMQ .

-- Thanks

Kevin Shumaker

Personal Tech Support https://kevinshumaker.wixsite.com/thethirdlevel

N38° 19' 56.52" W85° 45' 8.56"

Semper Gumby “Don't tell people how to do things. Tell them what to do and let them surprise you with their results.” - G.S. Patton, Gen. USA Ethics are what we do when no one else is looking. Quis custodiet ipsos custodes? “There is no end to the good you can do if you don’t care who gets the credit.” - C Powell You know we're sitting on four million pounds of fuel, one nuclear weapon and a thing that has 270,000 moving parts built by the lowest bidder. Makes you feel good, doesn't it?

[sensslen]

While modifying stock images does work, it is not sufficient to properly fulfill hol licensing requirements (at least to my understanding) which requires to provide a copy of the source code as well as the instructions on how to do the build. So I‘m currently using a stock image and a custom repository to install additional stuff (and remove unnecessary stuff). To me this process does not feel too robust, since I do have to manage a repository and a set of scripts that do the modification. It would be much easier if I could maintain a set of rules to build the operating system and as a result would get all required information to fulfill licensing requirements and also rebuild a image (possibly with minor modifications).

[MichaelTiernan]

I got a minute to do the digging. First, I tracked this down: https://www.raspberrypi.org/forums/viewtopic.php?t=235664 Which explains about how a silent failure causes the build to actually fail but there's no outward indication. That leads you to: https://github.com/RPi-Distro/pi-gen/issues/271

Which details the failure and the end effect resulting in a build that actually fails silently. The outward sign is the icons are not there but I don't know if there's more problems behind the scenes.

I want to be able to produce an image that from the get-go has my networking, packages, users, etc already configured so that when the device is booted, its ready to go and there's an IMAGE file on my hard drive that would allow me to replicate (quickly) this same machine.

The "test" I used was, can I build, using this tool, that which is distributed as a "stock" image.

[starbasessd]

I had forgotten my input on that issue (verified, with my build and usage environments), and my solution, until I hear differently, was to switch to a Debian 32bit VM for all my builds. Then I simply archive the .img files as I build them (noting if I made any changes to the build script(s)), and saving the build script outputs to show which versions of apps were included from the repositories. If needed, then I can take an image and force a re-install of an older pkg if needed. Sometimes painful, but it works for me.

On Fri, Sep 25, 2020 at 10:40 AM Michael Tiernan notifications@github.com wrote:

I got a minute to do the digging. First, I tracked this down: https://www.raspberrypi.org/forums/viewtopic.php?t=235664 Which explains about how a silent failure causes the build to actually fail but there's no outward indication. That leads you to:

271 https://github.com/RPi-Distro/pi-gen/issues/271

Which details the failure and the end effect resulting in a build that actually fails silently. The outward sign is the icons are not there but I don't know if there's more problems behind the scenes.

I want to be able to produce an image that from the get-go has my networking, packages, users, etc already configured so that when the device is booted, its ready to go and there's an IMAGE file on my hard drive that would allow me to replicate (quickly) this same machine.

The "test" I used was, can I build, using this tool, that which is distributed as a "stock" image.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/RPi-Distro/pi-gen/issues/427#issuecomment-698968623, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEZTUHL3HBM7WLFNQD7XKFDSHSTWTANCNFSM4OZI6TMQ .

-- Thanks

Kevin Shumaker

Personal Tech Support https://kevinshumaker.wixsite.com/thethirdlevel

N38° 19' 56.52" W85° 45' 8.56"

Semper Gumby “Don't tell people how to do things. Tell them what to do and let them surprise you with their results.” - G.S. Patton, Gen. USA Ethics are what we do when no one else is looking. Quis custodiet ipsos custodes? “There is no end to the good you can do if you don’t care who gets the credit.” - C Powell You know we're sitting on four million pounds of fuel, one nuclear weapon and a thing that has 270,000 moving parts built by the lowest bidder. Makes you feel good, doesn't it?

[MichaelTiernan]

I guess my view of it would be that the "tool" provided, pi-gen, should be able to, from the start, produce a reliable output. I know that it's my opinion but that's the way I see.

With that said, can you detail what steps you use? It might help all of us.

And thanks or the response/exchange on this.

[starbasessd]

No problem, I'm an opinionated old cuss. :) I'll be off and on this weekend (IRL I have grandkids) and will try to write something up as I get a break from them.

On Sat, Sep 26, 2020 at 9:21 AM Michael Tiernan notifications@github.com wrote:

I guess my view of it would be that the "tool" provided, pi-gen, should be able to, from the start, produce a reliable output. I know that it's my opinion but that's the way I see.

With that said, can you detail what steps you use? It might help all of us.

And thanks or the response/exchange on this.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/RPi-Distro/pi-gen/issues/427#issuecomment-699495041, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEZTUHLFOW3FUPINZXQ4OZTSHXTEZANCNFSM4OZI6TMQ .

-- Thanks

Kevin Shumaker

Personal Tech Support https://kevinshumaker.wixsite.com/thethirdlevel

N38° 19' 56.52" W85° 45' 8.56"

Semper Gumby “Don't tell people how to do things. Tell them what to do and let them surprise you with their results.” - G.S. Patton, Gen. USA Ethics are what we do when no one else is looking. Quis custodiet ipsos custodes? “There is no end to the good you can do if you don’t care who gets the credit.” - C Powell You know we're sitting on four million pounds of fuel, one nuclear weapon and a thing that has 270,000 moving parts built by the lowest bidder. Makes you feel good, doesn't it?

[sensslen]

Well I do kind of have two requirements:

1. I would like to be able to reproduce the exact same image byte by byte at a later point in time (when packages in the repository got updated).
2. I would like to get an output which allows me to fulfill the licensing requirements of all packages installed. (If I do read gpl correctly I do have to provide source code for used software if someone requests it.)

What I actually don‘t care about is how these requirements are fulfilled. I would like pi-gen to be able to fulfill these requirements. I also assume that anyone using raspberry pi in a commercial product would appreciate this too!

[starbasessd]

If you install apt-cacher-ng on your build machine sudo apt install apt-cacher-ng add APT_PROXY=http://127.0.0.1:3142 after your build, you will have the files from your install in the directory /var/cache/apt-cacher-ng It will list packages installed and version name. partial example:: (using 'tree') (sudo apt install tree)

├── debrep │ └── dists │ └── buster │ ├── contrib │ │ ├── binary-i386 │ │ │ ├── by-hash │ │ │ │ └── SHA256 │ │ │ │ ├── 142aa90a34bb5e7fa21528ce76ac85a2fa9453ad1eef8f308f75ad7e322fe88f │ │ │ │ └── 142aa90a34bb5e7fa21528ce76ac85a2fa9453ad1eef8f308f75ad7e322fe88f.head │ │ │ ├── Packages.xz │ │ │ └── Packages.xz.head │ │ ├── i18n │ │ │ ├── by-hash │ │ │ │ └── SHA256 │ │ │ │ ├── 3a737c2da6b268aa8a86427c8c85fef3accee912d4d48428405bd6103185c6d2 │ │ │ │ └── 3a737c2da6b268aa8a86427c8c85fef3accee912d4d48428405bd6103185c6d2.head │ │ │ ├── Translation-en.bz2 │ │ │ └── Translation-en.bz2.head │ │ └── source │ │ ├── by-hash │ │ │ └── SHA256 │ │ │ ├── fddd4901b529b79fd29c0afe0c99b7e330dd62de394fd4f5dab2d14139fa8ee9 │ │ │ └── fddd4901b529b79fd29c0afe0c99b7e330dd62de394fd4f5dab2d14139fa8ee9.head │ │ ├── Sources.xz │ │ └── Sources.xz.head │ ├── InRelease │ ├── InRelease.head │ └── main │ ├── binary-i386 │ │ ├── by-hash │ │ │ └── SHA256 │ │ │ ├── 0da33846fe6e46d6ada7caeee98e346b1c1bfea74144d1581f4988eee6f1f13b │ │ │ └── 0da33846fe6e46d6ada7caeee98e346b1c1bfea74144d1581f4988eee6f1f13b.head │ │ ├── Packages.xz │ │ └── Packages.xz.head │ ├── i18n │ │ ├── by-hash │ │ │ └── SHA256 │ │ │ ├── 4809d181a0d1b0aeff372d55cc94fe14d7255789dcf73644e9849de5dd6b3265 │ │ │ └── 4809d181a0d1b0aeff372d55cc94fe14d7255789dcf73644e9849de5dd6b3265.head │ │ ├── Translation-en.bz2 │ │ └── Translation-en.bz2.head │ └── source │ ├── by-hash │ │ └── SHA256 │ │ ├── 03520237aa4a93c8e453d00c2191257886737fdedf9958ac4a4cf5d652b1d492 │ │ └── 03520237aa4a93c8e453d00c2191257886737fdedf9958ac4a4cf5d652b1d492.head │ ├── Sources.xz │ └── Sources.xz.head ├── _expending_damaged ├── _expending_dat ├── raspbian.raspberrypi.org │ └── raspbian │ ├── dists │ │ └── buster │ │ ├── contrib │ │ │ └── binary-armhf │ │ │ ├── Packages.xz │ │ │ └── Packages.xz.head │ │ ├── InRelease │ │ ├── InRelease.head │ │ ├── main │ │ │ └── binary-armhf │ │ │ ├── Packages.xz │ │ │ └── Packages.xz.head │ │ └── non-free │ │ └── binary-armhf │ │ ├── Packages.xz │ │ └── Packages.xz.head │ └── pool │ └── main │ ├── a │ │ ├── acl │ │ │ ├── libacl1_2.2.53-4_armhf.deb │ │ │ └── libacl1_2.2.53-4_armhf.deb.head │ │ ├── adduser │ │ │ ├── adduser_3.118_all.deb │ │ │ └── adduser_3.118_all.deb.head │ │ ├── apparmor │ │ │ ├── libapparmor1_2.13.2-10_armhf.deb │ │ │ └── libapparmor1_2.13.2-10_armhf.deb.head │ │ ├── apt │ │ │ ├── apt_1.8.2.1_armhf.deb │ │ │ ├── apt_1.8.2.1_armhf.deb.head │ │ │ ├── apt-utils_1.8.2.1_armhf.deb │ │ │ ├── apt-utils_1.8.2.1_armhf.deb.head │ │ │ ├── libapt-inst2.0_1.8.2.1_armhf.deb │ │ │ ├── libapt-inst2.0_1.8.2.1_armhf.deb.head │ │ │ ├── libapt-pkg5.0_1.8.2.1_armhf.deb │ │ │ └── libapt-pkg5.0_1.8.2.1_armhf.deb.head │ │ ├── argon2

Will this meet your requirements?

On Sat, Sep 26, 2020 at 2:22 PM sensslen notifications@github.com wrote:

Well I do kind of have two requirements:

1. I would like to be able to reproduce the exact same image byte by byte at a later point in time (when packages in the repository got updated).
2. I would like to get an output which allows me to fulfill the licensing requirements of all packages installed. (If I do read gpl correctly I do have to provide source code for used software if someone requests it.)

What I actually don‘t care about is how these requirements are fulfilled. I would like pi-gen to be able to fulfill these requirements. I also assume that anyone using raspberry pi in a commercial product would appreciate this too!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/RPi-Distro/pi-gen/issues/427#issuecomment-699530505, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEZTUHJ4CEGXN5F6NLESINDSHYWMXANCNFSM4OZI6TMQ .

-- Thanks

Kevin Shumaker

Personal Tech Support https://kevinshumaker.wixsite.com/thethirdlevel

N38° 19' 56.52" W85° 45' 8.56"

Semper Gumby “Don't tell people how to do things. Tell them what to do and let them surprise you with their results.” - G.S. Patton, Gen. USA Ethics are what we do when no one else is looking. Quis custodiet ipsos custodes? “There is no end to the good you can do if you don’t care who gets the credit.” - C Powell You know we're sitting on four million pounds of fuel, one nuclear weapon and a thing that has 270,000 moving parts built by the lowest bidder. Makes you feel good, doesn't it?

[starbasessd]

Here's my result this afternoon on my buildbuster VM Debian on VBox VM kevin@buildbuster:~$ uname -a Linux buildbuster 5.4.0-4-686-pae #1 SMP Debian 5.4.19-1 (2020-02-13) i686 GNU/Linux pi-gen build.sh with modified config to use apt-cacher-ng apt proxy on 127.0.0.1:3142 (cleaned cache prior to run)

files.txt

Ran cd /var/cache/apt-cacher-ng tree -I *.head >> /home//files.txt (to remove header files listed) Since I archive all my builds (except failed builds) I don't do anything else special. If I wanted, I could tar.xz the folder tree, too, to save the files. I don't do any run-time compiles from source, so don't feel the need to keep any.

Would this meet your needs? As to gpl requirement, I think it's if YOU modify a package source, you need to supply what modification you made, to which version. If you don't modify a package, no need to supply source.

[sensslen]

Sorry for the late reply. This is actually a similar approach that I am using currently. The difference is that I use the prebuilt image provided which I then modify by redirecting the repository to a self hosted one containing exactly the packages avalialble at the point in time the setup was created. To me this approach seems to be a bit hacky.

[XECDesign]

If archive.raspberrypi.org had snapshots, it would be a bit more reasonable, but as is, I can't think of a good way to support this.

One way might be to save archive lists at build time and then copy them next time rather than use apt update. And for raspbian.org, it would need to figure out which snapshot to use. I think this is maybe something to add to a wishlist but is a wontfix for now.

[MichaelTiernan]

I've taken to using a stock image. I pried it open, made some mods then closed it up. I 'dd'ed the result as my image to build from.

Ugly but it gives me that "fixed in time" feeling to work from.