Create deterministic Debian source packages

Currently orig.tar.gz is created at each build, using plain tar. This means that each time it is different, even if no change was made (because of mtimes, files order and such issues). Later, when adding such package to repository (linux-deb), reprepro complains about adding already existing package but with different size/checksum (this happens because orig.tar.gz is added as part of every target release - jessie, wheezy, stretch). This is the reason why no source packages are uploaded to http://deb.qubes-os.org/.

Additionally the command creating orig.tar.gz should be standarized and placed in builder-debian - currently it is duplicated in every component, for example: https://github.com/QubesOS/qubes-core-agent-linux/blob/master/Makefile.builder#L14-L18 https://github.com/QubesOS/qubes-core-qubesdb/blob/master/Makefile.builder#L25-L29

Most of packages (in quilt format) have exacty the same lines. Two of them are slightly different: vmm-xen and app-thunderbird. This is probably easy to handle.

For creating stable tar.gz, git archive --format=tar.gz HEAD should be used - it takes care of mtimes and such problems. The archive will contain only committed content and dpkg-buildpackage will complain about local changes (if any), but that's not a problem - in case of local changes builder-debian may simply skip source package creation as it would be useless anyway (dpkg-buildpackage -b). Or even skip source package creation for every devel build (INCREMENT_DEVEL_VERSION=1).

Packages in native format I believe are properly handled by dpkg-source already (care needs to be taken to not include binary packages in the source archive, but currently is seems to be ok).

Afaik upstream tarball creation is usually is implemented by make dist. For a Debian policy conform and deterministic shell function on how to create a upstream tarball, have a look at this genmkfile function: https://github.com/Whonix/genmkfile/blob/472850c285714a080316778296dda4aefd2b06c6/usr/share/genmkfile/make-helper.bsh#L491-L515 (needs a patch using touch to the same date, that I have ready and need to clean up)

Using git archive to create upstream tarballs is a Debian packaging bug. Because Debian upstream tarballs are supposed to not include git folders. In Debian, by the time the upstream tarballs is uploaded to the build server, the git folder is not available.

For whatever it's worth, Whonix's packages are mostly deterministic, but suffer from being depend build on the same file system. Debian upstream bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=650077

So I might be able to get the the normal files build deterministic. Would be up to you to make the files compile deterministic.

On Wed, Sep 30, 2015 at 10:00:19AM -0700, Patrick Schleizer wrote:

Afaik upstream tarball creation is usually is implemented by make dist. For a Debian policy conform and deterministic shell function on how to create a upstream tarball, have a look at this genmkfile function: https://github.com/Whonix/genmkfile/blob/472850c285714a080316778296dda4aefd2b06c6/usr/share/genmkfile/make-helper.bsh#L491-L515 (needs a patch using touch to the same date, that I have ready and need to clean up)

I don't think that's enough. For example tar file header contains device major:minor of the filesystem from which it was created. git archive handles all of such quirks.

Using git archive to create upstream tarballs is a Debian packaging bug. Because Debian upstream tarballs are supposed to not include git folders. In Debian, by the time the upstream tarballs is uploaded to the build server, the git folder is not available.

git archive doesn't include .git in the output tarball. Here I meant to use git archive to prepare "upstream tarball" before debian package build. Currently it is done using plain tar call (which, as you know, is far from being deterministic).

For whatever it's worth, Whonix's packages are mostly deterministic, but suffer from being depend build on the same file system. Debian upstream bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=650077

So I might be able to get the the normal files build deterministic. Would be up to you to make the files compile deterministic.

Yes, getting deterministic build process it totally different task. This one is prerequisite for it.

Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

I have been working on the Debian packaging over the last few weeks.

I have created a solution that removes the creation of the tars and patching from Makefile.builder and moves it into debian/rules. I moved the code into debian/rules instead of Makefile.debian to allow building package independent of Makefile.builder using targets recommended by Debian standards (get-orig-source and patch).

Currently I created a sub-directory called snippets within builder-debian and install those small Makefile includes the the chroots /usr/share/qubesbuilder directory, but plan on either creating a package that can be included as a build depend for packages that require it, or maybe adding it as a gitrepo (like gittree) and include the requires Makefile snippet in debian/rules, if needed.

Other related items I have completed are:

Removing the need for debian to be placed in debian-vm folder on packages like vmm-xen.
Installation of choot environment using pbuilder or sbuild. I am still working on sbuild over the next few days since I initially needed to build the packages in a Debian AppVM as it sbuild was not available for Fedora. I could package sbuild for Fedora if we find it useful.
Ability to have different binary depends, recommends, etc per DIST or VENDOR (wily, debian) for any package using a packagename.depends configuration file within the debian directory
Ability to apply patches based of DIST or VENDOR (wily, debian)
Added buildinfo to each build.
Experimented with using CDBS, Common Debian Build Systemwhich I used forcore-agent-linuxandvmm-xen` but decided it is not as flexible and transparent as I would like.
Qubuntu refresh almost complete

I will have four qubes repos ready for inspection by end of weekend (builder-debian, vmm-xen, core-agent-linux and giu-agent-linux) for more feed back (currently in qubuntu feature branches). Currently those feature branches have some CDBS and pbuilder code in them I need to rip back out.

Sample of get-orig-source snippet:

DEB_ORIG_TGZ = $(tarball_dir)/$(DEB_SOURCE)_$(DEB_VERSION_UPSTREAM).orig.tar.gz  

### Create Debian <package>_<version>.orig.tar.gz file                           
$(DEB_ORIG_TGZ):                                                                 
        @$(shell rm -f $@)                                                       
        @$(shell tar cfz $@ --exclude-vcs --exclude=pkgs --exclude=debian .)     
#        @$(shell git archive HEAD --output=$@)                                   

get-orig-source: $(DEB_ORIG_TGZ)                                                 
        @true

Sample of patch snippet:

# Top source directory to apply patches in                                       
QUILT_TOPDIR ?= $(DEB_SRCDIR)                                                    

QUILT_DIR ?= $(debdir)/patches                                                   
QUILT_SERIES ?= series                                                           
QUILT_SERIES_PATH = $(QUILT_DIR)/$(QUILT_SERIES)                                 

# Master series patch file (if it exists) will be used to copy patches contained 
# in its series file to $QUILT_DIR and will append its series entry with the     
# new patches.                                                                   
#                                                                                
# This allows patches to be shared amongst other packaging or allow different    
# sets of patches for both Debian, Ubuntu, etc while making sure the patches are 
# in place to allow creation of a proper orig.tar.gz.                            
MASTER_QUILT_SERIES ?= \                                                         
        $(wildcard \                                                             
            series-$(DEB_DISTRIBUTION)-$(packageset).conf \                      
            series-$(call LOWERCASE, $(DEB_VENDOR))-$(packageset).conf \         
)                                                                                

# Path to stamp file to use                                                      
DEB_STAMP ?= debian/stamp-patched                                                

### Move any patches to debian/patches
$(QUILT_SERIES_PATH):                                                            
        @# Clear patch series.conf file to prepare for update                    
        @$(shell rm -f $(QUILT_SERIES_PATH))                                     
        @$(shell touch $(QUILT_SERIES_PATH))                                     

        @# Move patches to debian/patches directory                              
        @while read patch_file; do \                                             
            echo "I: $(red)patch_file: $(blue)$(packagedir)/$${patch_file}$(normal)"; \
            if [ -e "$(packagedir)/$${patch_file}" ]; then \                     
                echo "$${patch_file##*/}" >> "$(QUILT_SERIES_PATH)"; \           
                cp "$(packagedir)/$${patch_file}" "$(QUILT_DIR)"; \              
            fi \                                                                 
        done < $(firstword $(MASTER_QUILT_SERIES))                               

$(MASTER_QUILT_SERIES): $(QUILT_SERIES_PATH)                                     
         @true

prepare-patches: $(MASTER_QUILT_SERIES)                                          
        @true

patch: prepare-patches $(DEB_STAMP)                                              
$(DEB_STAMP):                                                                    
        @cd $(QUILT_TOPDIR) && \                                                 
        QUILT_SERIES=$(QUILT_SERIES) QUILT_PATCHES=$(QUILT_DIR) \                
                quilt --quiltrc /dev/null push -a || test $$? = 2                
        @touch $(DEB_STAMP)                                                      

unpatch:                                                                         
        @cd $(QUILT_TOPDIR) && \                                                 
        QUILT_SERIES=$(QUILT_SERIES) QUILT_PATCHES=$(QUILT_DIR) \                
                quilt --quiltrc /dev/null pop -a -R || test $$? = 2              
        @rm -rf .pc $(DEB_STAMP)                                                 

.PHONY: patch unpatch

Ability to have different binary depends, recommends, etc per DIST or VENDOR (wily, debian) for any package using a packagename.depends configuration file within the debian directory

Try to make it as easy as possible to build in non-qubes-builder. Ideally if the source package could be build any standard Debian upstream build tool without any Qubes-specific tools for that. For example:

apt-get source qubes-core-agent
apt-get build-dep qubes-core-agent
cd qubes-core-agent*/
dpkg-buildpackage

If that means changing something in our packaging files, that's fine, lets do it (but keep builder also compatible with old packages - so it will be possible to build for example R3.0 packages).

If that means some additional build dependency with those snippets, it may be an option, but preferably package should be buildable using standard environment. Having some custom things done by qubes-builder (like currently orig.tar.gz...) is no-go.

Be as close to the upstream solution (package format and build process) as possible.

### Create Debian <package>_<version>.orig.tar.gz file                           
$(DEB_ORIG_TGZ):                                                                 
        @$(shell rm -f $@)                                                       
        @$(shell tar cfz $@ --exclude-vcs --exclude=pkgs --exclude=debian .)

This is not deterministic, take a look here: https://reproducible-builds.org/docs/archives/

Also note that z option (gzip) makes it non-deterministic because of timestamp included. AFAIR there is some option to skip it - search linked page.

or ...

# @$(shell git archive HEAD --output=$@)

... use this one.

On 28 January 2016 at 17:50, Marek Marczykowski-Górecki < notifications@github.com> wrote:

Ability to have different binary depends, recommends, etc per DIST or VENDOR (wily, debian) for any package using a packagename.depends configuration file within the debian directory

Try to make it as easy as possible to build in non-qubes-builder. Ideally if the source package could be build any standard Debian upstream build tool without any Qubes-specific tools for that. For example:

apt-get source qubes-core-agent apt-get build-dep qubes-core-agent cd qubes-core-agent*/ dpkg-buildpackage

If that means changing something in our packaging files, that's fine, lets do it (but keep builder also compatible with old packages - so it will be possible to build for example R3.0 packages).

If that means some additional build dependency with those snippets, it may be an option, but preferably package should be buildable using standard environment. Having some custom things done by qubes-builder (like currently orig.tar.gz...) is no-go.

I agree. You will see the four packages I refactored will be able to built in that manner since I am actually working on getting them to build with buildd as well. At the same time I am also being careful that they will still build for 3.0 and making sure stuff builds in a Debian environment (I can build all the components, just can't create the template rpm yet).

I think if you don't want to have to depend on a yet to be created qubes-builder package that would install the snippets onto the chroot that I should consider using git-subrepo ( https://github.com/ingydotnet/git-subrepo) which would allow updating the snippets without using submodules. I want to stay away from submodules since there are issues using them with Debian packaging using some of the git package tools (I got an error saying git submodules are not yet supported).

Be as close to the upstream solution (package format and build process) as possible.

Create Debian _.orig.tar.gz file

$(DEB_ORIG_TGZ): @$(shell rm -f $@) @$(shell tar cfz $@ --exclude-vcs --exclude=pkgs --exclude=debian .)

This is not deterministic, take a look here: https://reproducible-builds.org/docs/archives/

Also note that z option (gzip) makes it non-deterministic because of timestamp included. AFAIR there is some option to skip it - search linked page.

I have code to create a tar that is deterministic, just have not implemented it yet (https://wiki.debian.org/onlyjob/get-orig-source). I am also going to change get-orig-source to do just that and create the .orig using the dist target.

I am starting to also use dh_buildinfo: Register the versions of build-dependencies used to build a package. dh_buildinfo is a debhelper program that registers in a file the list of packages declared as build-time dependencies, as well as build-essential packages, together with their versions, as installed in the build machine.

As well as --list-missing which looks for files that are not in package but built, similar to what happens at the end of building an rpm.

Finally, I also noticed none of the source packages are being signed. I will fix that as well look into using dput to move packages to repo.

Oh, one last thing in regards to using native Debian packages like some of the qubes packages do: 5.3. Native Debian package https://www.debian.org/doc/manuals/debmake-doc/ch05.en.html

The non-native Debian package in the “3.0 (quilt)” format is the most normal Debian binary package format. The above workflow and the following packaging examples always use this format.

The native Debian package is the rare Debian binary package format. It may be used only when the package is useful and valuable only for Debian. Thus, its use is generally deprecated.

or ...

@$(shell git archive HEAD --output=$@)

... use this one.

— Reply to this email directly or view it on GitHub https://github.com/QubesOS/qubes-issues/issues/1244#issuecomment-176466241 .

I think if you don't want to have to depend on a yet to be created qubes-builder package that would install the snippets onto the chroot that I should consider using git-subrepo ( https://github.com/ingydotnet/git-subrepo) which would allow updating the snippets without using submodules.

If some additional snippets are really required, build-depends on them is ok. But make sure that builder package itself can be built without them (to not introduce dependency cycle)... And generally if some package can be built without such additions, do not depend on that package.

QubesOS / qubes-issues

Create deterministic Debian source packages #1244

Create Debian _.orig.tar.gz file

@$(shell git archive HEAD --output=$@)