flux-framework / flux-core

core services for the Flux resource management framework
GNU Lesser General Public License v3.0
167 stars 50 forks source link

package flux-core for ubuntu #2585

Open garlick opened 4 years ago

garlick commented 4 years ago

It would be useful to have regular builds of flux-core available in an Ubuntu PPA for

I have a rather dumb (single) deb built following this process that can be installed and superficially works. I'm not sure I'll have time to get all the way to committing something to a PPA repo right away, so wanted to checkpoint notes here.

After creating a dist tarball of flux-core with make dist and a new scratch directory for creating the deb called flux-core-deb:

$ sudo apt-get install dh-make bzr-builddeb
$ bzr whoami "Jim Garlick <garlick.jim@gmail.com>"
$ cd flux-core-deb
$ cp ../flux-core/flux-core-0.13.0-419-gf9780be36.tar.gz .
$ bzr dh-make flux-core 0.13.0-419-gf9780be36 flux-core-0.13.0-419-gf9780be36.tar.gz
#    Note: answer prompt for "single executable" package
$ cd flux-core/debian
$ rm -f *ex *EX README*
$ vi changelog  control copyright flux-core-docs.docs rules
#    Note: see examples below
$ bzr add debian/source/format
$ bzr commit -m "initial commit of debian packaging"
$ bzr builddeb -- -us -uc

changelog

flux-core (0.13.0-419-gf9780be36-0ubuntu1) bionic; urgency=medium

  * Initial release

 -- Jim Garlick <garlick@gmail.com>  Fri, 27 Dec 2019 07:53:59 -0800

control

Source: flux-core
Section: devel
Priority: optional
Maintainer: Jim Garlick <garlick.jim@gmail.com>
Build-Depends:
 debhelper (>= 10),
 autotools-dev,
 libsodium-dev,
 libzmq3-dev,
 libczmq-dev,
 libjansson-dev,
 liblz4-dev,
 libhwloc-dev,
 libsqlite3-dev,
 lua5.1,
 liblua5.1-dev,
 lua-posix,
 python3-dev,
 python3-cffi,
 python3-six,
 python3-yaml,
 python3-jsonschema
Standards-Version: 4.1.2
Homepage: http://github.com/flux-framework/flux-core

Package: flux-core
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: Core support for the Flux reource manager framework
 flux-core implements the communication layer and lowest level
 services and interfaces for the Flux resource manager framework.
 It consists of a distributed message broker, plug-in modules
 that implement various distributed services, and an API and set
 of utilities to utilize these services.

rules

#!/usr/bin/make -f

%:
    dh $@

override_dh_auto_configure:
    PYTHON_VERSION=3 dh_auto_configure -- --with-systemdsystemunitdir=/lib/systemd/system

override_dh_autoreconf:
    @echo not running autogen.sh on dist product

copyright N.B. needs to include copyright/license for vendored libs

Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: flux-core
Source: <http://github.com/flux-framework/flux-core>

Files: *
Copyright: 2014-2019 Lawrence Livermore National Security
License: LGPL-3.0

Lintian complaints not yet addressed:

W: flux-core: package-name-doesnt-match-sonames libflux-core2 libflux-idset1 libflux-optparse1 libflux-schedutil1
E: flux-core: binary-or-shlib-defines-rpath usr/lib/python3.6/site-packages/_flux/_core.so /usr/lib
E: flux-core: binary-or-shlib-defines-rpath usr/lib/x86_64-linux-gnu/flux/modules/pymod.so /usr/lib
W: flux-core: python-module-in-wrong-location usr/lib/python3.6/site-packages/_flux/ usr/lib/python3/dist-packages/_flux/
W: flux-core: python-module-in-wrong-location usr/lib/python3.6/site-packages/_flux/__init__.py usr/lib/python3/dist-packages/_flux/__init__.py
E: flux-core: package-installs-python-pycache-dir usr/lib/python3.6/site-packages/_flux/__pycache__/
W: flux-core: python-module-in-wrong-location usr/lib/python3.6/site-packages/_flux/__pycache__/ usr/lib/python3/dist-packages/_flux/__pycache__/
W: flux-core: python-module-in-wrong-location ... use --no-tag-display-limit to see all (or pipe to a file/program)
E: flux-core: package-installs-python-pycache-dir usr/lib/python3.6/site-packages/flux/__pycache__/
E: flux-core: package-installs-python-pycache-dir usr/lib/python3.6/site-packages/flux/core/__pycache__/
E: flux-core: package-installs-python-pycache-dir ... use --no-tag-display-limit to see all (or pipe to a file/program)
E: flux-core: python-package-missing-depends-on-python
E: flux-core: missing-dep-for-interpreter lua => lua | lua40 | lua50 | lua5.1 | lua5.2 (usr/lib/x86_64-linux-gnu/flux/cmd/flux-cron) #!lua
W: flux-core: executable-not-elf-or-script usr/share/lua/5.1/flux/alt_getopt.lua
W: flux-core: executable-not-elf-or-script etc/flux/shell/lua.d/openmpi.lua
W: flux-core: executable-not-elf-or-script usr/share/lua/5.1/fluxometer/Test/More.lua
W: flux-core: executable-not-elf-or-script ... use --no-tag-display-limit to see all (or pipe to a file/program)
E: flux-core: non-empty-dependency_libs-in-la-file usr/lib/lua/5.1/flux.la
E: flux-core: non-empty-dependency_libs-in-la-file usr/lib/python3.6/site-packages/_flux/_core.la
E: flux-core: non-empty-dependency_libs-in-la-file usr/lib/x86_64-linux-gnu/flux/connectors/local.la
E: flux-core: non-empty-dependency_libs-in-la-file ... use --no-tag-display-limit to see all (or pipe to a file/program)
W: flux-core: non-dev-pkg-with-shlib-symlink usr/lib/x86_64-linux-gnu/libflux-schedutil.so.1.0.0 usr/lib/x86_64-linux-gnu/libflux-schedutil.so
W: flux-core: non-dev-pkg-with-shlib-symlink usr/lib/x86_64-linux-gnu/libflux-idset.so.1.0.0 usr/lib/x86_64-linux-gnu/libflux-idset.so
W: flux-core: non-dev-pkg-with-shlib-symlink usr/lib/x86_64-linux-gnu/libflux-core.so.2.0.0 usr/lib/x86_64-linux-gnu/libflux-core.so
W: flux-core: non-dev-pkg-with-shlib-symlink ... use --no-tag-display-limit to see all (or pipe to a file/program)
stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had activity for 365 days. It will be closed if no further activity occurs within 14 days. Thank you for your contributions.

garlick commented 1 year ago

As suggested offline by @grondo, maybe a simple make deb rule in our build system to essentially package a "scratch build" would be handy for those of us running little test clusters. You'd think I'd remember how to do this in my sleep after this project and in fact there might be some usable hints in there, but I found this brief article to be a helpful review.

grondo commented 1 year ago

I actually started with flux-security. I came up with the following script based approach for rolling one-off debian packages. Note that the dh-make command for bzr seems to have been removed upstream, so the script just creates most of the files by hand. I'm not sure this script is that useful, except as self contained instructions for building a bare-minimum .deb from a build tree. Actually packaging for debian or ubuntu would be a lot more work.

#!/bin/sh
PACKAGE=flux-security
USER=$(git config --get user.name)
DEBFULLNAME=$USER
EMAIL=$(git config --get user.email)
DEBEMAIL=$EMAIL

die() { echo "debbuild: $@" >&2; exit 1; }
log() { echo "debbuild: $@"; }

test -z "$USER" && die "User name not set in git-config"
test -z "$EMAIL" && die "User email not set in git-config"

log "Running make dist"
make dist >/dev/null || exit 1

tarball=$(ls -tr *.tar.gz | tail -1)
version=$(echo $tarball | sed "s/${PACKAGE}-\(.*\)\.tar\.gz/\1/")

rm -rf build
mkdir -p build  && cd build

mv ../$tarball .

log "Unpacking $tarball"
tar xvfz $tarball >/dev/null

log "Creating debian directory and files"
cd ${PACKAGE}-${version}
mkdir -p debian/source

export DEBEMAIL DEBFULLNAME
log "Creating debian/changelog"
dch --create --package=$PACKAGE --newversion $version build tree release

cat <<EOF >debian/source/format
3.0 (quilt)
EOF

log "Creating debian/control"
cat <<EOF >debian/control
Source: ${PACKAGE}
Section: devel
Priority: optional
Maintainer: ${USER} <${EMAIL}>
Standards-Version: 4.1.2
Build-Depends:
  debhelper (>= 10),
  libsodium-dev,
  libjansson-dev,
  uuid-dev,
  libmunge-dev

Homepage: https://github.com/flux-framework/flux-security
Package: flux-security
Architecture: any
Depends: \${shlibs:Depends}, \${misc:Depends}
Description: Independent project for Flux security code and APIs.
 flux-security implements Flux security code and APIs, including the
 privileged IMP executable.
EOF

log "Creating debian/rules"
cat <<EOF >debian/rules
#!/usr/bin/make -f
# See debhelper(7) (uncomment to enable)
# output every command that modifies files on the build system.
#export DH_VERBOSE = 1

# see FEATURE AREAS in dpkg-buildflags(1)
#export DEB_BUILD_MAINT_OPTIONS = hardening=+all

# see ENVIRONMENT in dpkg-buildflags(1)
# package maintainers to append CFLAGS
#export DEB_CFLAGS_MAINT_APPEND  = -Wall -pedantic
# package maintainers to append LDFLAGS
#export DEB_LDFLAGS_MAINT_APPEND = -Wl,--as-needed

%:
    dh \$@

override_dh_autoreconf:
    @echo not running autogen.sh on dist product

override_dh_auto_install:
    dh_auto_install
    find . -name '*.la' -delete

override_dh_fixperms:
    dh_fixperms --exclude flux-imp
    find . -name flux-imp -exec chmod 4755 {} \;
EOF
chmod +x debian/rules

log "Creating debian/copyright"
cat <<EOF >debian/copyright
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: flux-security
Source: <https://github.com/flux-framework/flux-security>

Files: *
Copyright: 2014-2023 Lawrence Livermore National Security
License: LGPL-3+

Files: src/libtomlc99/*
Copyright: 2017 CK Tan

Copyright (c) 2017 CK Tan
https://github.com/cktan/tomlc99

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Files: debian/*
Copyright: 2023 Mark A. Grondona <grondo@unknown>
License: GPL-2+
 This package is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation; either version 2 of the License, or
 (at your option) any later version.
 .
 This package is distributed in the hope that it will be useful,
 but WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.
 .
 You should have received a copy of the GNU General Public License
 along with this program. If not, see <https://www.gnu.org/licenses/>
 .
 On Debian systems, the complete text of the GNU General
 Public License version 2 can be found in "/usr/share/common-licenses/GPL-2".

# packaged work, as it may make Debian's contributions unacceptable upstream.
EOF

log "Creating debian/compat"
cat <<EOF >debian/compat
10
EOF

log "Creating debian/${PACKAGE}-docs.docs"
cat <<EOF >debian/${PACKAGE}-docs.docs
README.Debian
EOF
touch debian/README.source
touch debian/README.Debian

log "Running debian-buildpackage -b"
dpkg-buildpackage -b

Just run from the top level directory of the flux-security repo. I'll create a similar script for flux-core.

I'm not sure if long-term the best plan would be to commit the contents of the debian directory for these projects to a repo somewhere or if there is some other standard way for maintaining this stuff.