PPA repo replacement for Debian

[Firefishy]

Create a replacement for our Ubuntu PPA build process for custom packages.

Requirements:

• Adhoc builder for packages (ARM64 & AMD64)
• Debian Repository for packages (Debian 12, future stable releases)

Forked from: https://github.com/openstreetmap/operations/issues/844

[Firefishy]

https://packagecloud.io/ is only repo hosting, a separate builder is required.

[mmd-osm]

Today we have around 6 packages with OSM focus and 5-6 others which I don't really know what they're good for (seems to be some system mgmt tools): https://launchpad.net/~osmadmins/+archive/ubuntu/ppa?field.series_filter=jammy

That's covering roughly half of the OSM projects used in production, the other half has dedicated build instructions in Chef. Do we plan to change this ratio somehow, and either build everything in Chef, or use packages everywhere?

Out of these 6 packages, I believe osm2pgsql, libosmium, libapache2-mod-tile and pyosmium are already available in Debian. Are these recent enough for our purposes, or do we need our own version here?

Regarding adhoc builder for packages, osmdbt already has some relevant config. We're discussing to move CGImap to cmake, and maybe use cpack to generate .deb packages. -> already in place now!

Maybe we could try to convince https://wiki.debian.org/DebianGis that osmdbt and cgimap would be a good fit for them as well, and don't run a Debian repository on our own.

What's the plan for ARM64 like? Is that only relevant for tile rendering, or do we expect other apps to move there as well?

[mmd-osm]

https://github.com/openstreetmap/chef/pull/659 made me curious what the current plan for 2024 is.

If we can keep using Ubuntu launchpad for PPA generation, I will put our cpack activities on low priority for now.

[tomhughes]

I think we're still planning to move towards Debian but existing machines, at least in the short term, are likely to be upgraded instead hence that PR.

[Firefishy]

As discussed with Operations team I will produce a proof of concept "PPA" GitHub action builder. The repo hosting is to be defined. ARM64 building will be backed by a Raspberry PI 5 cluster until Public Open Source GitHub Action Runners are available late 2024 / early 2025.

[grischard]

For whatever it's worth, the Editor Layer Index happily runs in production on github pages.

[Firefishy]

We now have aptly running a repo at https://apt.openstreetmap.org/ Thanks @tomhughes Does not include a method for building packages. See below.

I have started testing brief testing of GitHub Action builder methods:

• Using debootstrap is tricky on GitHub Actions as it requires sudo or fakeroot. action/cache backup / restore runs as default action user and breaks if file permissions do not allow.
• https://github.com/marketplace/actions/build-debian-packages looks like a nice solution (docker) and is wired directly into github repo. Uploader to apt.openstreetmap.org still needs investigation. Unsure as yet of Ubuntu compatibility.

[tomhughes]

Basically options for uploading are...

• Enable the aptly API which require some sort of layer in front to do security, then you can upload files with POST /api/files/:dir and add them to a repository with POST /api/repos/:name/file/:dir
• Implement an https/ftp/sftp/scp/rsync upload endpoint that dput can use and then use aptly repo include to import packages from that upload queue which has the advantage that it will ignore packages that aren't signed by an authorised uploader

[grischard]

A third option is running apt.openstreetmap.org on github pages.

[tomhughes]

Ignoring the fact that we've already implemented something else, just HTF is that supposed to work?

[grischard]

HTF RTFM: https://docs.github.com/en/pages/configuring-a-custom-domain-for-your-github-pages-site/about-custom-domains-and-github-pages

[tomhughes]

Yes I know how to point a domain name at gihub pages.

What I don't know is how to make it generate metadata for an apt repository.

[grischard]

The github action that Grant mentioned above looks perfect for it:

github.com/marketplace/actions/build-debian-packages looks like a nice solution (docker) and is wired directly into github repo. Uploader to apt.openstreetmap.org still needs investigation. Unsure as yet of Ubuntu compatibility.

[tomhughes]

It builds the packages sure. As far as I can see it does nothing about adding those packages to a pool tree or building a dists tree with Release and Packages files describing the available packages.

[Firefishy]

• openstreetmap-cgimap now builds debian packages as part of release process: https://github.com/zerebubuth/openstreetmap-cgimap/releases/tag/v2.0.1

[Firefishy]

osmdbt deb PR https://github.com/openstreetmap/osmdbt/pull/41

[mmd-osm]

@Firefishy : regarding mod_tile, can't we use the Debian GIS repo instead? https://salsa.debian.org/debian-gis-team/libapache2-mod-tile

[Firefishy]

@mmd-osm OSM run some custom patches on top of the standard releases (osm2pgsql and mod_tile). The current example is size/length limits which help mitigate vandalism overloading our servers.

[mmd-osm]

https://salsa.debian.org/debian-gis-team/libapache2-mod-tile/-/tree/master/debian?ref_type=heads has all needed Debian config in place, which might serve as a starting point. There’s even a patches directory where you could add your changes (assuming this all is copied over to some other place to build packages for apt.osm specifically).

None of this is currently available in the original mod_tile repo.

[santiagorr]

Debian contributor here. Glad to read OSM has moved to Debian! :beers:

In case that helps for inspiration, my approach to have extra/preview repositories is to rely on salsa CI and its aptly job: https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built-apt-repository.

Currently, Salsa CI is built on gitlab only, but you could adapt the aptly job for github if your have your packaging here.