Buildable Source Package Support

wbraswell commented 2 years ago

Soon fpm will support unbuildable SRPMs: https://github.com/jordansissel/fpm/issues/237 https://github.com/jordansissel/fpm/pull/1657 https://github.com/jordansissel/fpm/pull/1952

What we really need is support for buildable SRPMs: https://github.com/jordansissel/fpm/issues/237#issuecomment-415941828 https://github.com/bernd/fpm-cookery/issues/8#issuecomment-1302654409

This is a new issue created for adding buildable SRPM support, per request of @jordansissel: https://github.com/jordansissel/fpm/pull/1657#issuecomment-1303911534

NicholasBHubbard commented 2 years ago

If we were able to merge in the fpm-cookery "recipe" functionality, then we could use FPM to build any kind of source package, not just SRPM's.

jordansissel commented 2 years ago

From the parent issue:

An SRPM with an empty %build can not be built, and thus defeats the primary purpose of an SRPM existence.

these can still be built with rpmbuild, though, and empty %build sections is how I’ve used rpm specs for many years even prior to fpm. I do understand this may seem an unusual use, but the tooling didn’t complain (rpmbuild, mrepo, etc).

Can you describe your workflow and what tools are used? I’ll share one case from myself:

My first job where I used CentOS (~2007ish), we built rpm’s almost entirely with empty %build sections. This deployed dozens of applications to production systems and the workflow was mostly, as I recall:

rpmbuild -bb on a .spec file to build a binary rpm. We configured it such that the spec only had a %files section and flags were set on rpmbuild to teach it where to get the files, similar to how fpm does it today.
Copy the resulting rpm to the yum repository and have mrepo do its work to
Install the rpms on servers with Puppet’s package provider fetching the package with the usual tooling from the central company repo.

For my scenario, I think the build steps (what rpm might call %prep, %build, etc) were run before the rpmbuild process. IIRC, this allowed us more freedom to use the developer’s build tooling and was a little easier to integrate than trying to replicate the build tooling/process in the rpm spec.

wbraswell commented 2 years ago

We have our own internal Perl building tools, which have not yet been publicly released on CPAN etc. Our goal is to generate automatically-buildable source packages via properly-populated %build data. We do not want source packages with empty %build data.

My current understanding is that we simply need the ability to utilize "recipes" from fpm-cookery in order to correctly populate %build. Do you have any reason to be opposed to using fpm-cookery as the proposed solution?

jordansissel commented 2 years ago

Ahh this helps. Thanks! And, I'm not opposed -- I'm trying to think how this might fit into fpm's workflow, internals, and use cases.

As for integration with fpm-cookery (or a similar recipe system approach), at this time, I'm not quite sure how it would produce a result you wanted. In my mental model of fpm and how it works, it's unclear to me what the path towards this solution would look like, but such things can still be imaginable and possible ;)

The proposal in https://github.com/bernd/fpm-cookery/issues/8 would have likely treated fpm-cookery recipes in fpm as an input type. In fpm, the "build" steps are part of the input phase, and the only expectation from fpm's internals is two things:

the input phase put some files (or no files) in the staging_path
the input phase should note any package metadata like the package name, dependencies, etc.

After input is done, fpm internally has no record of the steps taken to achieve the resulting files+metadata. Further, some of these steps aren't even shell executions -- sometimes they're pure Ruby which may not translate.

In the fpm-cookery integration issue (fpm-cookery#8), the actual steps performed by fpm-cookery (if integrated in fpm as --input-type recipe) would be forgotten, so during fpm's output phase, producing a source rpm would have no way to know what shell commands should be executed, and thus no knowledge of what script should be put into the %build, %prep, %clean, etc steps for the SPEC file.

Additionally, the proposal in #1526 is a fairly minimal change that tells rpmbuild to emit an srpm while it builds the binary rpm at the same time. The major use case I expect for #1526 is for folks who have internal policy requirements to use srpm and would basically be working around a policy this way -- that is, maybe their software production pipeline uses mock which requires an srpm. FPM's goal in this is to make things easier.

Using fpm-cookery with fpm to produce a SPEC file with a %build/%prep that matches what fpm-cookery executes? I'm not sure what changes would be required for that.

What of non %build steps or features like rpm's BuildRequires (or Debian's Build-Depends)? I imagine there are many modules on CPAN that require external tooling during the perl Makefile.PL step, and might require a compiler, for example. Ruby's nokogiri is historically infamous for this where part of installing nokogiri compiles libxml2. In these scenarios, for an RPM SPEC, you'd add BuildRequires settings to indicate that you need a C/C++ compiler and whatever other dependencies. Debian, Arch, and pretty much all other package building systems have a similar concept, and unfortunately nobody names their packages or dependencies the same, so there's some functionality that we'll likely miss even if we can generate an rpm spec with %build/%prep/etc steps. That might be ok, but I don't know, and would love to figure out what gaps are OK and what would need solving for a solution to be accepted.

wbraswell commented 2 years ago

@jordansissel Thanks for the informative reply, I will review with @NicholasBHubbard and we will get back to you soon.

NicholasBHubbard commented 2 years ago

fpm-cookery does not support outputting source based packages, however if we were to add a similar recipe feature to FPM then we could support any source based package format.

For any source based target a --recipe flag would be required.

fpm -s cpan -t srpm --recipe ./app-foo-recipe.rb App::Foo

The recipe file should allow you to specify everything that an SRPM spec file and Debian DSC file allow you to specify. The other source based package formats I have seen (Gentoo and Slackware) provide the same functionality as SRPM's and Source Deb's, just with a different flavor. I believe that we can encapsulate the general information/behavior of source based packages into a class such as FPM::Package::Source that all concrete source based packages could inherit from. This would allow FPM to be extended to support any kind of source based package.

We could use the same strategy that fpm-cookery uses, where recipe files are just Ruby code that define a class that inherits from a recipe base class. We should try to use the exact same syntax and vocabulary that fpm-cookery uses so that it is easier to use its ideas, and so users of fpm-cookery can seemlessly use FPM to build source packages.

It is important to point out that some modifications to existing code will need to be made. One example is that some package types input method actually builds the software (such as in cpan.rb), which is not appropriate when creating a source based package. Such input methods will need to be modified to check that a source based package is not being targeted before performing the build. Surely there will be other edge cases that need to be addressed as well.

Does this sound reasonable to you @jordansissel? If so I would be eager to start trying to solve the problem.

wbraswell commented 2 years ago

Thanks for the great work as usual, @NicholasBHubbard !

jordansissel commented 2 years ago

I'm still struggling to understand what the implementation would look like. I understand the goals, or at least, I think I do, to have some general-purpose recipe which describes "how to build" or, roughly, an equivalent to rpm's %build, %prep, etc, including BuildRequires and other information used by rpmbuild -- and similar for Debian's tooling.

If we're limiting out scope to only fpm and fpm-cookery as possible tools to use, alone or together, then fpm-cookery alone might be the right place, but it's not without adding more code.

In the text below, I'll use rpm concepts like %build and %install to refer to the general idea of a packaging compile/installation step as you might see in an rpm spec file, and that Debian and other platforms have identical concepts with different names, I'm not focusing specifically on rpm.

Build and Install steps

fpm-cookery can provide "build" steps like in this redis example, but for the most part, the build steps are not shell, they're ruby.

Looking at the code for the redis example:

    make :install, 'PREFIX' => destdir / 'usr'

    var('lib/redis').mkdir

    %w(run log/redis).each {|p| var(p).mkdir }

    bin.install ['src/redis-server', 'src/redis-cli']

    etc('redis').install 'redis.conf'
    etc('init.d').install workdir('redis-server.init.d') => 'redis-server'

Only one expression, I think, causes a separate process to be invoked, make which runs the make command. The rest of the expressions are implemented in ruby, not shell, so producing a %build and %install step for this recipe would require changes to fpm-cookery to print shell command equivalents to the Ruby parts which do file manipulation (mkdir, file copying, etc).

Build-time dependencies

fpm-cookery has a concept of build dependencies, and it installs them using Puppet as invoked directly from Ruby. It also uses fixed strings for the build dependencies, but it requires recipe authors to specify the correct dependency names for each platform because Red Hat and Debian (and its derivatives) use different names for the same things, such as "libffi-dev" in Ubuntu and "libffi-devel" in Fedora. An example of this is found in the fpm-cookery ruby recipe

Language-specific packages

I haven't checked deeply, but it's worth noting that fpm-cookery invokes fpm for a large number of tasks, such as packaging things from cpan, npm, and rubygems. When this happens, fpm-cookery doesn't have much knowledge of /how/ the package is built or installed, so it wouldn't necessarily be able to generate an appropriate %build or %install step in shell.

As mentioned previously, even fpm doesn't keep an internal record of the steps required -- it doesn't intrinsically know the differences between any of the following:

searching metacpan
downloading and extracting the correct perl module (currently handled by cpanm in fpm)
ensuring any dependencies are available
Performing any %build, %install, and %clean steps for the given module.
Doing tricks like searching for shared objects to automatically set the architecture to 'native'

The first 3 steps would be necessary to gather the package metadata (name, version, etc) and for fetching the source file(s). fpm doesn't expose these steps, nor does it have an API for representing these steps.

The 4th item (build, install, clean) are a mixture of ruby and system calls, and there's no mechanism (yet) to expose these steps as with category labels like prep, build, install, clean, etc.

The final item often requires installing the package itself to make that determination (as in cpan), but sometimes fpm discovers this by checking the package metadata (as in gem).

In fpm's gem support, the metadata and download steps are done separate from the prep/build/install/clean steps, but even the install step isn't just "gem install" as fpm will do extra steps patching files which is written in Ruby and would need to be converted to shell.

Unless we don't require shell scripts for this. It could be pure-ruby, I suppose. Nothing preventing an rpm spec from having something like the following:

%install
ruby -e <<HEREDOC
# Do stuff in ruby
HEREDOC

All that said, I agree in order to make this work some possibly significant structural changes would be required. It feels like fpm would need to be taught a few new tricks as well as changing internally how some of these steps are executed. Copying files in Ruby would need shell equivalents (and, if not shell, whatever equivalent step syntax in any supported source package format, if we are thinking that far ahead).

jordansissel commented 2 years ago

If y'all want to try a minimal prototype that would demonstrate fpm+fpm-cookery working together to produce an srpm, that could be a good first step.

Short of a prototype, we could also talk through the changes required and walk through those together.

wbraswell commented 1 year ago

@jordansissel Thanks for the informative reply as usual! @NicholasBHubbard and I will discuss and get back to you soon.

NicholasBHubbard commented 1 year ago

I have created a prototype (#1963) that is meant to show how we could produce source-based packages. This prototype is somewhat hacked together with the only goal being to produce a real SRPM.

I overestimated the value of fpm-cookery for this problem, and it will not be relevant going forward.

I created a subclass of FPM::Package called FPM::SourcePackage that concrete source-based packages can inherit from. I then created a FPM::Recipe class that produces a generic Recipe object by parsing an inputted recipe file. To initialize a FPM::SourcePackage you must pass a file path to a recipe file, that is parsed into a FPM::Recipe object which is an instance variable of the FPM::SourcePackage class. This recipe object can be used by concrete source-based packages to write a spec file.

The recipe syntax I used is a simple tag based syntax where you create section tags such as [install], [build],[download_url], etc. We can extend this syntax to support any kind of feature necessary for producing source-based packages.

Do you think that this strategy will work for allowing FPM to be capable of producing any kind of source-based package? Are there any major problems you see?

jordansissel commented 1 year ago

Did some thinking beyond my comment on #1963 yesterday, and here's an idea I came up with - https://github.com/jordansissel/fpm/pull/1967

The interface should feel familiar to anyone already using fpm:

% fpm -s cpan -t srpm --no-cpan-test File::Temp          
Created package {:path=>"perl-File-Temp-0.2311-1.src.rpm"}

The internals of this prototype are quite messy, but they do enough to get the job done. I'll annotate #1967 about any large internal concerns I have about the prototype and possible steps forward.

Back to our srpm, we can use the rpm with rpmbuild --rebuild:


% rpmbuild --rebuild --target noarch perl-File-Temp-0.2311-1.src.rpm
...

By default, rpmbuild puts things into ~/rpmbuild/ so let's check out the resulting binary rpm:

% rpm -qlp ~/rpmbuild/RPMS/noarch/perl-File-Temp-0.2311-1.noarch.rpm
...
/usr/local/lib64
/usr/local/lib64/perl5
/usr/local/lib64/perl5/5.36
/usr/local/lib64/perl5/5.36/auto
/usr/local/lib64/perl5/5.36/auto/File
/usr/local/lib64/perl5/5.36/auto/File/Temp
/usr/local/lib64/perl5/5.36/auto/File/Temp/.packlist
/usr/local/share
/usr/local/share/man
/usr/local/share/man/man3
/usr/local/share/man/man3/File::Temp.3pm
/usr/local/share/perl5
/usr/local/share/perl5/5.36
/usr/local/share/perl5/5.36/File
/usr/local/share/perl5/5.36/File/Temp.pm

The RPM SPEC

``` # Hello packaging friend! # Allow building noarch packages that contain binaries %define _binaries_in_noarch_packages_terminate_build 0 Name: perl-File-Temp Version: 0.2311 Release: 1 Summary: return name and handle of a temporary file safely AutoReqProv: no # Seems specifying BuildRoot is required on older rpmbuild (like on CentOS 5) # fpm passes '--define buildroot ...' on the commandline, so just reuse that. BuildRoot: %buildroot Prefix: / Group: default License: perl_5 Vendor: Tim Jenness URL: https://github.com/Perl-Toolchain-Gang/File-Temp Packager: Source0: /tmp/package-srpm-build-63f0bb2fa18b36078b29042e1c09f43d5cf2ec2da0c99c8ba8be8df33a5e/File-Temp-0.2311.tar.gz Requires: perl(Carp) Requires: perl(Carp::Heavy) Requires: perl(Cwd) Requires: perl(Exporter) >= 5.57 Requires: perl(Fcntl) >= 1.03 Requires: perl(File::Path) >= 2.06 Requires: perl(File::Spec) >= 0.8 Requires: perl(IO::Handle) Requires: perl(IO::Seekable) Requires: perl(POSIX) Requires: perl(Scalar::Util) Requires: perl(Symbol) Requires: perl(constant) Requires: perl(overload) Requires: perl(parent) >= 0.221 Requires: perl >= 5.006 Requires: perl(strict) Provides: perl(File::Temp) = 0.2311 %description return name and handle of a temporary file safely %prep %autosetup -n File-Temp-0.2311 %build mkdir -p ./cpan cpanm -L ./cpan . --installdeps perl Makefile.PL make %install %make_install %files %defattr(-,root,root,-) / %changelog ```

jordansissel commented 1 year ago

Are y'all thinking about targeting this feature to systems other than Red Hat's platform? Debian,etc? Are we thinking newer systems like flatpak, snap, appimage, etc?

wbraswell commented 1 year ago

@jordansissel Thanks for putting together the new prototype #1967 , I will review it with @NicholasBHubbard and we will get back to you with any technical concerns.

As for the Red Hat question, the answer is yes we are targeting all possible packaging systems that FPM can or ever will support. Nothing we are doing is Red Hat specific, it just so happens that RPM was the first package format that I personally chose to start working with a few years ago when you and I first met here online. We have already started preparing for DEB and snap etc etc... So whatever source package solution we choose, it needs to work with DEB and any other packaging system that supports "source packages".

jordansissel / fpm