SRPM output

[jordansissel]

This issue now tracks a slightly different issue than originally -- It now tracks the feature that has fpm output SRPM, rather than originally proposed inputing.

That is, fpm --output-type rpm should output a .src.rpm

---

Original text:

Idea:

fpm -s srpm -t deb <url or path to srpm>

Could be fun.

[mitchellh]

"Could be fun"

Your definition of fun.

But its why I love you.

[kwilczynski]

LOL, definitely a +1

[wzzrd]

Would this also bring -t srpm?

[jordansissel]

How would '-t srpm' behave (given fpm generally produces what rpm calls 'binary' packages, a 'binary srpm' seems like a strange idea) - Can you explain how you'd like that to work? :)

[robredpath]

+1, I'd love to see this as a feature

[jordansissel]

For folks +1'ing this, do you have a specific use case in mind?

[robredpath]

I'm currently packing some quite heavily modified (modified paths/filenames to fit around other binaries, rather than code) Ruby RPMs, and I've found adapting Fedora RPMs for el5 use frustrating because of various dependencies and other issues. FPM creates lovely RPMs of my dir trees (built with rbenv and then modified by hand) but as a build process it's not really sustainable; as soon as we start patching, we're in the kind of mess than RPM helps avoid. I'd have loved FPM to knock out an SRPM that I know will build and that I can then modify as needed or just keep around in case an urgent patch or whatever is required; in my case, I'm just writing spec files from scratch and updating them as needed but each ones takes that bit of time and has those few complexities that FPM's normal functions are nice at avoiding.

[wzzrd]

What he said ;)

It also would make sure I can reproduce the RPM from the SRPM at some point with absolute certainty it will be the same as the RPM I had FPM make originally, save for some fix I need to make, where using FPM to recreate the RPM might yield slightly different results as the program evolves.

[jordansissel]

@robredpath have you tried bernd/fpm-cookery? Might be useful here. It helps you write repeatable build processes that are simple and invokes fpm to do the packaging step at the end.

[robredpath]

I hadn't! Looks interesting, thanks.

[herlo]

What about having output be srpm or dsrc? I think there are some valid use cases for this, including being able to adjust a few things and rebuild. Plus, rebuilding for the OS (like Fedora) requires source code. It could be very helpful.

I understand that gem2rpm is there, but why not provide both the srpm and the binary rpm as an example.

[ViktorNova]

+1 I would love to have this

I build and package a lot of Debian/Ubuntu packages from source, and occasionally there is a source RPM but no Debian control/etc files, so this could really speed up the process of hunting down the dependencies, figuring out the proper flags, looking up the syntax for how to properly set the DESTDIR in that particular build system, etc.

[BIAndrews]

+1 would love to have it at a target. It's normal to create an SRPM with rpmbuild -bs

# rpmbuild --help
Usage: rpmbuild [OPTION...]

Build options with [ <specfile> | <tarball> | <source package> ]:
  -bp                           build through %prep (unpack sources and apply patches) from <specfile>
  -bc                           build through %build (%prep, then compile) from <specfile>
  -bi                           build through %install (%prep, %build, then install) from <specfile>
  -bl                           verify %files section from <specfile>
  -ba                           build source and binary packages from <specfile>
  -bb                           build binary package only from <specfile>
  -bs                           build source package only from <specfile>

[spacekookie]

I would like this feature because of the Fedora packaging process that requires an SRPM and spec file. I know there are other tools to do that. But it would be nice having it integrated into fpm to make the whole workflow simpler :P

[strizhechenko]

I want to push my project to fedora's repos and I need SRPM + .spec for it and I would really like something like this:

fpm -s python -t srpm netutils-linux

I can also try to be a "proxy" with feedback about submission packages built with fpm from fedora community. :)

[strizhechenko]

Update: setuptools can generate src.rpm's!

python setup.py bdist_rpm

will provide you:

cat build/bdist.macosx-10.12-x86_64/rpm/SPECS/netutils-linux.spec

[timhughes]

@strizhechenko take a look at https://github.com/fedora-python/pyp2rpm and https://fedoraproject.org/wiki/Packaging:Python

Also think about hosting your RPMs on copr https://copr.fedorainfracloud.org as this will give people the ability to have people use them and give you feedback on any improvements needed to get them into Fedora.

[centminmod]

@timhughes coincidentally i was looking at fedora copr and saw the SRPM requirement, so having fpm support SRPM would be great :)

[wbraswell]

Since this issue is still open after all this time, and still quite pertinent, I will use it as the focal point for discussion of the need to create source packages for both DEB and RPM output formats. (For the sake of this discussion, whenever I refer to SPEC or SRPM or "source package" for RedHat/CentOS, the same needs and concerns can also be extended to include the DSC control files for Debian/Ubuntu.)

To be clear, there are 2 very different issues regarding SRPM files: accepting them as input to FPM, and generating them as output from FPM. There seems to be confusion about this distinction within this very discussion thread, and obviously a significant number of FPM users are upvoting this issue for one or both.

This Issue #237 was created in 2012 for the purpose of enabling SRPM files as input to FPM, which means accepting fpm -s srpm as a valid command syntax as specified by @jordansissel in the first post of this discussion thread. Similarly, fpm -s spec could also be implemented, because sometimes you have the SRPM (containing a SPEC) and sometimes you just have the SPEC file itself.

On the other hand, Issue #17 was created (and closed) in 2011 for the purpose of enabling SRPM files as output from FPM, which means accepting fpm -t srpm and fpm -t spec as valid command syntax, plus hopefully some other command-line option(s) enabling output of RPM & SRPM & SPEC files all during the same execution of FPM.

In fact, using SPEC/SRPM as input is currently listed as Feature Request 2 and SPEC/SRPM as output is Feature Request 3: https://github.com/jordansissel/fpm/wiki/Feature-Requests

Enabling SPEC/SRPM as input seems to be within reach of FPM as it is currently implemented, so I will not address this feature request any further.

Enabling SPEC/SRPM as output is "out of scope" for FPM as it is currently implemented, because for any arbitrary input (such as a tarball) we do not know "the download, configure, make, etc steps" necessary to build the source into a form which FPM can then package. https://github.com/jordansissel/fpm/issues/17#issuecomment-1408376

Luckily, as noted by @jordansissel in this very discussion thread, there is an FPM-compatible packager called FPM-Cookery, which adds the knowledge of said build steps encoded in the domain-specific language of so-called "recipe" files. Like normal FPM users in this thread and elsewhere, users of FPM-Cookery have also asked for support of generating SRPM output: https://github.com/bernd/fpm-cookery/issues/149

As noted by FPM-Cookery creator @bernd "Once FPM gets support for [SRPM output] to make it work on multiple target platforms, we can consider supporting it as well." Well now we have a real chicken-and-egg problem, because FPM needs recipe input support in order to add SRPM output, and FPM-Cookery has recipe input support but is waiting for FPM to add SRPM output support!

In other words, we are stuck in DEADLOCK!

Interestingly, the creators of both FPM & FPM-Cookery discussed the possibility of merging (or at least tighter integration) back in 2012, but never followed up on it: https://github.com/bernd/fpm-cookery/issues/8

[ BEGIN EDIT ] I forgot that I had also opened a new issue about this, and naively assumed we could simply pass a flag such as '''rpmbuild -bs''' or similar to achieve this, when in fact this only creates a mostly-empty SRPM containing a non-rebuildable SPEC file. https://github.com/jordansissel/fpm/issues/1526 [ END EDIT ]

I hereby officially propose that the time has finally come to now integrate or merge FPM-Cookery into FPM, with the immediate goal of generating output SRPM/SPEC files for RPM systems and output DSC files for DEB system.

I am not a Ruby programmer (I wrote the RPerl compiler), but I am eager to assist in any way. What can I do to help move this forward?

[jordansissel]

I think about srpm (and debian dsc/source packages) support from time to time. I'm still not sure how it could work with the current design of FPM -- I'll try to write down what's in my head --

When you do fpm -s SOURCe -t TARGET foo, fpm basically does three main things:

1) Prepare the source SOURCE. "input". This is responsible for populating a staging area with files as well as setting any discovered properties. 2) Does any necessary conversion steps (file renames, property handling) from SOURCE to TARGET 3) Outputs a TARGET package containing the staging area files plus the properties (package name, dependencies, etc)

The 'TARGET output' (step 3) has no knowledge of any prior steps (how to build, where to download, etc), so an fpm -t srpm (output srpm), as fpm is currently designed, would only have a list of files plus the properties. "Building" from this hypothetical srpm would execute no build steps (in the SPEC, %prep and %build sections would be empty, for example, and %install would just copy files.)

Few (if any) of these steps will run external shell commands.

In order to produce an "SRPM" of a rubygem, for example, I don't think FPM's design allows for it.

SRPM to me is a file format which contains source files and build instructions as shell commands. In order for FPM to emit a successful SRPM (in the spirit of SRPM where you can actually "build" from the SRPM), FPM would have to emit build steps as shell commands. For the -s gem, this might look like rewriting much of the code (like this method) to emit shell commands instead of computing things in Ruby.

If we want to ignore the build system features of SRPM, then it would be easier for FPM to emit an SRPM which simply has no build steps at all and just says "when you build, copy these files".

There's prior art in the example (gem to srpm/spec) in the gem2rpm project that shows some of the complexity.

Looking at the concepts in an RPM SPEC file against similar things (Homebrew, fpm-cookery, FreeBSD Ports, Gentoo Portage, NetBSD pkgsrc, etc) they all present the same ideas: A set of properties (package name, version, download url, dependencies, etc) and a set of build instructions, and even these build instructions aren't always pure shell in all of these build systems, and as such may not even be automatically convertible even between the same kinds of tools (homebrew to rpm spec, for example).

Predecessors to fpm may be of interest:

• gem-packaging - a shell script which reads a .gem with gem2rpm, writes a .spec and then calls rpmbuild.
• python-packaging - a shell script which inputs a python package name, installs the python package to a temporary directory, then invokes dh_make and debuild to build a .deb simply from a set of files (no build step)

---

All told, I think there's two major options for srpm output (and any other build-recipe output)

1) Have fpm produce an srpm as output which contains only files and has no build steps. rpmbuild on such a thing would never do any actual build steps but would still produce an RPM. 2) Rewrite most of fpm's input methods to emit some kind of abstraction we can convert to whatever build recipe (SRPM, RPM SPEC, Debian Makefile, Homebrew, etc)

For option 2, it is the most flexible, but is basically building a DIY system which replicates what RPM SPEC, FreeBSD Ports, etc already do.

For option 1, it might work, but rpmbuild would never actually compute anything -- an srpm would basically just be a file archive waiting to be converted to an rpm. In this way, having the srpm, to me, is not useful, because there are no build steps.

Thoughts on which options? Other ideas?

[jordansissel]

To frame it in terms of fpm and rpm spec, from my above comment:

1) Prepare the source SOURCE. "input". This is responsible for populating a staging area with files as well as setting any discovered properties.

This is similar to what a SPEC's %prep, %build, and %install sections do. Notably, SPEC steps are written in shell, not ruby, and fpm's are written mostly in ruby with some shell execution.

2) Does any necessary conversion steps (file renames, property handling) from SOURCE to TARGET

This step is not present nor necessary in an rpm spec.

3) Outputs a TARGET package containing the staging area files plus the properties (package name, dependencies, etc)

This is roughly the RPM SPEC Tags (Name: ..., Requires: ...) plus the %files section.

Given this, at "output" time, fpm only knows about the files and the properties (what rpm calls tags), so with today's fpm, at best we could have a spec file plus a tarball of files. Because fpm uses rpmbuild today as a way to produce an rpm file, fpm already builds exactly this spec file (no build steps, only %files and rpm tags -- see the rpm spec template).

You can see the SPEC that fpm -t rpm produces with the --edit flag:

% fpm --edit -s dir -t rpm -n myzshrc ~/.zshrc
...

An abbreviated rendition of the SPEC file shown with the above command looks like this:

Name: myzshrc
Version: 1.0
Release: 1
Summary: no description given
AutoReqProv: no

Prefix: /

Group: default
License: unknown
Vendor: jls@localhost.localdomain
URL: http://example.com/no-uri-given
Packager: <jls@localhost.localdomain>

%description
no description given

%prep
# noop

%build
# noop

%install
# noop

%clean
# noop

%files
%defattr(-,root,root,-)
/home/jls/.zshrc

%changelog

Hopefully this helps illustrate the internals of fpm as they are today.

[wbraswell]

@jordansissel Great info, and yes I was already aware of most all of these details, as I have been digging into the FPM workdir which does preserve the SPEC and SRPM files as they currently stand.

You have presented 2 options, the first is basically a "fake" SRPM which I have already been able to extract from the FPM workdir. This SRPM currently only contains a SPEC file, and can (as you said) be made into a file archive waiting to be rebuilt, neither of which actually serves the real purpose of an SRPM which is to be rebuilt from source on a different target system with a different compiler, etc.

So, we need to go with your second option, which (I believe) is a "real" SRPM which actually contains a source tarball and a SPEC file with real configure/build/install commands.

The first step should be to use FPM-Cookery recipes as the input source for this new "real" SRPM output format, because FPM-Cookery is already essentially built for this purpose.

The second step, for me at least, is to use CPAN build commands perl Makefile.PL && make && make test && make install or perl Build.PL && Build && Build test && Build install for building Perl packages.

And of course, everything that applies to SRPMS will also need to translate into DSC files for Debian, and possibly other "source package" output formats as well. (I am only familiar w/ RPM & DEB package formats.)

I have re-read your posts in more detail and I think there are no other pertinent technical issues, so the main idea is that we need to go with option 2, which is the "real SRPM" option.

[wbraswell]

@bernd Can you please give us your ideas and input on how best to proceed with greater compatibility between FPM and FPM-Cookery? Thanks in advance! :-)

[trevor-vaughan]

PR should be ready to go and tests are passing if the implementation is acceptable.

[jordansissel]

Based on work in #1956/#1657, I am able to use the srpm with mock:

% fpm -s gem -t rpm rails
...

% mock --rebuild rubygem-rails-7.0.4-1.src.rpm
... lots of output omitted for brevity ...
Wrote: /builddir/build/RPMS/rubygem-rails-7.0.4-1.noarch.rpm
Finish: rpmbuild rubygem-rails-7.0.4-1.src.rpm
Finish: build phase for rubygem-rails-7.0.4-1.src.rpm
INFO: Done(rubygem-rails-7.0.4-1.src.rpm) Config(default) 2 minutes 45 seconds
INFO: Results and/or logs in: /var/lib/mock/fedora-36-x86_64/result
Finish: run