Provide RPM/specfile?

[fearful-symmetry]

For large, mutli-tenant environments, deploying CLI tools via and RPM is more maintainable than a global pip install across multiple nodes. This is also how singularity itself is distributed. So, getting some kind of RPM, or at least a specfile, from upstream would be nice. Is this something someone has considered before?

There is one caveat I can think of, which is the multiple optional dependencies against various backends (dropbox, google, shub, etc). Not sure what the best way to do that is.

[vsoch]

This would be a great idea! I've never done one from scratch, but I have an (example) from a user that could get us started - this is called sregistry-python.rm

# https://www.ibm.com/developerworks/library/l-rpm1/
#
# spec file for package python-sregistry
#
Name:           python-sregistry
Version:        0.0.76
Release:        0
Url:            http://www.github.com/singularityhub/sregistry-cli
Summary:        Command line tool for working with container storage
License:        LICENSE (FIXME:No SPDX)
Group:          Development/Languages/Python
Source:         https://files.pythonhosted.org/packages/source/s/sregistry/sregistry-%{version}.tar.gz
BuildRoot:      %{_tmppath}/%{name}-%{version}-build
BuildRequires:  python-devel
%description
# Singularity Global Client
Hi Friends! Are your containers lonely? Singularity containers thrive in happiness when they are shared. This means that wherever you might have them in these cloudy places, they are easy to find and move around.
## What is this?
Singularity Global Client is an interface to interact with Singularity containers in many different storage locations. We are able to use modern APIs by way of providing and using the software within a Singularity container! For older architectures, we provide a [Singularity container](Singularity) for you to use instead. You can build it from this repository, or use the provided container on [Singularity Hub](https://www.singularity-hub.org/collections/379).
See our [installation guide](https://singularityhub.github.io/sregistry-cli/install) to get started. For more details, please refer to our [documentation](docs)
## License
This code is licensed under the Affero GPL, version 3.0 or later [LICENSE](LICENSE).
%prep
%setup -q -n sregistry-%{version}
%build
python setup.py build
%install
python setup.py install --prefix=%{_prefix} --root=%{buildroot}
%clean
rm -rf %{buildroot}
%files
%defattr(-,root,root,-)
%{python_sitelib}/*
/usr/bin/sregistry
%changelog

Would you be able to help / take lead on finishing this guy up? I am pinging @griznog who can take credit for this work! If we can work together it would be awesome! I would appreciate it greatly as I'd learn a lot too :)

[fearful-symmetry]

Yah, some help would be appreciated as I'm not very good at spec files. That being said, I can poke around with this or wait to see if @griznog has anything to add.

[vsoch]

I think if I can understand the steps to build and deploy, I could easily provide as a package!

[griznog]

I've since dismissed the brain cells that had looked at this for dereliction of duty, but I think the above spec file may Just Work(tm) if the repo tags a release for it to reference and download and the release versions match in the repo and spec.

[vsoch]

Cool, so what does one do with that file?

[griznog]

First thing I'd try is putting it in the repo as sregistry.spec. Then set the line for

Version:        0.0.76

To whatever the github release version is. Then tag a release (no idea how that works) so that github generates the .tar.gz file with the version and that spec file included.

No with the resulting .tar.gz, it's just

rpmbuild -ta sregistry-$version.tar.gz

and you'll either get errors or rpms. One of which will be an srpm which can then be distributed and anyone can rebuild with rpmbuild --rebuild blahblah.srpm

That's all from memory and untested, so grains of salt as required.

[griznog]

Also, dependencies are left as an exercise for the reader :)

[vsoch]

ok testing this out!

[vsoch]

okay @fearful-symmetry you can take it from here! I have a basic .tar.gz built that you can download and (see if you are able to get it to build / make tweaks to do so). Here is what I was doing to test it:

# download
cd /tmp
wget https://github.com/singularityhub/sregistry-cli/archive/0.0.91.tar.gz

# I messed up a label somewhere because it wants the file with and without the name
$ cp 0.0.91.tar.gz sregistry-0.0.91.tar.gz

$ rpmbuild -ta 0.0.91.tar.gz 
error: Failed build dependencies:
    python-devel is needed by sregistry-cli-0.0.91-0.x86_64

I tried in a container to get around installing random stuff on my host:

docker run -it centos bash
yum update && yum install -y python-devel wget

cd /tmp
wget https://github.com/singularityhub/sregistry-cli/archive/0.0.91.tar.gz

# oh yeah this thing
yum install -y rpm-build

# I messed up a label somewhere because it wants the file with and without the name
$ cp 0.0.91.tar.gz sregistry-0.0.91.tar.gz

$ rpmbuild -ta 0.0.91.tar.gz 
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.gzYtyU
+ umask 022
+ cd /root/rpmbuild/BUILD
+ cd /root/rpmbuild/BUILD
+ rm -rf sregistry-0.0.91
+ /usr/bin/gzip -dc /tmp/sregistry-0.0.91.tar.gz
+ /usr/bin/tar -xf -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd sregistry-0.0.91
/var/tmp/rpm-tmp.gzYtyU: line 35: cd: sregistry-0.0.91: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.gzYtyU (%prep)

RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.gzYtyU (%prep)
[root@db4b9c5b7b12 tmp]# 

So I think at this point it's mostly debugging the build steps (e.g., the above is a path error!) So I think this would be a good start for you to jump in? When you get a set of changes that warrant a new version let me know and I'll make it so.

insert Picard meme

[vsoch]

oh I see the name issue:

%prep
%setup -q -n sregistry-%{version}
...

that would be the first thing to fix!

%prep
%setup -q -n %{version}
...

[griznog]

Is it possible to make github name the release file sregistry-${version}.tar.gz instead of ${version}.tar.gz? I find tarballs with only have a version in the and those which unpack into a directory with just a version as the name to be really annoying. If the project name is in the tarball and is the same for the folder, then the spec file can have the same name and everything is consistent.

[vsoch]

yeah let me try it. I've never done it before because it didn't matter, but now I know it does!

[vsoch]

okay, updated one --> https://github.com/singularityhub/sregistry-cli/releases/tag/sregistry-cli-0.0.93

[fearful-symmetry]

I seem to be having issues with the release name scheme that @griznog suggested. The releases with the name form ${name}-${version}.tar.gz seem to result in an unpacked dir called sregistry-cli-sregistry-cli-0.0.93 which rpmbuild doesn't like.

[griznog]

The file ${name}-${version}.tar.gz should untar into ${name}-${version} directory. I have no idea how to get github to make it do that though.

[fearful-symmetry]

the ${version}.tar.gz releases seem to be untarring into the correct folder name.

[vsoch]

Should we change it to be ${version}.tar.gz instead?

[vsoch]

Let me much around with the Github interface and see if I can break thi... I mean, fix things :)

[vsoch]

@fearful-symmetry what aspect doesn't it like? It looks like when we have:

${name}-${version}

and name is set to sregistry-cli

it downloads sregistry-cli-0.0.91.tar.gz and extracts to sregistry-cli-0.0.91. What doesn't rpmbuild like about that? Two dashes?

[vsoch]

okay I am testing again.

[fearful-symmetry]

I've been testing by having rpmbuild download the tar file. wget https://github.com/singularityhub/sregistry-cli/archive/sregistry-cli-0.0.93.tar.gz && tar xvf sregistry-cli-0.0.93.tar.gz

results in a directory called sregistry-cli-sregistry-cli-0.0.93/

via rpmbuild, it can't cd into the directory:

/var/tmp/rpm-tmp.78cWgx: line 35: cd: sregistry-cli-0.0.93: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.78cWgx (%prep)

If I use one of the older releases that use the form https://github.com/singularityhub/sregistry-cli/archive/0.0.91.tar.gz then the rpm builds fine, as it unpacks to sregistry-cli-0.0.91/

[fearful-symmetry]

I checked another project I've been playing with and they seem to do the same thing with the release urls, just using the version string.

[vsoch]

I don't think the rpm is right, I'm looking at other projects and seeing some variance.

[fearful-symmetry]

There does appear to be an issue with the specfile as well: https://github.com/singularityhub/sregistry-cli/archive/${version}.tar.gz should be https://github.com/singularityhub/sregistry-cli/archive/%{version}.tar.gz I was going to submit a PR, but I got distracted by the naming thing.

[vsoch]

Yeah I fixed that locally :) I'm trying to figure out a way to test this locally without needing to do a Github release every time...

[vsoch]

okay here is updated file : https://github.com/singularityhub/sregistry-cli/blob/master/sregistry-cli.spec

The issue I'm running into locally is about a group/name permission, which is likely because I'm trying this in a container. See if you can clone and compress the folder into an archive and test locally - I just can't create a release to wget every time.

[vsoch]

We have hope!! A gift from @trophime :)

He has been creating (and using) these build recipes, which I've added here:

https://github.com/singularityhub/sregistry-cli/tree/master/release/rpm

@fearful-symmetry do you think we can get this generalized to work with an export from Github? I can write some helpful "how to package" and "how to use" docs if we get this down right! Looking quickly - I think the .dsc file is akin to the .spec, and that the archives coincide with the .tar.gz we've been using.

[vsoch]

And more help! I need to look through this carefully too, I've never done this before:

prior to build the package you have to install a clean env for the build. To do so use cowbuilder and pbuilder

To build the package:

run: dpkg-source -x *.dsc
cd into the create directory
run: pdebuild 

Refs: http://packaging.ubuntu.com/html/ https://wiki.ubuntu.com/PbuilderHowto

Note that there are more recent tools: https://wiki.debian.org/PackagingWithGit https://wiki.debian.org/PackagingWithGit/GitDpm

[fearful-symmetry]

So, uhhhh, I feel like this is perhaps anticlimactic, but apparently you can just run python setup.py bdist_rpm

And it works fine. Builds a basic RPM and all. The only issue I'm having is that the RPM doesn't specify dependencies or things like that, so the installed RPM doesn't work unless you already have spython and requests and all already installed.

[vsoch]

haha are you kidding? This is music to my ears! You basically said:

• oh, someone else has figured this out before
• it's like a two line solution
• problem solved!

Let me give it a test, write up some docs, and I'll post here for all to look at.

[vsoch]

Okay here is a quick documentation:

https://singularityhub.github.io/sregistry-cli/contribute-packaging

This is mostly working - but it's not clear to me why when I specify a pre-build script (and I can see that the code shows up in the pre section of the rpm spec when I have --spec-only when I actually run rpm-build it doesn't run that section (so in testing I ran it on my own). Any ideas?

And it's really my fault for not starting fresh with a Google search for "rpm package for python" because I would have found that quickly. My entire thinking was based on a starting point of "oh, I need a spec file!" Anyway who cares, we are figuring it out, and lesson learned!

[vsoch]

No response for a month, closing issue. the docs are live and anyone that runs into issue I hope will bring up here. Thanks again! Closing.