search

akrennmair / newsbeuter

Newsbeuter is an open-source RSS/Atom feed reader for text terminals.
http://www.newsbeuter.org/
MIT License
780 stars 100 forks source link

snap package #456

Open thibran opened 7 years ago

thibran commented 7 years ago

It would be nice if Newsbeuter would be packaged as snap-package and published to the Ubuntu Store. I would be willing to do the work but would need some help to get things right.

About Snap Packages: Snap packages are confined and can be installed on different distributions (Debian, openSuse, Fedora, Ubuntu...).

Minoru commented 7 years ago

What are the benefits? I skimmed Snapping philosophy, but don't see anything important for Newsbeuter there.

Also, what version of Newsbeuter do you want to package? Will it be only our stable releases, or do you want to do periodic snapshots from Git, or what?

I admit I'm sceptical at the moment, but let's talk.

thibran commented 7 years ago

What are the benefits?

but don't see anything important for Newsbeuter there

Canonical moves Ubuntu to a snap-package future (phone, desktop, server) and with it millions of users. Some of those users like to read RSS feeds using Newsbeuter ^^

what version of Newsbeuter do you want to package?

Snap-packages are created with a tool called Snapcraft using a "build-file" called snapcraft.yaml. I would build a .yaml for the current stable Newsbeuter and help you test, build & publish it to the Ubuntu Store.

With Launchpad.net it is possible to build snaps automatically out of an arbitrary git branch. So it would also be possible to build beta-snaps using it.

PS: No, I'm not an Ubuntu user, most time I use openSUSE. I just want to give something back to the open source ecosystem.

Minoru commented 7 years ago

Snap packages contain all dependencies

That means I have to track them all (and transitive dependencies too!) in order to update on next Heartbleed—something that maintainers would've done for me.

Therefore a once build snap can run forever on a system that supports snap packages (unlike the current .deb or rpm packages).

Running the old version forever is just delaying the inevitable: one of these days user will run into a bug or something and will have to either start maintaining the project or migrate to something else. Is it honest to mask this issue?

Overall it feels like Snap is for companies who aren't willing to work with maintainers and just want their packages to be runnable on Linux. This isn't the case for Newsbeuter: I'm grateful to maintainers for packaging the program for their distros because that frees me from thinking about it.

Canonical moves Ubuntu to a snap-package future (phone, desktop, server)

Any prooflinks for that?

Did you package anything before? How much work does packaging require from me? I take it you want to help me publish a package, not publish it yourself? Is there a technical reason for that?

thibran commented 7 years ago

The Ubuntu phone is switching to snap in the coming months. There is already a server/cloud snap-only version of Ubuntu and the desktop is currently in the work and looks like this. Right now it is possible to install snap packages in Ubuntu 16.04 LTS and in near future on Ubuntu 14.04 too. In the Ubuntu dev-bubble there is a lot of talking about snap, unity8 (the new desktop shell) and mir (the new display server). I can assure you that the movement to snap is happening on Ubuntu and it is therefor alone worth to package Newsbeuter.

Creating snaps is not that hard, take a look a snapcraft.yaml.

That means I have to track them all (and transitive dependencies too!) in order to update on next Heartbleed—something that maintainers would've done for me.

The snapcraft.yaml contains a list of lib debendencies which are needed to build the project. If a lib is no longer needed or a new lib is required the .yaml must be edited (it's a single file and the syntax is easy).

Running the old version forever is just delaying the inevitable: one of these days user will run into a bug or something...

I would suggest to get started by packaging the current stable version. Since snap packages can be published anytime you could release a snap version for every Newsbeuter release or something like once a year. Because of confinement it is even save for the user to use outdated libraries in a snap (the app process can access no resources - that's what confinement means ^^). Newsbeuter for example might only need access to the internet and to the per-app read/writable app folder.

Did you package anything before?

I packaged my own little command-line tool (src) and did also some .deb packaging before and know how to build Android Apps (and package it) - snap is by fare the easiest packaging system.

How much work does packaging require from me?

It does not require much work once the .yaml file is build. All you/your build service would have to do is to execute command snapcraft in the directory where the .yaml file is placed.

I take it you want to help me publish a package, not publish it yourself?

The idea of snap is that the upstream publish their software directly, like on Android or iOS and have therefor control about when, what and how often the software is updated. Snap will e.g. allow Ubuntu/Debian users to get the newest version of a snap-packaged software but still relay on an otherwise stable (but outdated) system.

I could publish the snap myself, but would not be allowed to call it newsbeuter. I would have to call it something like newsbeuter-thibran.

Any other questions?

Minoru commented 7 years ago

Thanks for the detailed response!

If a lib is no longer needed or a new lib is required the .yaml must be edited (it's a single file and the syntax is easy).

I was thinking of another situation. Suppose a vulnerability is found in one of the libraries we depend on. With ordinary packages, a maintainer of this library will update their package and everything will be fine. Maintainer of Newsbeuter don't need to do anything as library is dynamically loaded. We (the upstream) might never even hear of the vulnerability.

With Snap, we're suddenly not only the upstream developers but the publishers as well, and need to track all the things we depend on (including the things our dependencies depend on—for example, we depend on curl and curl depends on OpenSSL in some configurations, so we have to track OpenSSL even though we don't use it directly.)

This is a lot of work which I simply don't want to do. This leads us directly into the next question:

The idea of snap is that the upstream publish their software directly, like on Android or iOS and have therefor control about when, what and how often the software is updated.

What does Ubuntu provide for people who don't want all this control?

I could publish the snap myself, but would not be allowed to call it newsbeuter. I would have to call it something like newsbeuter-thibran.

Oh, that's interesting! Where can I find more on these requirements? Say, will a commit bit in this repo make you eligible to maintain a snap called "Newsbeuter"?

(I can't really hand out commit bits at the moment, but I'm working on getting this capability.)

thibran commented 7 years ago

I was thinking of another situation. Suppose a vulnerability is found in one of the libraries we depend on.

It's the same as with mobile apps. If a lib you depend on and ship inside your package has a bug/security hole the effect on the users or his system is limited by app confinment. Lets say libssl has a bug, what would that mean for the user. In worst case someone could send the user wrong feeds/data, but that's it. You could not take control of the user system or manipulate any other files on his computer.

The app model Android, iOS and Ubuntu follows with snap is they have understood that at a certain scale you can not audit all packages. It takes fare too long and there might even still be security problems in the audited packages (also users will stay forever with outdated packages). Therefor they all go the rout of building a box, giving the running process as few resources as possible and control the access to resources through something they call "trusted helpers". On traditional Linux if you want access to something, like recording with the microphone, you just access it. On a phone-like security system it goes like this. App needs permission to record sound. When the app process asks the API to record, the user is prompt by the trusted helper (which is run by the OS) if the user wants to grant access. If yes, the trusted helper gives the app process access to the microphone.

What does Ubuntu provide for people who don't want all this control?

In the end nothing. A third party could pick-up the code of a project and build a snap or the upstream could build a snap once and than forget about it (which would be okay too, because the snap would work forever).

Oh, that's interesting! Where can I find more on these requirements? Say, will a commit bit in this repo make you eligible to maintain a snap called "Newsbeuter"?

I followed to some extend the development of snap and snapcraft, the "name-extension" is just by convention. I could release any software as snap (as long as the license permit it) under any name, but if I release it as newsbeuter and you come later along and release it under the same name, you would as upstream get the name and my package would be re-named. Therefor by convention non-upstream packages should have a name-postfix. If you allow me to release the official newsbeuter snap (using my Launchpad account) all would be fine. The next question is than, if Newsbeuter has an official snap, why is the snapcraft.yaml not shipped with the source code?

If you want I could do the snap releasing and have an eye on the snapcraft.yaml, helping you to keep it up-to-date.

Minoru commented 7 years ago

(also users will stay forever with outdated packages)

So in response, Snap takes away the very ability to update. Though I do see where they're coming from here—compartmentalization helps reduce the damage (if it is really inescapable, that is.)

the upstream could build a snap once and than forget about it (which would be okay too, because the snap would work forever).

Users will still come here, or to IRC channel, or to mailing list, and ask for help or new features. If we start supplying Snap package, we got to do it properly and responsibly.

Convention is good, but who checks that I am indeed the upstream? And what is checked? Newsbeuter is currently in this funny situation where the owner of the repo is practically inactive, but I was given the commit bit and tend to the issue tracker and the code. Yet I don't have admin access to the website and mailing list.

if Newsbeuter has an official snap, why is the snapcraft.yaml not shipped with the source code?

That's the crux of the matter, really: if it's in the repo, I can't have it out of my mind—I got to know how to build it, how to make a release it and all the little tricks and gotchas that embroil even the smallest of infrastructures.

Or maybe I can. Please let me sleep on it; I'll respond in 20 hours or so.

And thanks once again for detailed responses. Googling and sifting through docs to get the same overall picture would've probably taken me a few days at the very least.

thibran commented 7 years ago

So in response, Snap takes away the very ability to update.

The current situation is on a non rolling-release distributions that the user has almost always an outdated application after some time. Also not all application/libs get security updates, only the ones that are supported (in Ubuntu they are in a special repository). Snap improve the status quo by improving the security of the user and make it possible to get newer versions of an application, as soon as the package owner updates it (snaps auto-update).

Users will still come here, or to IRC channel, or to mailing list, and ask for help or new features.

I understand, but don't think there can be done a lot to change that.

Convention is good, but who checks that I am indeed the upstream?

It's not checked, everyone can claim an app-name for themselves. Only if there is trouble Canonical takes action.

I'll respond in 20 hours or so.

Do that, take you time.

Minoru commented 7 years ago

That's the crux of the matter, really: if it's in the repo, I can't have it out of my mind

Now I think I can.

So with what limited power over the project I have at the moment, I give you permission to release the package under the official name.

I'm looking forward for your PR with snapcraft.yaml in it. I saw mention of AppArmor and SELinux in “Snapping Philosophy”—do policy files go into the repo, too? If there's a number of them, can you shove them into a separate directory?

What I'd really be interested in is nightly builds. Can Launchpad (or some other related service) do that for us? Essentially, this should enable more adventurous people to run Git version without compiling it themselves; this would aid our testing efforts.

thibran commented 7 years ago

Great, thanks.

I think the best is to put all snap related files into it's own directory, with it's own .gitignore and README.md.

I saw mention of AppArmor and SELinux in “Snapping Philosophy”—do policy files go into the repo, too?

No, I don't have to write any security profiles myself, that's done by the snappy tools (either snapcraft or the snappy daemon).

Next steps for me:

I don't know how long this will take – like always with software, who knows the unknowns...

Edit: If I have questions, how do I best reach to you?

Minoru commented 7 years ago

I think the best is to put all snap related files into it's own directory, with it's own .gitignore and README.md.

Sounds good to me.

let's talk later if under mine account or a new one for newsbeuter

Well, since you mentioned it, we can start the dialog now :) Basically I'm fine with you using your own account unless you don't want to or see some technical problems with this (e.g. snaps aren't transferable between accounts, so you won't be able to hand them over to new maintainer without disrupting users.)

setup auto-builds from dev branches

We keep everything we plan to release in master. Other branches are work in progress and aren't meant to be used by anyone.

I don't know how long this will take

That's okay; just keep me posted (via this issue, preferably, as this gives a nice public record of everything).

If I have questions, how do I best reach to you?

Depending on the kind of questions:

thibran commented 7 years ago

For the Launchpad account, it might be the easiest to create a new one and have my stuff and Newsbeuter separated.

My work-in-progress snap branch using current master.

... and the first bunch of questions:

Minoru commented 7 years ago

Is setting them to an empty-string okay?

Setting LANG to empty string doesn't seem right to me. How do you expect programs to know how to format dates, currencies, numbers, what symbols to treat as whitespace and so on? Why are you even doing that?

That said, if you set only LANG, things should be fine.

does Newsbeuter read/write from any other directory

/etc/newsbeuter/. gettext also needs access to internationalization files, which are under /usr/local/share/locale by default, but that might be overriden by distribution.

thibran commented 7 years ago

Setting LANG to empty string doesn't seem right to me... Why are you even doing that?

If I don't set LANG="" in launcher.sh I get setlocal failed: No such file or directory. Seems not right to me either. Even if I set LANG to a fix value like en_US.utf8 I get the above error.

Edit: I only have to set LANG="" to get it working, LC_CTYPE and LC_MESSAGES don't need to be changed from their default. The LANG default before setting it is: en_US.UTF-8.

Edit 2: Maybe Newsbeuter searches the local files under /usr/local/... but on a snap they are in /snap/newsbeuter/current/usr/local/.... Could that be the cause of that problem? If yes, can I somehow tell Newsbeuter to use the snappy path?

Since I build from source there is no etc directory, but it works without.

Minoru commented 7 years ago

setlocal failed: No such file or directory

Just to be sure: you meant setlocale, right?

You seem to be experiencing something akin to #32. See the issue referenced from there—maybe Snap environment doesn't have locales generated?

Does resetting LANG in launcher.sh prevent users from changing it? E.g., will users still be able to run LANG="C" newsbeuter?

Since I build from source there is no etc directory, but it works without.

There isn't one by default at all. But Newsbeuter does read from there if it exists, so it's important for Newsbeuter to have ability to read it.

thibran commented 7 years ago

Just to be sure: you meant setlocale, right?

Ups, yes.

LANG="C.UTF-8" works. I try to understand what is happening and how to get things right. If locale can't find /usr/share/i18n/locales the above error seems to be shown.

The confined process in a snap has no access to the filesystem and can't therefore read /etc or any other folder (like /usr/...). I checked that LANG and the other locale ENVs are by default set and point to the correct values. I try to find a good solution for the LANG problem, but don't have one right now at hand.

Minoru commented 7 years ago

Well, IMHO you shouldn't touch LANG in the first place. Users have it set to their locale; Newsbeuter needs it, for example, to show translated interface instead of default English one. By setting it in launcher.sh you effectively force everyone to use default interface and UTF-8 (Russians who didn't re-setup their systems in the last 10 years might still use KOI8-R, and Newsbeuter will look garbled for them if it tries to use UTF-8.)

Try running launcher.sh under strace -e open. This will show you a log of all the things that are opened. Towards the end you should see that some open call fails and then the error is shown. You'll know the path to the file that's missing.

You also have this comment in launcher.sh:

must be set to empty-string to pass !setlocale(LC_CTYPE,"")

This isn't true. From setlocale(3):

char *setlocale(int category, const char *locale);

If locale is an empty string, "", each part of the locale that should be modified is set according to the environment variables. The details are implementation-dependent. For glibc, first (regardless of category), the environment variable LC_ALL is inspected, next the environment variable with the same name as the category (see the table above), and finally the environment variable LANG. The first existing environment variable is used. If its value is not a valid locale specification, the locale is unchanged, and setlocale() returns NULL.

The check above ensures that setlocale didn't return NULL.

I also just noticed "Edit 2" in your earlier post! I think it's okay to post multiple comments in a row, reserving edits to typos. There's no emails about edits, so I might've missed that question entirely if I didn't re-read older posts.

Maybe Newsbeuter searches the local files under /usr/local/... but on a snap they are in /snap/newsbeuter/current/usr/local/.... Could that be the cause of that problem? If yes, can I somehow tell Newsbeuter to use the snappy path?

It could be! Newsbeuter actually hard-codes path to locales. You could see LOCALEDIR passed to compiler—that's the path.

But I thought Snap provides a container, and the app inside the container has its own filesystem?

If that /snap/newsbeuter/current/usr/local/... thing is stable, i.e. it doesn't change from invocation to invocation or from version to version—you can compile Newsbeuter with make localedir=/snap/newsbeuter/current/usr/local/share/locale (note the case—it's localedir not LOCALEDIR; see the Makefile for details.)

Minoru commented 7 years ago

After a debugging session over IRC, here's what @thibran and me discovered:

I believe this to be a deficiency in our Makefile. What I'm going to do is:

Minoru commented 7 years ago

I checked uappexplorer and I can't find a snap for Git. This makes it pointless to look at Git as an example of how to handle LOCALEDIR (in our case, at least). I tried "mutt" and "mcabber", but these aren't packaged as snaps either.

@thibran, are you aware of any small open source apps written in C or C++ that depend on gettext and are packaged as snaps?

Minoru commented 7 years ago

In the meantime, I looked at how Debian works around that problem, and it turned out that it doesn't. In their rules file (available in .debian.tar.xz archive from the package page) they call dh_auto_install, which in turn sets DESTDIR environment variable and runs make install.

It also turns out that Snap already does the same! At least judging by the docs for make plugin, which claim that make-install-var is DESTDIR by default (i.e. Snap sets DESTDIR to current snap's directory in hope that the Makefile will use it and redirect installations into it. Our Makefile indeed does that.)

What's the problem, then? I believe it lies in the fact that locale files are installed into DESTDIR/snap/newsbeuter/current/usr/local/share/locale, which upon installation of the snap becomes /snap/newsbeuter/current/snap/newsbeuter/current/usr/local/share/locale (because snap's internal filesystem is mounted under /snap/newsbeuter/current.)

I believe this to be a deficiency in our Makefile.

I no longer believe this to be a deficiency; it's just a baked-in assumption: we might be asked to install everything under an alternative root (i.e. DESTDIR), but files we create will be moved under real root before the program is started. Snap violates this assumption by moving the files to /snap/newsbeuter/current instead of /.

Now, we can either remove the assumption from our Makefile, or make snapcraft.yaml a bit more complicated:

Given Snap is the only platform I'm aware of that requires such treatment, I'd rather push complications our of our Makefile and into snapcraft.yaml. @thibran, what do you think?

thibran commented 7 years ago

The gettext problem is because snap does not include or give access to /usr/share/locale. The problem is known to snap upstream #LP1576282. I applied the suggested workaround from the LP issue.