Creating Debian Linux installation package

[imrehg]

Create install images as Debian/Ubuntu deb files, and possible other types like rpm as well.

The idea would be to avoid the "this is how you make it executable, this is how you run it from the file manager", etc issues, but getting as simple as possible to install Etcher on common distros. Reported by @limator

[dlech]

I tried packaging this for Debian/Ubuntu about a month ago since I have a bit of experience packaging software, but I quickly gave up. Debian policy is to build everything from source, so the packaging infrastructure is geared towards this.

The easiest thing to do is to have http://lanuchpad.net build the packages and host the ppa. However, the launchpad build servers do no allow Internet access during the build. So, we cannot fetch node and bower packages during the build process. This means in order to use launchpad, we also would have to package all of the node and bower packages as debian packages. This sounds like too much work for me.

I suppose we could do like the Arch Linux package and just throw the AppImage into a debian package. There seems like a reason why I decided this was not a good idea and didn't try it, but I can't remember what is was right now.

[imrehg]

Hi @dlech thanks for the comments! Yeah, our Launchpad setup has stalled a bit, and good to hear your comments, makes sense.

Debian/Ubuntu

As for packaging, as far as I remember, Ubuntu/Debian packages have the sources in there, so you think the build path could be adjusted by splitting the package downloads and the build to two clearly defined steps. Then could prepare a zip with all sources, which launchpad then builds? As I have no experience with their build system yet, would love to hear if this makes any sense or there are any gaps in the process. And whether @jviotti thinks that the build can be split like that :)

Arch

For ArchLinux, you made me think that indeed building it from source would make a lot of sense, will check it out, now that I have more experience building it locally (and some node bugs that were affecting the build were fixed up).

[dlech]

so you think the build path could be adjusted by splitting the package downloads and the build to two clearly defined steps. Then could prepare a zip with all sources, which launchpad then builds?

This could work. I don't know anything about how node and bower work though, so it would take me quite a long time to figure out how to do this. If someone can figure out how to get all of the source files zipped up, I will be glad to do the rest of the debian packaging.

[glensc]

i guess you can attach the source tarballs to github release page as well: https://github.com/resin-io/etcher/releases

i'm also interested building from source the rpm package. but considering this is electron app, the node dependencies need to be at least included in the source package.

[jviotti]

Hi @imrehg , @dlech

After some discussion, mainly with @alexandrosm , we decided the Etcher GUI will not provide official standard distribution packages (e.g: .deb, .rpm), and will focus entirely on AppImages, at least for the time being, since it aligns much better with desktop applications, and making .deb packages for an Electron app violates a lot of the "debian" rules (not sure if this holds for .rpm).

As a solution for the AppImages execution permissions problems, we can ship a zip containing the AppImage and potentially the user documentation file. Zip will make sure the execution bit is re-added when expanding.

That being said, we do want to provide packages for the Etcher CLI, so I'll close this issue and we can continue discussing Etcher CLI packaging on https://github.com/resin-io/etcher/issues/355.

@glensc

i guess you can attach the source tarballs to github release page as well: https://github.com/resin-io/etcher/releases

We've considered this as well, however the analytics that we get from GitHub Releases are not enough for us at the time being.

[alexandrosm]

Thinking about it a bit more, it would be nice to have etcher in package managers, but only if it can be done without a large impact on our release flow, at least for the time being.

On 25 Sep 2016 5:16 p.m., "Juan Cruz Viotti" notifications@github.com wrote:

Hi @imrehg https://github.com/imrehg , @dlech https://github.com/dlech

After some discussion, mainly with @alexandrosm https://github.com/alexandrosm , we decided the Etcher GUI will not provide official standard distribution packages (e.g: .deb, .rpm), and will focus entirely on AppImages, at least for the time being.

As a solution for the AppImages execution permissions problems, we can ship a zip containing the AppImage and potentially the user documentation file. Zip will make sure the execution bit is re-added when expanding.

That being said, we do want to provide packages for the Etcher CLI, so I'll close this issue and we can continue discussing Etcher CLI packaging on #355 https://github.com/resin-io/etcher/issues/355.

@glensc https://github.com/glensc

i guess you can attach the source tarballs to github release page as well: https://github.com/resin-io/etcher/releases

We've considered this as well, however the analytics that we get from GitHub Releases are not enough for us at the time being.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/resin-io/etcher/issues/632#issuecomment-249456042, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLUCOEB3Z2-ipCnOsPW07Mot51AvAQjks5qtw7kgaJpZM4Je3im .

[imrehg]

@alexandrosm I thought that was the discussion above, figuring out how to create packages with minimal-to-no changes in existing setup. :) Hence it would have been nice to just keep the issue open as a place to collect ideas. I guess it won't stop people working on it, anyways.

[alexandrosm]

I think we can reconsider the closure, especially since people are organically interested in stepping in.

On 25 Sep 2016 9:02 p.m., "Gergely Imreh" notifications@github.com wrote:

@alexandrosm https://github.com/alexandrosm I thought that was the discussion above, figuring out how to create packages with minimal-to-no changes in existing setup. :) Hence it would have been nice to just keep the issue open as a place to collect ideas. I guess it won't stop people working on it, anyways.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/resin-io/etcher/issues/632#issuecomment-249474933, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLUCKEj-j4n5JMiVsMr-w488pyiPw5qks5qt0PBgaJpZM4Je3im .

[jviotti]

Sounds good to me, lets keep it open then!

[dlech]

:tada: An unofficial debian package is now availible: https://github.com/dlech/etcher/wiki

Feel free to open issues at https://github.com/dlech/etcher/issues with suggestions or problems or edit the wiki.

[jviotti]

Hi @dlech,

Amazing news! I'll give it a go tomorrow! How do you feel about sending a PR with your changes to the main repo, so we can publish .deb packages as part of every release?

[jviotti]

Works great in my Ubuntu 16.04 👍

[dlech]

FYI, I have updated the etcher-electron package on in bintray repo for beta 16. I did this using the changes that have been merged into official Etcher.

So, if resin.io want to start publishing debian packages, I think it is good to go. If you need help with a deployment script, let me know.

[jviotti]

@dlech Awesome, thanks a lot!

So, if resin.io want to start publishing debian packages, I think it is good to go. If you need help with a deployment script, let me know.

Yeah, sounds good. We have a Bintray account, so if you could give me a hand with a standalone deployment script, I'd really appreciate it!

[dlech]

There are some administrative things to take care of first. I'm looking at https://bintray.com/resin-io, is this correct?

You will need to manually setup a repository and then a package in the repository (this is a one-time setup). What to name things will have an effect on what people will have to use in apt. There is the line to add to /etc/apt/sources.list.d/etcher.list. Mine looks like this:

deb https://dl.bintray.com/dlech/etcher-deb precise main

And here is where each name comes from:

deb https://dl.bintray.com/<user/org>/<repository> <distribution> <component>

So, for you, <user/org> will be resin-io (if I have guessed the correct account).

I think it would make sense to just have a single Debian repository for resin.io as a whole. So, you will need to create a new repository of type Debian. I would recommend naming it debian.

You will need to decide if you want to use the bintray GPG key or your own key. If you want to use your own, you need to create the key and set it up in bintray.

Then, in this new repository, you will need to create a new package. I would suggest calling it etcher-electron to match the name of the Debian package itself.

Once you have done that, there will be a "Set me up" button on the page for the package that you just created. This will have the curl commands needed to upload the .deb to bintray. This is where you choose the <distribution> and <component>. You can pick anything you like for these. <distribution> is traditionally a name like jessie or xenial. So, you could pick something like wheezy or precise to indicate the lowest supported OS version. Or, I suppose you could make up your own name, for example resin. The <component> is usually main, but I would suggest making it etcher. This way, resin.io could use this repository for other project besides Etcher, but people could opt into only the projects they want.

So, in the end, you might have the following the people will need to save as /etc/apt/sources.list.d/etcher.list:

deb https://dl.bintray.com/resin-io/debian wheezy etcher

[lurch]

I know almost nothing about Debian packaging, and even less about sources.list ;-) Would choosing a <distribution> of wheezy allow all Ubuntu users to install it too? Does this relate to PPAs on Ubuntu, or are they something different entirely?

Edit: And where do the different arches (x86 / x64) fit into this?

[dlech]

Would choosing a <distribution> of wheezy allow all Ubuntu users to install it too?

<distribution> is just a name that has to be matched, so it can be anything. You can use wheezy in your sources.list file and it will work just fine on Ubuntu.

Does this relate to PPAs on Ubuntu, or are they something different entirely?

I suppose technically, PPAs are debian package repositories that are hosted on launchpad.net (and therefore targeted for Ubuntu), but some people use the term PPA in a more general sense to mean any debian package archive that host just a few packages rather than a whole linux distribution.

So, this relates to PPA in that we are hosting our own package repository, just not on launchpad.

And where do the different arches (x86 / x64) fit into this?

The arch is inherent in the debian package, so we will be uploading one *_amd64.deb and one *_i386.dev for each release. apt knows the arch of your computer, so it knows which package to use. Or if you want to install the 32-bit version on a 64-bit machine for some reason, you can apt-get install etcher-electron:i386.

[jviotti]

I'll take care of this today.

is just a name that has to be matched, so it can be anything. You can use wheezy in your sources.list file and it will work just fine on Ubuntu.

What about naming it "stable"? Putting a distribution release name here is very confusing, and using a naming convention like "stable"/"development"/"alpha" or something like that would allow us to push continuous builds for testers.

[lurch]

Something else that occurred to me last night: if/when Etcher implements auto-update functionality, I guess that ought to be disabled in the Debian package, as we'll want people to update using apt-get?

[jviotti]

@lurch Exactly. Currently the debian packages @dlech built define ETCHER_DISABLE_UPDATES=1 to disable update checks.

[jviotti]

deb https://dl.bintray.com/<user/org>/<repository> <distribution> <component>

Since we will be publishing an etcher CLI package soon, I propose the following convention:

# GUI
 deb https://dl.bintray.com/resin-io/etcher stable etcher-electron

# CLI
 deb https://dl.bintray.com/resin-io/etcher stable etcher-cli

I think its cleaner to create different repositories for different applications we might want to publish.

[lurch]

IMHO (and again with the caveat that I don't know much about Debian packaging) it does make sense to have separate .deb files for etcher-electron and etcher-cli, but I don't think it makes sense to store them in separate repositories.

[jviotti]

@lurch I might be confusing things, but in the way I outlined above, we create a single etcher repository containing all packages. The user can pick which they like to install using the corresponding component, so the etcher package contains two components: etcher-electron and etcher-cli.

[lurch]

And regarding the distro, I guess Etcher is unusual in that the same binary works on multiple different distros, and doesn't need different versions of the binary to link against different versions of system libraries on different distros?

@lurch I might be confusing things, but in the way I outlined above, we create a single etcher repository containing all packages. The user can pick which they like to install using the corresponding component.

Sorry, I got confused by your "I think its cleaner to create different repositories for different applications we might want to publish." statement, where I guess you actually meant components rather than repositories. Even then, I'm still not sure it's worth having etcher-electron and etcher-cli in different components. Perhaps component is a better place for the "stable"/"development"/"alpha" ? shrug I defer to @dlech 's greater knowledge.

[jviotti]

And regarding the distro, I guess Etcher is unusual in that the same binary works on multiple different distros, and doesn't need different versions of the binary to link against different versions of system libraries on different distros?

Yeah, exactly. This is the reason why I prefer making "distribution" something like "stable"/"development" rather than an actual GNU/Linux distribution name.

Sorry, I got confused by your "I think its cleaner to create different repositories for different applications we might want to publish." statement, where I guess you actually meant components rather than repositories. Even then, I'm still not sure it's worth having etcher-electron and etcher-cli in different components. Perhaps component is a better place for the "stable"/"development"/"alpha" ? shrug I defer to @dlech 's greater knowledge.

Hm, I guess having them in different components is indeed a bit weird, however if we publish both etcher-electron and etcher-cli under the same component, the component "history" will be even more confusing, since it will contain entries from two different packages, and we will also need to somehow clarify which is which on each version description.

[jviotti]

As far as I can see, the component is what its used to signal the release type (e.g: stable/development). In that case, we could adopt something like:

deb https://dl.bintray.com/resin-io/etcher generic stable

A "generic" distribution name would mean it works everywhere.

[lurch]

the component "history" will be even more confusing

Is there even such a thing as "component history"? Surely the only thing that matters is the package history.

[dlech]

:+1: for <distribution> names of stable, beta, alpha, etc. :-1: for separate components for the two different Etcher packages. I think many people with want to install both packages, and it will be annoying to have to specify two components.

Then you can provide a etcher.list file as follows:

deb https://dl.bintray.com/resin-io/debian stable etcher
# uncomment the line below to enable the beta channel
#deb https://dl.bintray.com/resin-io/debian beta etcher

[dlech]

Or if you want separate repositories, then...

Then you can provide a etcher.list file as follows:

deb https://dl.bintray.com/resin-io/etcher stable main
# uncomment the line below to enable the beta channel
#deb https://dl.bintray.com/resin-io/etcher beta main

[dlech]

The problem with using "etcher" as the repository name though is that it will be Debian only. If you ever want to add another "etcher" repository on bintray (for docker or npm or whatever), it will have to have a different name. This is why I named the repository etcher-deb on my account.

[lurch]

I guess the only reason to have etcher-electron and etcher-cli in different components / distributions would be if etcher-cli does have version-specific dependencies on external system libraries (which I guess is a possibility, given that #355 hasn't been nailed down yet).

[jviotti]

The problem with using "etcher" as the repository name though is that it will be Debian only. If you ever want to add another "etcher" repository on bintray (for docker or npm or whatever), it will have to have a different name. This is why I named the repository etcher-deb on my account.

Fair enough. You can give this a go by adding the following to /etc/apt/sources.list:

deb https://dl.bintray.com/resin-io/debian stable etcher

I'll PR the script to publish to Bintray very soon.

[jviotti]

@dlech Do you know if there is a way Bintray can automatically infer the package version and architecture from the file name?

[lurch]

When I did sudo apt-get update

W: GPG error: https://dl.bintray.com stable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 379CE192D401AB61

got printed at the bottom.

When I did sudo apt-get install etcher-electron:

The following NEW packages will be installed
  etcher-electron
0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.
Need to get 48.7 MB of archives.
After this operation, 223 MB of additional disk space will be used.

Wow, is it really supposed to be that big??!

But it all installed and seems to work fine :+1:

[jviotti]

W: GPG error: https://dl.bintray.com stable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 379CE192D401AB61

Hm, weird. I configured Bintray to sign using its own GPG signature, but that might not be working very well, however I don't get that error on my machine at all. I'll investigate.

Wow, is it really supposed to be that big??!

Yeah, size is one of the biggest complaints about Etcher. All of it comes from the Electron framework. There are usually on going conversations on that project about how to reduce the final size, but there is not much we can do at the moment.

[dlech]

Do you know if there is a way Bintray can automatically infer the package version and architecture from the file name?

The version and arch are part of the file name, but Bintray does not look at these. Instead, you specify all of the information in the upload command. For example....

curl -T <FILE.deb> -udlech:<API_KEY> https://api.bintray.com/content/dlech/etcher-deb/etcher-electron/<VERSION_NAME>/<FILE_TARGET_PATH>;deb_distribution=<DISTRIBUTIONS>;deb_component=<COMPONENTS>;deb_architecture=<ARCHITECTURES>

[lurch]

I guess I probably need to import the bintray GPG key, @dlech ?

[dlech]

@lurch, yes, instructions are here.

[jviotti]

That sucks, since it means we have to either write code to inspect the .deb package, or make the publish script take options for the architecture and the version.

[lurch]

Thanks @dlech , no warnings from sudo apt-get update now :-)

@jviotti Can't the publish script determine the architecture and version from the filename? (sorry if I'm misunderstanding things!)

[jviotti]

@lurch Yeah, looks like we're going to have to do that.

[dlech]

It's easy, file name is etcher-electron_1.0.0~beta.16_amd64.deb, so remove the .deb and split on _.

[dlech]

oh, and change ~ back to - if you like.

[jviotti]

oh, and change ~ back to - if you like.

I was just going to ask about that :) Do you think we should use the tilde or the hyphen? Is the tilde the "right" way to do it in the debian world?

[lurch]

FILENAME=etcher-electron_1.0.0~beta.16_amd64.deb
echo $FILENAME
BASENAME=`basename $FILENAME .deb`
VERSION=`echo $BASENAME|cut -d_ -f2`
ARCH=`echo $BASENAME|cut -d_ -f3`
echo $VERSION
echo $ARCH

:-)

[dlech]

The ~ is needed in the debian package to make sure beta versions are < non- beta versions. But when it comes to publishing on bintray, it doesn't matter what the bintray version is, so if you want to change it back to - for <VERSION_NAME> in the upload command, it is not problem - just don't change it in the file name.

[dlech]

I am in favor of - since that is the way the version is written everywhere else.

[jviotti]

@lurch Oh, your cut approach is way better than the tr + awk one I was currently developing :)

The ~ is needed in the debian package to make sure beta versions are < non beta versions. But when it comes to publishing on bintray, it doesn't matter what the bintray version is, so if you want to change it back to - for in the upload command, it is not problem - just don't change it in the file name.

Sounds good, I'll change it to a hyphen!

[lurch]

Why does /usr/share/etcher-electron/LICENSE say:

Copyright (c) 2016 GitHub Inc.

...

?!

[jviotti]

Maybe Electron its adding its own license (Electron its made by GitHub)?