Support new travis docker images

[fommil]

which don't allow sudo... so emacs must be installed manually

[rolandwalker]

Hi!

Interesting that Travis is supporting Docker images, but not custom Docker images. Doesn't compiling Emacs every time overwhelm the speed advantage of using Docker in the first place?

[fommil]

our problem with the older images is that they keep killing our tests when they use more than about half a thread.

[rolandwalker]

Don't get me wrong, I am in favor of doing something about this.

It's just that the naive solution, for the Docker infrastructure, is a custom Docker image with the desired Emacs. And Travis says they will let us do that, but there is no timeline.

[fommil]

yeah, it's a real pain and would solve a lot of our CI issues.

There are two workarounds:

1. compile all the things! (for each branch)
2. manually download/extract the .deb files for the relevant version of emacs and set the correct PATH/LD_LIBRARY_PATH/etc

We've used technique 2 for full JDK installs (we need the src.zip in our tests)

[fommil]

FYI https://github.com/travis-ci/travis-build/pull/372

[dgutov]

Also FYI:

https://github.com/travis-ci/apt-source-whitelist/issues/25 https://github.com/travis-ci/apt-package-whitelist/issues/367

[swsnr]

And FYI as well: For Flycheck, I'm now building Emacs during Travis builds, since Travis is so slow to accept additional PPAs, and there's no good PPA for Emacs anyway—cassou is dead, ubuntu-elisp poorly maintained.

When configured with minimal features, it builds quite fast actually. The Tarballs take about two minutes. Git head takes longer, but mostly stays under ten minutes. That's fast enough that I didn't bother to make use of Travis‘ caching facilities.

You can take a look at the corresponding Makefile of Flycheck. Please feel free to copy as you like. If you'd like to use it, I think it'd be a good idea to move the Makefile into a separate repo, from we could all wget it in our .travis.yml files.

[dgutov]

I'm now building Emacs during Travis builds

That's pretty radical. I'm still using Cassou's PPA on Travis with an eye towards migrating to ubuntu-elisp. I was not aware of it being poorly maintained (how so?).

You can take a look at the corresponding Makefile of Flycheck.

Thank you, I'll do that sometime.

[swsnr]

@dgutov I'm not sure why that is “radical”, but as things stand now it's the only way to use Travis CI‘s new infrastructure for Emacs Lisp.

As for ubuntu-elisp, did you look at their build dates, and the frequency of builds, particularly for older Ubuntu versions?

[dgutov]

As for ubuntu-elisp, did you look at their build dates, and the frequency of builds

Ah yes. Apparently they explain that by infrequent updates to the LP Emacs fork: https://www.reddit.com/r/emacs/comments/2w8p5b/ubuntu_emacs_lisp_team_ppa_didnt_update_for_one/

, particularly for older Ubuntu versions?

That is indeed troubling, in the context of using it for Travis.

[tripleee]

I have raised this issue with the Launchpad emacs-lisp team (where nominally I am even a member; but I have no involvement in the PPA, or much anything with Ubuntu recently) -- I hope the PPA could provide a reasonably recent emacs-snapshot for Ubuntu 12.04 as long as it is supported. Maybe emacs24 could be added to the mix as well.

[swsnr]

@dgutov I think that the explanation—its age, more precisely—only confirms my point: They're aware of this situation for at least seven months, but still haven't changed their build procedure, let alone added a simple notice to the PPA description?

Don't get me wrong, I'd love to have a proper source for Emacs binaries, but so far I've found none.

[dgutov]

@lunaryorn I don't disagree. Building Emacs each time you need to run tests feels quite wasteful, though.

@tripleee Thank you.

[swsnr]

@dgutov Well, as I said, when stripped down, it actually builds quite fast. ./configure --without-all --without-x --with-x-toolkit=no does wonders to the build time :grin:

[rolandwalker]

Thanks for this update/advice @lunaryorn. Building from source does seem like the most reliable way to go.

[tripleee]

The Launchpad PPA has been updated to include binaries for Ubuntu 12.04 / Precise as well as 14.04 / Trusty.

The issue with the bzr imports still remains. I'm thinking perhaps Travis itself should be building a canonical (no pun intended) PPA from the upstream Emacs sources directly instead.

[swsnr]

@tripleee I don't think that this is Travis’ job—and in any case, they won't do that. It's our job as a community to do that, but so far we haven't succeeded.

I'm not sure why, though, because building Emacs continuously doesn't seem that hard to me—Emacs for OS X has been doing that for ages. I would appear that the infrastructure we currently try to use—namely PPAs—is not appropriate for this purpose.

Has anyone tried to build Emacs on OBS? Alternatively, Emacs for OS X runs their own server.

[tripleee]

@lunaryorn The reason I am suggesting for Travis to step in is because it seems rather haphazard to whitelist random PPAs on the grounds that they currently don't do anything malicious. I have created a prototype build job just now, and I'll be happy to contribute the results once I have the final parts in place, but do you trust me? Forever? Why?

Just like Travis provides vetted, stable, official versions of Ruby, GCC, etc, so should there be an official Emacs build IMHO. It's a platform for other tools, and it's extremely wasteful to have each project build its own custom version, and rather risky to rely on external PPAs for this.

[swsnr]

@tripleee I think there's no use in discussing here what Travis CI should do or should not do, without anyone from Travis being involved.

We can't change what Travis CI does, and I for my part do not want to passively wait until Travis CI supports Emacs out of the box.

Building Emacs is what works for Flycheck, but I'd be happy to learn about better solutions. I'm just not going to wait passively until Travis CI steps in and adds proper support for Emacs.

[npostavs]

Building Emacs is what works for Flycheck, but I'd be happy to learn about better solutions.

Ahem. I believe I have a better solution: build a Emacs in the Travis script of a dedicated repo, and upload the result as a github "release asset". Implementation is in .travis.yml and supporting travis-steps.sh file.

Example usage by yasnippet: Emacs install time 2.5s, total time 8s.

[fommil]

Fyi I don't use Travis anymore, I have an installation of drone (libre software) on my server machine and ensime devs contribute worker nodes. Our docker images (also libre) are available for all on docker hub have various versions of Emacs / openjdk preloaded.

[swsnr]

@npostavs I understand how that'd work for releases, but how would we ensure nightly builds of snapshot with this approach?

@fommil For Flycheck a privately hosted server is not an option. I don't want to come up for the costs alone not be responsible for its maintenance, and there's no model to share the costs and control reasonably among all participants without a lot of legal hassle.

[npostavs]

I understand how that'd work for releases, but how would we ensure nightly builds of snapshot with this approach?

A new build can be triggered by pushing to the dedicated emacs-travis repository. Alternatively Travis provides a button to rerun the test from the web, maybe this can be triggered by API too. For the initial test I built only 24.5, but any git reference in the Emacs repo can be retrieved.

Added some more versions: https://travis-ci.org/npostavs/emacs-travis/builds/104835135 Hmm, apparently version 23 needs a slightly different build process.

[fommil]

@lunaryorn if you write that generalised layer for ensime to use then you're welcome to use our build farm :wink:

[swsnr]

@npostavs Pushing to a repo or pressing a button means manual action, so no automatic nightly builds. Such setups tend to age very very quickly. Automation is essential for our setup.

@fommil Uhm, well… seriously? :confused:

[fommil]

@lunaryorn yup!

[fommil]

@lunaryorn you should be able to log in at http://fommil.com

There is currently no access control for repos so you can use it for any of your projects, just don't take the piss :wink:

some examples

• https://github.com/ensime/ensime-server/blob/master/.drone.yml
• https://github.com/ensime/ensime-emacs/blob/master/.drone.yml
• https://github.com/fommil/dotfiles/blob/master/.drone.yml

you can use your own docker images, but if you have an improvement to the images we already use then they are at

• https://github.com/ensime/ensime-docker/branches
• https://github.com/fommil/dotfiles/tree/docker

Same offer stands for anybody else providing software in the ENSIME stack! :smile:

[swsnr]

@fommil Well, I believe your offer is honest, but your precondition leaves a bad taste (hence the emoji). See, I'm not going to bargain my work for resources nor let myself being pressured into support for a particular client extension—I'm sure you didn't mean to say that, but that's how your precondition "if you write…" reads to me :confused:

I appreciate your offer, but Flycheck is not part of the Ensime stack.

[fommil]

Well as long as https://github.com/flycheck/flycheck/issues/774 is being considered, we will do what we can to help (I thought it was? Are the tags out of date?)

[swsnr]

@fommil It is considered, even planned, but it still feels like bargain.

I won't tie our access to build resources to whether we are part of someone else's stack and (implicitly) to how well we support a particular 3rd party extension. That'd set implicit expectations and assumptions on either side that won't do no good, in my opinion, and I want to keep control of our priorities.

If I were using your infrastructure under the precondition that Flycheck was part of your stack I'd feel obliged to priorise differently with regards to Ensime, and that's something I'd like to avoid.

Thank you for your offer, but I'm afraid I have to turn it down. No offense meant.

[npostavs]

Pushing to a repo or pressing a button means manual action, so no automatic nightly builds. Such setups tend to age very very quickly. Automation is essential for our setup.

Hmm, ideally it would be triggered by pushes to https://github.com/emacs-mirror/emacs/, but it seems a server would be required to forward the signal.

[Silex]

You might want to consider using https://hub.docker.com/r/silex/emacs for the docker images :-)

I'm in the process of building those automatically with travis, and will probably setup some kind of push monitoring on the main git repo to trigger rebuilds on pushes.

Feel free to suggest other branches/tags that I should build.

[Silex]

@dgutov: given how the discussion about docker images went on the ML, I prefer asking you here. I was originally going to propose including a .travis.yml in the main emacs repo, but I think that'd be very painful to manage (assuming it gets merged at all). Do you know who handles https://github.com/emacs-mirror/emacs? I was thinking about adding a webhook to this repository, which would help automatising the "builds on pushes" part.

Basically:

• The travis build keep the images up to date.
• One webhook at emacs-mirror that triggers the travis build on pushes.
• One webhook at hub.docker.com which triggers builds when the base image changes.

If adding a webhook at emacs-mirror is not possible, travis has basic "cron" support which should be sufficient, like build images every couple hours.

[npostavs]

Do you know who handles https://github.com/emacs-mirror/emacs?

I think this is a mystery, unfortunately: https://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00686.html, they've maintained complete radio silence.

travis has basic "cron" support which should be sufficient, like build images every couple hours.

Oh, neat! Looks like daily is the max resolution right now, but that should be good enough. I've enabled this for https://github.com/npostavs/emacs-travis.

https://github.com/npostavs/emacs-travis/releases/download/bins/emacs-bin-25.tar.gz and https://github.com/npostavs/emacs-travis/releases/download/bins/emacs-bin-master.tar.gz should now be updated daily.

[Silex]

Do you know who handles https://github.com/emacs-mirror/emacs?

I think this is a mystery, unfortunately: https://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00686.html, they've maintained complete radio silence.

Oh well. Maybe I'll try to send a new message asking the same thing (I joined the ML after this message), maybe someone missed the original message.