Add Dockerfile

[sirikid]

:^)

[doublep]

I actually never used Docker myself. What is this for? Is it needed for some projects that use Eldev? How would it be used?

[sirikid]

This is more an invitation to discussion than a ready-made solution. I used eldev in my project, so I had to add it to CI.

At first I installed eldev at every run (CI config below). Dead simple, I hope the cons are obvious.

# .drone.yml -- first attempt
kind: pipeline
type: docker
name: default

steps:
  - name: lint
    image: alpine
    commands:
      - apk add --no-cache curl emacs-nox
      - curl -fsSL https://raw.github.com/doublep/eldev/master/bin/eldev > eldev
      - chmod a+x eldev
      - ./eldev -dt compile
      - ./eldev -dt lint

To cache the binary and make it system-wide I made and published (my first) docker image:

# .drone.yml -- custom docker image
kind: pipeline
type: docker
name: default

steps:
  - name: compile
    image: sirikid/eldev
    commands:
      - eldev -dt compile

  - name: lint
    image: sirikid/eldev
    command:
      - eldev -dt lint

[sirikid]

Summing up: it is for CI and those who don't want to install eldev system-wide. The image will exist anyway, but I think it should be developed together with eldev.

[doublep]

At first I installed eldev at every run (CI config below). Dead simple, I hope the cons are obvious.

Yeah, but it's not that terrible. In a larger project you'd have several dependencies and those have to be installed before running tests anyway. You are only sparing one Emacs package installation per CI machine by using the Docker image.

# Initiate bootstrap RUN eldev version

The problem here is that bootstrapped Eldev package will be fixed in the image, as far as I understand it. When a new version of Eldev is released, users of such image will not receive it automatically.

 - curl -fsSL https://raw.github.com/doublep/eldev/master/bin/eldev > eldev
 - chmod a+x eldev
 - ./eldev ...

In the documentation there are several one-liners for different CI services that install Eldev and make it accessible from $PATH in one go. Though I don't know which CI you are using,

[sirikid]

In a larger project you'd have several dependencies and those have to be installed before running tests anyway.

In such a project I would make another image, based on the eldev one, and cache dependencies in it.

The problem here is that bootstrapped Eldev package will be fixed in the image, as far as I understand it. When a new version of Eldev is released, users of such image will not receive it automatically.

Yep, this is how docker is. New image should be built and pushed to registry when version bumps. This can be automated tho.

In the documentation there are several one-liners for different CI services that install Eldev and make it accessible from $PATH in one go. Though I don't know which CI you are using,

I used Drone CI, not that it matters.

[doublep]

In such a project I would make another image, based on the eldev one, and cache dependencies in it.

How would you track dependency upgrades? Also, if the list of dependencies for your project changes, you'd need to rebuild the image. Though if you forget, Eldev would automatically install missing dependencies, but are you not trying to avoid that?

RUN apk add --no-cache emacs-nox

This means installing whatever Emacs version is available in the target distro. How would you test on different Emacs versions? Unlike for dependencies, this is considerably more important, at least a couple of the latest releases in the last stable branches.

From real CI run logs (this project has two dependencies installed from MELPA), we are talking about saving a few seconds per CI run. Do you think it's worth the additional hassle?

[sirikid]

How would you track dependency upgrades?

Idk, I don't upgrade dependencies unless necessary. I didn't pay attention before, but Eldev doesn't have a lock file. How is it?

Also, if the list of dependencies for your project changes, you'd need to rebuild the image.

CI will do it.

How would you test on different Emacs versions?

Fair enough. I guess the image should be based on silex/emacs then.

[doublep]

About Dockerfile: how would someone use it? If there is no automated way (i.e. basically only way is to copy into your project and modify as needed), I feel like a better place would be the documentation, e.g. another subsection in "Continuous integration" section with example Dockerfile.

Eldev doesn't have a lock file. How is it?

I have thought about adding one, especially because of Flycheck. But there are problems I don't know how to overcome. First, Flycheck kill -9 checkers it doesn't want anymore, meaning in case of Eldev (i.e. flycheck-eldev) there would be problems with "lock file is there, but no-one manages it anymore", i.e. false locks often. Second, even if Flycheck sent SIGTERM or some other signal that could be processed, handling of kill signal in Emacs sucks. The most I could do is perhaps to erase all the installed dependencies and unlock if killing happens in the middle of installing dependencies, because there is apparently no way to ensure that dependency installation runs to the end or is otherwise atomic.

If you see a solution, please outline it. Or feel free negotiating with Emacs upstream (I gave up already, not the first time I write something only to get ignored, so fuck them).

[sirikid]

About Dockerfile: how would someone use it?

1) Use it as a base image for local one:

# example-repo/Dockerfile
FROM doublep/eldev:27.1
...

2) Use it in CI

# example-repo/.drone.yml
steps:
  - name: prepare
    image: doublep/eldev:27.1
    commands:
      - eldev prepare

I have thought about adding one, especially because of Flycheck. But there are problems I don't know how to overcome. First, Flycheck kill -9 checkers it doesn't want anymore, meaning in case of Eldev (i.e. flycheck-eldev) there would be problems with "lock file is there, but no-one manages it anymore", i.e. false locks often. Second, even if Flycheck sent SIGTERM or some other signal that could be processed, handling of kill signal in Emacs sucks. The most I could do is perhaps to erase all the installed dependencies and unlock if killing happens in the middle of installing dependencies, because there is apparently no way to ensure that dependency installation runs to the end or is otherwise atomic.

I meant lock file for dependencies, like mix.lock.

I had a little chat with @silex, he'll later include eldev in his image, see thread.

[doublep]

FROM doublep/eldev:27.1

I presume doublep/eldev is the GitHub repository name. How is 27.1 processed? Is Dockerfile looked up in repository root?

I meant lock file for dependencies, like mix.lock.

Then please explain, I don't know what this is.

I meant a lock file that would prevent concurrently executed Eldev processes from messing up. But I have outlined above why I don't see a good way to implement it currently.

I had a little chat with @Silex, he'll later include eldev in his image, see thread.

I feel this would be a better way. E.g. Eldev doesn't try to replicate EVM functionality, so why should it provide Docker images with different Emacs versions? Then this PR could be turned into documentation changes that describe such external images.

[sirikid]

I presume doublep/eldev is the GitHub repository name.

No, it's a user/image on Docker Hub. It is a registry from the creators of Docker. Many people develop images on Github and host them on Docker Hub. E.g. https://github.com/silex/docker-emacs and https://hub.docker.com/r/silex/emacs

How is 27.1 processed?

It's a tag, you can choose whatever you like, but it makes sense to use the emacs version for it.

Then please explain, I don't know what this is.

The lock file fixes dependency versions. Thus helps maintain reproducibility of builds because each dependency has a unique hash, even if the version is not unique. Thinking about it now, it might not be Eldev's job.

[doublep]

Tell me when @Silex does that, so I can add relevant links in the documentation.

[Silex]

A bit swamped between new-fatherhood and work, but I'll try to get the image with eldev going soonish. :sweat_smile:

[Silex]

Ok eldev was added to the images.

[sirikid]

Thanks, will try it asap

[Silex]

I realise I could bootstrap eldev instead of letting it "freshly installed". Would it make sense? (I mean, simply call eldev --version or whatever so it's ready to be used).

The advantage I see is that the image can then be used without internet access.

[sirikid]

@Silex yes, it definitely makes sense. Btw, alpine image is broken and I've already encountered this error.

[Silex]

@sirikid: thanks. Do you know what is broken? I mean, Emacs seems to work fine, I can install packages and everything. I see the recursive load but I don't really know where to investiguate.

Alpine images are a pain to make them work, I look at their repository and try to mimick the minimal amount of patches but still...

With my -ci refactor I could base all on ubuntu and have the images be quite small (500MB). We'll see if people really use alpine on the thread on my repo.

[doublep]

I realise I could bootstrap eldev instead of letting it "freshly installed". Would it make sense? (I mean, simply call eldev --version or whatever so it's ready to be used).

The advantage I see is that the image can then be used without internet access.

With pre-bootstrapped Eldev it might non-obvious for users that they might use a non-up-to-date Eldev.

I think the best way would be to combine the two approaches: images come with bootstrapped Eldev, but upon installation of an image, eldev upgrade-self would be run automatically, at least if Internet is accessible (if nothing newer is available, this would be effectively a no-op, simply querying stable.melpa.org once, but then doing nothing). But I have no idea how feasible it would be to achieve that.

Another approach would be to set up some automatic rebuilding of images if a new version of Eldev is released, but again, I don't know how feasible this is. Maybe @sirikid knows how to do that, as he mentioned this.

[doublep]

By the way, can someone provide an example (e.g. a simple link) to a project that uses *-eldev images, so that I have something to base documenation changes on? Also, it would be nice to know if images can be used anywhere or only on particular CI servers, because I have zero experience with this.

[Silex]

@doublep: too early for that. Atm I just added eldev to the dev images, but later on I plan on refactoring them to be smaller and only contain Emacs + CI tools + eldev, under a different name (emacs:27.1-ci-eldev ?)

About your images questions, I suggest you simply get more familiar with docker & containers, but the basic idea is that it's "chroot on steroids" aka a lightweight VM that can run anywhere a linux kernel is (and nowadays even on windows). The basic idea is that the whole OS is contained in the image so you never run in dependencie issues and the whole "dev setup" is included so deployement is easy.

[sirikid]

@doublep I used it in my project. CI configuration, builds.

[doublep]

too early for that

@Silex: sorry, for what exactly?

Do I understand it correctly that currently pre-bootstrapped Eldev is included in *-dev images, but you later plan to split them per tool and there would be *-eldev images in addition to *-cask etc.?

[Silex]

@doublep: too early to show examples using *-eldev images (which is what you asked). For examples using the -dev images I think @sirikid answered you.

And yes you understood correctly, I'll refactor the images into something lighter and more ci oriented where eldev cask etc will be part of the image name.

[Silex]

@doublep: oh, and sorry, I see you want to move this forward but I'm just not ready yet. I'll try to advance tomorrow.

[Silex]

I have all sorts of problems with alpine image. I'll revert the eldev inclusion in alpine images for now.

[doublep]

Hm, any particular problems caused by Eldev? It is only a small Shell script plus Elisp, no?

[Silex]

@doublep: the alpine images are broken in a weird way that eldev version reveals (see https://cloud.drone.io/sirikid/iex-mode/4/1/2). I'm sure that it means emacs itself is broken somewhere, but when you run it it seems to work fine.

I am trying to upgrade the alpine images from 3.9 to 3.12 in the hope that it fixes the problem.

You have to understand that alpine images are a bit trickier than ubuntu images, for some reasons I often run into weird infinite loops when building. I assume it's because of musl libc and docker restricted environment. If you look at alpine's repo we see they apply all sorts of patches.

Right now eldev is still included in the ubuntu images.