Better CI pipeline / project tools

victorolinasc commented 4 years ago

Current Emacs package development and its tools for CI pipelines have evolved a lot recently and I think that we could use some updating here to benefit from that.

WARNING: I am a complete noob at ELisp package development but someone interested on the subject for a while.

Things I think that could help us here:

Linters (checkdoc, relint, etc)
Different test tool not dependent on ert-runner like buttercup
Alternative package manager / build-tool like Eldev or makem.sh
Different CI setup/provider like setup-emacs with GitHub Actions

This issue is to start a, hopefully, health discussion on these matters and try to make this package more active in the meantime. I'll gladly try to help implement changes here if they are welcome.

axelson commented 4 years ago

I can't speak for everyone but these sound like good changes :+1:

Trevoke commented 4 years ago

All of these seem like good things. I'm trying eldev and buttercup, as of today, on a side project of mine. I wasn't aware of setup-emacs but I'll be curious to give it a shot as well.

tomekowal commented 4 years ago

Seems like a great idea! I only had the pleasure to use https://github.com/gonewest818/elisp-lint

victorolinasc commented 4 years ago

@Trevoke would love to hear some comments about Eldev... seems like a closer to mix tool than others.

jsmestad commented 4 years ago

I had to mess with a bunch of the Travis config in order to get the test matrix passing (see #459).

I probably butchered it 😞

Trevoke commented 4 years ago

ELDEV

You can see my usage of eldev here: https://github.com/Trevoke/org-gtd.el

The Eldev file looks like:

; -*- mode: emacs-lisp; lexical-binding: t; no-byte-compile: t -*-

;; Autodetermined by `eldev init'.
(eldev-use-package-archive 'gnu)
(eldev-use-package-archive 'melpa)

(setq eldev-test-framework 'buttercup)

(eldev-add-extra-dependencies 'test 'with-simulated-input)

BUTTERCUP

You can see the lone buttercup test I have here: https://github.com/Trevoke/org-gtd.el/blob/master/test/org-gtd-test.el

It looks like any BDD style framework - this test doesn't actually have an expectation because it was so much of a pain to get to where it is now but I think i can add one.

LINTING ET AL.

@tomekowal I don't know anything about elisp-lint, does it connect to package-lint, which is what melpa requests? (not that it's a requirement, it's probably fine if we have our own linting tool and we put it in the build)

https://github.com/melpa/melpa/blob/master/CONTRIBUTING.org#making-your-package-ready-for-inclusion

tomekowal commented 4 years ago

I don't know anything about elisp-lint, does it connect to package-lint, which is what melpa requests? (not that it's a requirement, it's probably fine if we have our own linting tool and we put it in the build)

Yes, it does https://github.com/gonewest818/elisp-lint#package-lint

package-lint does basic checks to ensure a package is ready for Melpa. elisp-lint does more checks like:

function declarations
docs
indentation
package-lint

Trevoke commented 4 years ago

Addendum - I am using a Github action on my org-gtd package to run the test suite with eldev:

https://github.com/Trevoke/org-gtd.el/blob/master/.github/workflows/test.yml

victorolinasc commented 4 years ago

I have the project running with Eldev locally (this is the easy part I guess). Tests run exactly as they are now without issues (except those that are already on the code base like expected failures and so on).

I plan to:

[ ] Narrow the support emacs version matrix (more on this below)
[ ] Try to make the format test that asserts failure not leave a temp file on the system
[ ] Switch fo GitHub Workflows and setup-emacs

About Emacs version support

We have a single test failing in emacs 24.5 which is quite odd to only fail on that version. We have a hack for Emacs 24.3 because of a bug in SMIE on that version.

I'd like to make a policy here about the versions we will support going on. The Elixir + Erlang combo of supported versions seems like very "aggresive". They usually target only the last 2 major versions. Current we support Emacs 24+ which I think is a somewhat old version by today's standard. 24.1 was release on 2012-06-10 while 25.1 on 2016-09-17.

There is no strong reason for us to deprecate support for Emacs 24 besides it being old and a bit buggier than newer versions. What I think we could do is to test only 25.1+ on CI from now on and leave the support for 24 as non-official. Hopefully all LTS distros out there have at least Emacs 25+ and version 24 would be as used as older Emacs versions in terms of percentage.

Hopefully this is not way too flawed of an argument. I think most packages are going with this strategy anyway. There will always be GIT history for those with older versions too as an option.

If that is okay I'll include that on this next PR

victorolinasc commented 4 years ago

Also, running the eldev lint command pops up hundreds of issues currently :) I guess this is expected and we can work on them progressively...

axelson commented 4 years ago

I am on board with dropping official support for Emacs 24, and your other planned improvements sound great :+1:

victorolinasc commented 3 years ago

Things here are done for now! Thanks all for your kind contributions!

elixir-editors / emacs-elixir