Alpha testing against existing projects

[akirak]

Now that elinter provides a GitHub action, I will start experimenting with it against existing projects developed by others. This will let me learn how people structure their projects and identify missing features and/or design flaws.

I consider this process a kind of acceptance testing, but it is going to be held by me, so it is alpha testing. It's still alpha, and there can be API changes. To actually replace existing Makefile/Cask-based solutions, it has to be reliable. After reaching beta, I may send PRs to authors of projects that can successfully adopt elinter, but I won't do that for now.

I will use Emacs Lisp projects created by others as input, preferably well-known ones. I will start with simple targets and gradually increase complexity. Variety also matters, because there seems to be no single agreed-upon way to organize Emacs Lisp projects right now.

Procedure

1. Fork a repository containing source code of Emacs Lisp packages.
2. Create elinter branch in the repository and install recipe files using elinter.el.
3. Check if linting and testing (if any) runs locally.
4. Create a GitHub workflow to run CI (example.
5. Push to GitHub and trigger the action.
6. If there is an existing Makefile for CI, it would be better to rewrite it using elinter to see if it can reduce configuration code.
7. Report the result to this repository. Include a link to the action page.

Feedback

• If there is an existing CI, does it produce the same result?
• If there is an existing CI, does elinter reduce the configuration?
• Is there any feature request or room for improvement in the API?

[akirak]

Example: lsp-dart

• Branch: https://github.com/akirak/lsp-dart/tree/elinter
• Action page: https://github.com/akirak/lsp-dart/actions?query=workflow%3ACI

Feedback:

• It successfully ran locally.
• On CI, experimental checks by melpazoid are enabled and they fail. I've fixed most of the errors, but it is impossible to fix the http warning.
• Because the linting fails, tests were not run on CI.
• It can simplify the Makefile to a great extent: https://github.com/akirak/lsp-dart/commit/b2c32d27809996ade0c200bec5d532b11dbd743a
• The user may want a feature to create a Git tag based on the library header.

Because the experimental checks of melpazoid can produce a false positive, it is necessary to allow failures of those checks. There could be a couple of options:

• Make melpazoid produce warnings, not errors, and exit with zero.
• Allow turning off the experimental checks in the action.

Tests are not run if the preceeding linting step fails. It's probably better to run tests unconditionally in the action. Thus the action needs an option to specify a type of test to run.

[akirak]

Maybe an issue template is desired for this process, but I'm afraid it can produce too many tickets.

[akirak]

Also note annotations doesn't work as expected. They should point to individual error locations.

[akirak]

I've changed the action to allow failures from melpazoid. Now lsp-dart passes all checks: https://github.com/akirak/lsp-dart/runs/999222903?check_suite_focus=true

Running tests is still a separate step from the main action. I think it's okay.

[akirak]

reformatter.el

• Branch: https://github.com/akirak/reformatter.el/tree/elinter
• Workflow result: https://github.com/akirak/reformatter.el/actions?query=workflow%3A%22elinter+CI%22

Feedback:

• shfmt is required for testing, but because tests are run in pure shells by elinter, it's impossible to make the dependency available in testing. Non-Emacs dependencies must be specified somehow or allow testing in impure shells.

Received the following warnings generated by melpazoid (fixed):

  - reformatter.el#L105: It’s safer to sharp-quote function names; use ‘#’‘
  - reformatter.el#L259: It’s safer to sharp-quote function names; use ‘#’‘
  - reformatter.el#L260: It’s safer to sharp-quote function names; use ‘#’‘
  - reformatter.el#L287: It’s safer to sharp-quote function names; use ‘#’‘
  - reformatter.el#L9: Prefer ‘;; Version‘ over ‘;; Package-Version‘ (MELPA automatically adds ‘Package-Version‘)

Encountered the following errors in Emacs 24.5. I had to rewrite some parts of my elisp code. I also disabled checkdoc, since checkdoc-file function is unavailable:

• Eager macro-expansion failure: (error "Unknown upattern(quote warning)'")`

Also failed to build melpazoid.el in Emacs 24.3, but it was not due to melpazoid itself, but because the build function in Nixpkgs uses functions that are not available in this version of Emacs. I am not sure if I need to work around this. It would be better to avoid linting using older versions of Emacs.

[akirak]

dhall-mode

• Branch: https://github.com/akirak/dhall-mode/tree/elinter
• Workflow result: https://github.com/akirak/dhall-mode/actions?query=workflow%3A%22elinter+CI%22

This package didn't have CI. Linting produced some errors. After fixing those errors, the new CI passes.

[akirak]

Now you can specify a list of Nix packages available in the testing environment using --from-nixpkgs option, e.g.

elinter -t --ert --from-nixpkgs shfmt -- reformatter-tests.el

Also Emacs 24.3 is skipped (see the comment in #29), and reformatter.el now passes the CI.

[akirak]

I've added a GitHub workflow to display all examples in a single job (#32).

[akirak]

I've created a separate repository for demonstration: https://github.com/akirak/elinter-demo

From now on, I'll add examples to the workflow in the repository.