invalid-regexp error when compiling on emacs 28

stuart-little commented 3 years ago

I have emacs 28.0.50 compiled from source and upgrade dash.el regularly through emacs' package.el; it's currently at v20210704.1302.

Every time I upgrade I get

dash.el:551:1: Error: Invalid regexp: "Invalid content of \\{\\}"

I've navigated to that line (on my own system as well as in this repo; the line is identical) but don't see what the error refers to. I'm guessing it's propagating from elsewhere.

This only happens on the emacs 28 system. I'm running 27.2.50 on other machines (also compiled from source) and install dash.el in the same fashion, but never see the error there.

basil-conto commented 3 years ago

I have emacs 28.0.50 compiled from source and upgrade dash.el regularly through emacs' package.el

Same here (though in my config, my checkout of dash.el.git shadows the version installed by package.el).

Every time I upgrade I get

I can't reproduce this. Any chance of seeing the error backtrace after either starting Emacs with --debug-init or turning on M-x toggle-debug-on-error? Can you reproduce the error from emacs -Q? What is your full M-x emacs-version (including OS)? Please say in more details which steps exactly you take that give rise to the error.

I'm guessing it's propagating from elsewhere.

Yes, my first guess would be that it's unrelated to Dash.

stuart-little commented 3 years ago

What is your full M-x emacs-version (including OS)?

GNU Emacs 28.0.50 (build 5, x86_64-pc-linux-gnu, GTK+ Version 3.24.20, cairo version 1.16.0) of 2021-07-04

The OS is Ubuntu 20.04, running Linux 5.4.0.

I can't reproduce this. Any chance of seeing the error backtrace after either starting Emacs with --debug-init or turning on M-x toggle-debug-on-error? Can you reproduce the error from emacs -Q?

Please say in more details which steps exactly you take that give rise to the error.

The error triggers when I byte-compile the file

~/.emacs.d/elpa/dash-20210704.1302/dash.el

(with M-x byte-compile-file) but all I get for a trace is

Compiling file /root/.emacs.d/elpa/dash-20210704.130\
2/dash.el at Sun Jul  4 21:54:37 2021
Entering directory ‘/root/.emacs.d/elpa/dash-2021070\
4.1302/’
dash.el:551:1: Error: Invalid regexp: "Invalid conte\
nt of \\{\\}"

even with toggle-debug-on-error or starting with --debug-init.

This does not happen, though, if I start with the -Q flag: in that case the byte compilation goes through fine.

How can I get a more verbose trace for byte compilation? toggle-debug-on-error is apparently not sufficient..

stuart-little commented 3 years ago

Aha! Getting closer: I've isolated the problem to loading this simple emacs.el file:

(setq-default fill-column 9999999)

If I put that in my init file I get the error (when byte-compiling dash.el, as described above); if I don't, all's well.

basil-conto commented 3 years ago

Thanks for digging into this, but I still can't reproduce the error. Here's what I tried:

$ cd "$(mktemp -d)"
$ mkdir .emacs.d
$ cat << EOF > .emacs.d/init.el
(setq-default fill-column 9999999)
(setq-default package-archives '(("gnu" . "https://elpa.gnu.org/devel/")))
EOF
$ HOME="$PWD" XDG_CONFIG_HOME="$PWD/.config" emacs

M-x package-install RET dash RET
M-x byte-compile-file RET .emacs.d/elpa/dash TAB /dash.el RET

This succeeds silently:

Compiling file /tmp/tmp.H6y525NKKO/.emacs.d/elpa/dash-2.18.1.0.20210704.130240/dash.el at Mon Jul  5 12:19:01 2021
Entering directory ‘/tmp/tmp.H6y525NKKO/.emacs.d/elpa/dash-2.18.1.0.20210704.130240/’

M-x report-emacs-bug RET RET has the following to say about my Emacs:

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0, Xaw3d scroll bars)
 of 2021-07-04 built on tia
Repository revision: 2f2afa0b310bbce43a8703f5467b2638082abdd9
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Debian GNU/Linux 11 (bullseye)

Configured using:
 'configure 'CC=ccache gcc' 'CFLAGS=-Og -ggdb' --config-cache
 --prefix=/home/blc/.local --enable-checking=structs
 --with-x-toolkit=lucid --with-file-notification=yes --with-x'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
X11 XAW3D XDBE XIM XPM LUCID ZLIB

Important settings:
  value of $LANG: en_IE.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

basil-conto commented 3 years ago

How can I get a more verbose trace for byte compilation? toggle-debug-on-error is apparently not sufficient..

I don't know; maybe package or bytecomp are suppressing the error from propagating.

basil-conto commented 3 years ago

M-x report-emacs-bug RET RET has the following to say about my Emacs:

Same story with the GTK3 toolkit as in your configuration, FWIW:

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.16.0)
 of 2021-07-05 built on tia
Repository revision: 579b0c006e407aef1623f3b42d28b666426406c7
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Debian GNU/Linux 11 (bullseye)

Configured using:
 'configure 'CC=ccache gcc' 'CFLAGS=-O2 -march=native' --config-cache
 --prefix=/home/blc/.local --program-suffix=-gtk
 --enable-checking=structs --with-file-notification=yes
 --with-x-toolkit=gtk --with-x'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
X11 XDBE XIM XPM GTK3 ZLIB

stuart-little commented 3 years ago

For me M-x report-emacs-bug RET RET says:

In GNU Emacs 28.0.50 (build 5, x86_64-pc-linux-gnu, GTK+ Version 3.24.20, cairo version 1.16.0)
 of 2021-07-04 built on gilead
Repository revision: ed15f3954c04e2039a565ca0d0ff810519da8197
Repository branch: master
System Description: Ubuntu 20.04.2 LTS

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
LCMS2 LIBOTF LIBSELINUX LIBXML2 M17N_FLT MODULES NOTIFY INOTIFY PDUMPER
PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM
GTK3 ZLIB

Important settings:
  value of $LANG: C.UTF-8
  locale-coding-system: utf-8-unix

I tried again: if I

start clean with emacs -nw -q (so no init at all)
then eval (setq fill-column 9999999)
and then byte-compile ~/.emacs.d/elpa/dash-20210704.1302/dash.el

all is well. If, however, I change that setq function to setq-default, I get the same error I reported before.

basil-conto commented 3 years ago

Ah, I figured out why I couldn't reproduce it: I was insisting on installing Dash from GNU-devel ELPA instead of MELPA ;).

I can reproduce the error when Dash is installed from MELPA, so I'll try to dig deeper. Thanks.

stuart-little commented 3 years ago

Thanks! I'm happy you've reproduced it and will be curious to learn what's up, but a couple of things:

One point that confuses me is that the file I was byte-compiling lives in ~/.emacs.d/elpa/, which to me suggested ELPA (not MELPA). But I never paid much attention to what gets installed where, so I am not sure how much that path means.

In any case, for reference, you can find the specific file here. That should remove the ambiguity, if anyone else finds this.

basil-conto commented 3 years ago

One point that confuses me is that the file I was byte-compiling lives in ~/.emacs.d/elpa/, which to me suggested ELPA (not MELPA). But I never paid much attention to what gets installed where, so I am not sure how much that path means.

ELPA is the protocol, GNU ELPA and MELPA are archives that implement it (where MELPA is the larger community archive, and GNU ELPA, as the name suggests, is the smaller set of "blessed" FSF-copyrighted packages that are co-maintained by their original authors and GNU Emacs developers).
With respect to tagged releases and snapshots, GNU ELPA and GNU-devel ELPA correspond to MELPA Stable and MELPA, respectively.
By default, anything you install with the built-in package manager package.el lives under ~/.emacs.d/elpa/, regardless of which archive it came from. This is controlled by the user option package-user-dir; see (info "(emacs) Package Files").
The set of archives known to Emacs is determined by package-archives and related user options. For example, in my config it contains GNU-devel ELPA, NonGNU-devel ELPA, MELPA, and the Org archive, in descending order of package-archive-priorities. That way I can install Dash and similar packages from GNU ELPA but still let e.g. Magit fetch its dependencies from MELPA.

HTH, and sorry about the dump :).

stuart-little commented 3 years ago

HTH, and sorry about the dump :).

Certainly nothing to apologize to me about: it was all very instructive :).

basil-conto commented 3 years ago

Here's the culprit in Emacs 28:

*** The byte-compiler now warns about too wide documentation strings.
By default, it will warn if a documentation string is wider than the
largest of 80 characters or 'fill-column'.  This is controlled by the
new user option 'byte-compile-docstring-max-column'.

This feature is currently implemented using a regexp of the form ^.\{N,\}$ (see byte-compile--wide-docstring-p), where N is the larger of 80 or fill-column, which in this case is very large.

However, as of Emacs 27 the limit for regexp repetition is 65535:

** The limit on repetitions in regexps has been raised to 2^16-1.
It was previously limited to 2^15-1.  For example, the following
regular expression was previously invalid, but is now accepted:

   x\{32768\}

So with sufficiently large fill-column, byte-compile--wide-docstring-p constructs an invalid regexp.

I'll report this as a bug upstream, but in the meantime, you should be able to work around it by:

reducing your fill-column; or
setting byte-compile-warnings to (not docstrings).

I still don't know why I only saw this with MELPA, BTW.

stuart-little commented 3 years ago

Excellent, thanks!

I think maybe I can live with lines that are at most 65535 columns long.. :)

basil-conto commented 3 years ago

I'll report this as a bug upstream

Here's the ticket: https://bugs.gnu.org/49426

I think maybe I can live with lines that are at most 65535 columns long.. :)

To each their own ;).

Since it doesn't look like Dash can do much if anything about this issue, I'm closing it for now. Feel free to report back / reopen if anything else comes up.

magnars commented 3 years ago

Good job figuring this one out! FWIW, this is why I stopped testing my Emacs libs on nightly builds of Emacs. These kinds of regressions very rarely make it into the proper builds, thankfully.

basil-conto commented 3 years ago

Good job figuring this one out!

Thanks, it was a team effort :).

FWIW, this is why I stopped testing my Emacs libs on nightly builds of Emacs.

Yes, that's why our CI on GitHub allows for failures with the latest Emacs snapshot:

https://github.com/magnars/dash.el/blob/3bd52a45aa81a3aab0d02ece800042415669399a/.github/workflows/test.yml#L23-L25

I still don't know why I only saw this with MELPA, BTW.

This one turned out to be because GNU ELPA also packages the project's .dir-locals.el file, which includes a custom fill-column setting:

https://github.com/magnars/dash.el/blob/3bd52a45aa81a3aab0d02ece800042415669399a/.dir-locals.el#L4

By contrast, MELPA only packages dash.el and its manual.

stuart-little commented 3 years ago

By the way, for my own edification, I did a binary search to find the precise boundary for (in)valid column widths. It is indeed 65534/65535. In other words,

(setq-default fill-column 65534)

in my init file does not produce the error, but

(setq-default fill-column 65535)

does.

So that does it. This was indeed an excellent piece of detective work on your part @basil-conto :) Thanks!

basil-conto commented 3 years ago

You're welcome, and thanks!

I'll report this as a bug upstream

Here's the ticket: https://bugs.gnu.org/49426

The fix is now on Emacs master if you want to rebuild (and go back to a fill-column not limited by 16 bits ;).

Avoid invalid regexp in wide docstring check 044742bfe8 2021-07-06 18:56:15 +0100 https://git.sv.gnu.org/cgit/emacs.git/commit/?id=044742bfe8c7c22e303242c40e16fbe9e564727a

magnars / dash.el

invalid-regexp error when compiling on emacs 28 #383