jorgenschaefer / emacs-buttercup

Behavior-Driven Emacs Lisp Testing
GNU General Public License v3.0
362 stars 44 forks source link

spy-on test fails with native-compilation #236

Closed podiki closed 1 year ago

podiki commented 1 year ago

I believe this is #230 again. When building with Guix and using native compilation, the tests fail for spy-on. Here is the relevant part of the output:

The Spy
  `spy-on' function
    replaces a symbol's function slot (0.05ms)
    restores the old value after a spec run (0.05ms)
    allows a spied-on command to be executed as a command (0.09ms)
    can spy on autoloaded functions (0.82ms)
    can spy on non-existing functions (0.03ms)
    only accepts ARG for keywords that use it (0.16ms)
    works on native-compilation primitives  Expected `(spy-on 'buffer-file-name)' not to throw a child signal of `error', but it threw (native-lisp-load-failed "file does not exists" "/homeless-shelter/.emacs.d/eln-cache/28.2-bf9ead57/subr--trampoline-6275666665722d66696c652d6e616d65_buffer_file_name_0.eln") (142.85ms)
    will signal en error if
      used in before-all (0.08ms)
      used directly in describe (0.06ms)
  :to-have-been-called matcher
    returns false if the spy was not called (0.08ms)
    returns true if the spy was called at all (0.07ms)
  :to-have-been-called-with matcher
    returns false if the spy was not called at all (0.10ms)
    returns false if the spy was called with different arguments (0.10ms)
    returns true if the spy was called with those arguments (0.18ms)
  :to-have-been-called-times matcher
    returns error if the spy was called less than expected (0.10ms)
    returns error if the spy was called more than expected (0.09ms)
    returns true if the spy was called the expected number of times (0.08ms)
    use plural words in error message (0.09ms)
    use singular expected word in error message (0.11ms)
    use singular actual word in error message (0.09ms)
  :and-call-through keyword functionality
    tracks calls to the function (0.04ms)
    passes the arguments to the original function (0.07ms)
  :and-return-value keyword functionality
    tracks calls to the function (0.03ms)
    returns the specified value (0.07ms)
    works with strings (0.07ms)
    works with vectors (0.07ms)
    works with symbols (0.07ms)
    works with conses (0.07ms)
    works with lists (11.05ms)
    works with alists (0.06ms)
  :and-call-fake keyword functionality
    tracks calls to the function (0.02ms)
    returns the specified value (0.05ms)
  :and-throw-error keyword functionality
    throws an error when called (0.06ms)
  error-recording functionality
    records the function as called even if it throws an error (0.09ms)
    counts both successful calls and calls that threw errors (0.09ms)
    records args to the function whether it throw an error or not (0.12ms)
    records the signal thrown by a call to the function (0.12ms)

and backtrace of the guix build:

The Spy `spy-on' function works on native-compilation primitives

Traceback (most recent call last):
  buttercup-fail("%s" "Expected `(spy-on 'buffer-file-name)' not to throw a ...
  signal(buttercup-failed "Expected `(spy-on 'buffer-file-name)' not to thro...
FAILED: Expected `(spy-on 'buffer-file-name)' not to throw a child signal of `error', but it threw (native-lisp-load-failed "file does not exists" "/homeless-shelter/.emacs.d/eln-cache/28.2-bf9ead57/subr--trampoline-6275666665722d66696c652d6e616d65_buffer_file_name_0.eln")

Ran 282 specs, 1 failed, in 314.17ms.
buttercup-run failed: ""
make: *** [Makefile:12: check-buttercup] Error 255

This is for v1.30, not sure what is wrong since this was supposed to be fixed. Note that the homeless-shelter is noting that HOME is not set during the build/test (though I did try setting it to /tmp in the package build, perhaps a red herring here as it normally indicates something is relying on HOME).

Could be a Guix related issue with how the build is done, but I don't see what that is so far...

podiki commented 1 year ago

Ah, the usual trick of submitting a bug report reveals the answer, of course.

The problem is from 686b817 where the fix for #232 used the variable name for more recent Emacs than release versions. So the variable should be comp-enable-subr-trampolines rather than native-comp-enable-subr-trampolines. This was noted by @sergiodj in https://github.com/jorgenschaefer/emacs-buttercup/issues/230#issuecomment-1445538722.

Tested by making this change in the Guix package definition and the buttercup tests complete successfully with Emacs 28.2.

What is the proper fix then, track current release of Emacs for this variable? Or condition on the version to be usable for folks on git?

EDIT: Probably the proper fix to account for the variable name change is to check if either native-comp... or comp-... is t and set to nil. But I am not sure what version buttercup is meant to work with by default or how it handles bleeding edge changes in Emacs.