Delay for testing async code?

[alphapapa]

Hi there,

This looks very, very useful! I've been wanting to be able to test UI-related code, like for helm-org-rifle, but the only way I've seen to do is, maybe, with Espuds, but I haven't tried that yet. This looks like a potentially very simple solution!

However, there is one issue, and that's async code. For example, trying to test like this:

(with-simulated-input "word :tag: RET"
  (helm-org-rifle-occur))

The way Helm works, the RET gets sent before Helm has returned any results, so there are no results displayed. Whereas if I run the command myself, type "word :tag" and wait a moment before pressing RET, the results are displayed.

So, I don't know if it would be feasible without a lot of work, but what do you think about having some kind of optional delay before concluding the command with a final sequence? Maybe something like:

(with-simulated-input 
  '("word :tag:" 1 "RET")
  (helm-org-rifle-occur))

To send the first sequence, then a 1-second delay, and then the final sequence.

Thanks.

[DarwinAwardWinner]

You haven't provided any description for this issue. Can you elaborate?

[alphapapa]

Sorry about the empty issue, I typed in the title and hit enter and it submitted the empty issue instead of sending me to the textarea like I meant. D: You were really quick on the draw! :D

[DarwinAwardWinner]

I can try to implement this, but it might not be possible. I don't know if simply doing sit-for will actually trigger idle timers. Also, implementing waiting using execute-kbd-macro might be tricky.

[alphapapa]

I understand. Maybe Emacs needs more primitive support for this kind of testing.

Anyway, don't put yourself out, just an idea I had. If you feel like working on it, cool. If not, maybe it'll get done someday. :) Thanks.

[DarwinAwardWinner]

Thinking about it, I could simulate idle time by just looking at timer-idle-list and running all the timers with a time less than a specified value. It might get a bit more complicated because each timer might add more idle timers, but it should be possible. Then I could allow the first argument of with-simulated-input to be a list of strings, which are passed to kbd, and lisp forms, which are evaluated at the appropriate point.

[DarwinAwardWinner]

Well, here's one piece of the puzzle:

(defvar simulated-idle-time nil)

(defadvice current-idle-time (around simulat-idle-time activate)
  (if simulated-idle-time
      (setq ad-return-value
            (when (> (float-time simulated-idle-time) 0)
              (seconds-to-time simulated-idle-time)))
    ad-do-it))

(cl-defun simulate-idle-time (secs &optional actually-wait)
  "Run all idle timers with delay less than SECS.

This simulates resetting the idle time to zero and then being
idle for SECS seconds. If ACTUALLY-WAIT is non-nil, this function
will also wait for the specified amount of time before running
each timers.

While each timer is running, `current-idle-time' will be
overridden to return the current simulated idle time."
  (cl-loop
   with already-run-timers = nil
   with stop-time = (float-time secs)
   with simulated-idle-time = 0.0
   ;; We have to search `timer-idle-list' from the beginning each time
   ;; through the loop because each timer that runs might add more
   ;; timers to the list, and picking up at the same list position
   ;; would ignore those new timers.
   for next-timer = (car (cl-member-if-not
                          (lambda (timer) (memq timer already-run-timers))
                          timer-idle-list))
   while next-timer
   for previous-idle-time = simulated-idle-time
   maximize (float-time (timer--time next-timer)) into simulated-idle-time
   do (when actually-wait
        (sit-for (float-time (time-subtract simulated-idle-time previous-idle-time))))
   while (time-less-p simulated-idle-time stop-time)
   do (when (not (timer--triggered next-timer))
        (timer-event-handler next-timer))
   do (push next-timer already-run-timers)))

(progn
  (setq my-var "bye")
  (run-with-idle-timer 500 nil 'set 'my-var "hello")
  (simulate-idle-time 501)
  my-var)

[alphapapa]

Very nice! I tried to add it to a version of the macro, and while there are no errors, it doesn't seem to make helm-org-rifle-occur return any results. I'm not sure if that's because I'm doing something wrong, or if this isn't the way to make Helm work. I'll try something else I thought of, but here's the code, only added a few things:

(defmacro with-simulated-input-and-idle-time (keys &rest body)
  "Eval BODY forms with KEYS as simulated input.

This macro is intended for automated testing of normally
interactive functions by simulating input. If BODY tries to read
user input (e.g. via `completing-read'), it will read input
events from KEYS instead, as if the user had manually typed those
keys after initiating evaluation of BODY.

If BODY tries to read more input events than KEYS provides, an
error is signalled. This is to ensure that BODY will never block
waiting for input, since this macro is intended for
non-interactive use. If BODY does not consume all the input
events in KEYS, the remaining input events are discarded.

The return value is the last form in BODY, as if it was wrapped
in `progn'."
  (declare (indent 1))
  (let ((temp-cmd (cl-gensym "temp-cmd"))
        (cmd-finished-tag (cl-gensym "cmd-finished"))
        (canary-sym (cl-gensym "canary")))
    `(cl-letf*
         (;; Wrap BODY in a command that evaluates BODY and throws the
          ;; result with `cmd-finished-tag'.
          ((symbol-function ',temp-cmd)
           (lambda ()
             (interactive)
             (throw ',cmd-finished-tag (progn ,@body))))
          ;; Set up the keymap for invoking the temp command
          (transient-map (make-sparse-keymap))
          (command-invoke-key-sequence "C-c e")
          (simulate-idle-time-key-sequence "C-c E")
          (simulated-key-sequence ,keys)
          (trailing-C-g-key-sequence
           ;; We *really* want to trigger `keyboard-quit' if we reach
           ;; the end of KEYS.
           "C-g C-g C-g C-g C-g C-g C-g")
          (full-key-sequence
           (mapconcat #'identity
                      (list
                       command-invoke-key-sequence
                       simulated-key-sequence
                       simulate-idle-time-key-sequence
                       trailing-C-g-key-sequence)
                      " ")))
       (when (seq-contains (kbd simulated-key-sequence) (elt (kbd "C-g") 0))
         (error "KEYS must not include C-g"))
       ;; Finish setting up the keymap for the temp command
       (define-key transient-map (kbd command-invoke-key-sequence) ',temp-cmd)
       (define-key transient-map (kbd simulate-idle-time-key-sequence) (lambda () (simulate-idle-time 1)))
       (set-transient-map transient-map)
       ;; Run the command followed by KEYS followed by C-g. The
       ;; `catch' ensures that the keyboard macro stops executing as
       ;; soon as BODY has finished evaluating, even if there are more
       ;; keys to interpret.
       (let ((result
              (condition-case err
                  (catch ',cmd-finished-tag
                    (execute-kbd-macro (kbd full-key-sequence))
                    ;; If the above doesn't throw, return the canary
                    ',canary-sym)
                ;; On `keyboard-quit', return canary
                (quit ',canary-sym))))
         (if (eq result ',canary-sym)
             (error "Reached end of simulated input while evaluating body")
           result)))))

[alphapapa]

My other idea didn't work. Anyway, I think you may be on the right track for idle time, which is really cool. But I think the problem with this Helm issue is Helm handles input asynchronously, and the sources in this case take a few moments to return results, so if Helm receives RET or C-g before it has any results, it stops searching the sources. (Approximately; I am not an expert on Helm internals. :) ) So I think it may need an actual, real-time delay.

Thanks.

[DarwinAwardWinner]

As far as I can tell, helm is using idle timers to do the asynchronous stuff (see helm-input-idle-delay). And simulate-idle-time seems to work in this example, which returns "hello" that is inserted from the idle timer:

(progn
  (run-with-idle-timer 1 nil 'insert "hello")
  (with-simulated-input "M-x eval-expression RET (simulate-idle-time SPC 2) RET RET"
      (read-string "Enter a string: ")))

[DarwinAwardWinner]

However, if the idle timers are invoking external processes asynchronously, that would be a problem.

[alphapapa]

As far as I can tell, helm is using idle timers to do the asynchronous stuff (see helm-input-idle-delay).

Right, and I tried setting that (actually helm-org-rifle-input-idle-delay) to different values within the body of the macro call, but it didn't seem to make any difference, there were still no results from the command.

In fact, I stumbled upon a recent issue in the Helm repo that seemed to be about this: https://github.com/emacs-helm/helm/issues/1750 But I'm using the current version of Helm, and it's not working:

(with-simulated-input-and-idle-time "word :tag: RET"
  (let ((helm-org-rifle-input-idle-delay 0))
    (helm-org-rifle-occur)))

And the sources in question are not async sources, so...I don't know. Helm is amazing but complicated. :)

Thanks for your help.

[alphapapa]

I even tried this just now, leaving the RET and the trailing-C-g-sequence off the main key sequence, then doing sit-for after the key sequence, and then doing RET and the trailing-C-g-sequence afterward, but sit-for isn't having any effect at all.

(defmacro with-simulated-input-and-sleep (keys &rest body)
  "Eval BODY forms with KEYS as simulated input.

This macro is intended for automated testing of normally
interactive functions by simulating input. If BODY tries to read
user input (e.g. via `completing-read'), it will read input
events from KEYS instead, as if the user had manually typed those
keys after initiating evaluation of BODY.

If BODY tries to read more input events than KEYS provides, an
error is signalled. This is to ensure that BODY will never block
waiting for input, since this macro is intended for
non-interactive use. If BODY does not consume all the input
events in KEYS, the remaining input events are discarded.

The return value is the last form in BODY, as if it was wrapped
in `progn'."
  (declare (indent 1))
  (let ((temp-cmd (cl-gensym "temp-cmd"))
        (cmd-finished-tag (cl-gensym "cmd-finished"))
        (canary-sym (cl-gensym "canary")))
    `(cl-letf*
         (;; Wrap BODY in a command that evaluates BODY and throws the
          ;; result with `cmd-finished-tag'.
          ((symbol-function ',temp-cmd)
           (lambda ()
             (interactive)
             (throw ',cmd-finished-tag (progn
                                         ,@body
                                         ))))
          ;; Set up the keymap for invoking the temp command
          (transient-map (make-sparse-keymap))
          (command-invoke-key-sequence "C-c e")
          (simulated-key-sequence ,keys)
          (trailing-C-g-key-sequence
           ;; We *really* want to trigger `keyboard-quit' if we reach
           ;; the end of KEYS.
           "C-g C-g C-g C-g C-g C-g C-g")
          (full-key-sequence
           (mapconcat #'identity
                      (list
                       command-invoke-key-sequence
                       simulated-key-sequence)
                      " ")))
       (when (seq-contains (kbd simulated-key-sequence) (elt (kbd "C-g") 0))
         (error "KEYS must not include C-g"))
       ;; Finish setting up the keymap for the temp command
       (define-key transient-map (kbd command-invoke-key-sequence) ',temp-cmd)
       (set-transient-map transient-map)
       ;; Run the command followed by KEYS followed by C-g. The
       ;; `catch' ensures that the keyboard macro stops executing as
       ;; soon as BODY has finished evaluating, even if there are more
       ;; keys to interpret.
       (let ((result
              (condition-case err
                  (catch ',cmd-finished-tag
                    (execute-kbd-macro (kbd full-key-sequence))
                    (sit-for 2)
                    (execute-kbd-macro (kbd "RET"))
                    (execute-kbd-macro (kbd trailing-C-g-key-sequence))
                    ;; If the above doesn't throw, return the canary
                    ',canary-sym)
                ;; On `keyboard-quit', return canary
                (quit ',canary-sym))))
         (if (eq result ',canary-sym)
             (error "Reached end of simulated input while evaluating body")
           result)))))

I guess that makes sense, since there's a call to execute-kbd-macro immediately after the sit-for, so there's input, and that cancels the sit-for. (I guess.)

Maybe the only way to really test Emacs interactively is to do so from outside the Emacs process, sending keyboard input to its window?

[alphapapa]

Yeah, even this doesn't work as I would expect:

(let ((transient-map (make-sparse-keymap)))
  (define-key transient-map (kbd "C-c e") 'helm-org-rifle-occur)
  (set-transient-map transient-map)
  (execute-kbd-macro (kbd "C-c e")))

I would think that this would start the command and then let me type in the minibuffer to give it input, but the helm-org-rifle-occur command seems to receive RET immediately, so it returns before I have a chance to do anything. I guess the let exits and that ends the command?

I also tried this, but it does the same thing:

(let ((transient-map (make-sparse-keymap)))
  (define-key transient-map (kbd "C-c e") 'helm-org-rifle-occur)
  (set-transient-map transient-map)
  (execute-kbd-macro (kbd "C-c e"))
  (execute-kbd-macro (kbd "word"))
  (execute-kbd-macro (kbd "RET")))

I guess Emacs just isn't made for this. Probably because of how the main loop works or something like that.

[DarwinAwardWinner]

It looks like sit-for doesn't count as being idle. And execute-kbd-macro seems to implicitly enter RET (or more likely directly call exit-minibuffer) if it finishes without exiting the minibuffer.

Anyway, I'm working on a full solution.

[DarwinAwardWinner]

Also, helm enforces a minimum idle delay of 0.01, so setting it to zero won't work.

[DarwinAwardWinner]

Ok, I think I got it working. The following:

(let ((my-var "NEVER SET")
      (temp "DIDN'T WAIT"))
  (run-with-idle-timer 500 nil 'set 'temp "WAITED FOR 500 SECONDS")
  (setq my-var
        (with-simulated-input '("I SPC got: SPC" (simulate-idle-time 501) (insert temp) "RET")
          (read-string "What did you get? ")))
  my-var)

returns "I got: WAITED FOR 500 SECONDS".

I'll take some time to clean up the code and push the change later.

[DarwinAwardWinner]

Ok, I just pushed the code that should enable this.

[alphapapa]

It works! This is amazing!

(with-simulated-input '("word SPC :tag:" (wsi-simulate-idle-time 1) "RET")
  (helm-org-rifle-occur))

And it opens the window, gets the results, and ends the command, just like I ran it manually! Now I can test the contents of the resulting buffer and verify that the command works correctly!

You're a genius! This is really fantastic, we need to spread the word about this, because this will be useful for testing a lot of Emacs packages!

[DarwinAwardWinner]

Thanks for the kind words. This macro has actually existed in the ido-ubiquitous test suite since https://github.com/DarwinAwardWinner/ido-ubiquitous/commit/8d89026a214978b2475cbf6b36c82656a19d47bd, but I only recently decided to split it out into a separate package. I also recently reimplemented it using the keyboard-macro method from the ivy-mode tests: https://github.com/abo-abo/swiper/blob/master/ivy-test.el.

[alphapapa]

Well, I still salute your genius and diligence for doing that. I had looked all over, several times, for a way to test interactive functions, and all I found was Espuds, which I wasn't sure would work (it might, but it's a whole 'nother testing suite to learn, with its own particular style, so I haven't yet). I'm sure that I and lots of other authors will benefit from this code being packaged, rather than being hid in a specific package's personal testing suite. Now we can all benefit from it. :)