Common-Lisp: Only first expression is evaluated using ConjureEvalBuf

[iicurtis]

When using the buffer or visual select evaluations, only the first expression is evaluated.

Neovim Config: https://github.com/LazyVim/starter with lua/plugins/example.lua updated to

return {
  {
    "Olical/conjure"
  },
}

Example file:

(defun hellofun ()
           (write-line "Hello, World"))

(hellofun)

Expected Ouput:

; eval (buf): ...src/test.lisp
; "Hello, World"

Actual Output:

; eval (buf): ...src/test.lisp
HELLOFUN

[Olical]

This might just be an issue with capturing the output from the REPL, I have a hunch that the code IS evaluating, we're just not capturing the results from the REPL properly in this scenario.

On Wed, 7 Feb 2024 at 01:21, Isaac Curtis @.***> wrote:

When using the buffer or visual select evaluations, only the first expression is evaluated.

Neovim Config: https://github.com/LazyVim/starter with lua/plugins/example.lua updated to

return { { "Olical/conjure" }, }

Example file:

(defun hellofun () (write-line "Hello, World"))

(hellofun)

Expected Ouput:

; eval (buf): ...src/test.lisp; "Hello, World"

Actual Output:

; eval (buf): ...src/test.lisp HELLOFUN

— Reply to this email directly, view it on GitHub https://github.com/Olical/conjure/issues/555, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACM6XJTWZKRRGK367A7B7LYSLJKXAVCNFSM6AAAAABC466PWWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGEZDCOJXGQ2DOMI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[iicurtis]

I'm not sure that's the case. For example, if I run a larger example file like the one below, none of the functions are defined if I try to evaluate current expression with <leader>ee

;;;; Day 1: Calorie Counting
(in-package #:advent-of-code-2022)

(defparameter *day1-input* #P"../inputs/day01.txt")

(defun run-day1 ()
  (let ((input (uiop:read-file-lines *day1-input*)))
    (time (format t "Part 1: ~a~%" (day01-part1 input)))
    (time (format t "Part 2: ~a~%" (day01-part2 input)))))

(defun day01-part1 (input)
  (reduce #'max (read-cals input)))

(defun day01-part2 (input)
  (reduce #'+ (read-cals input) :end 3))

(defun read-cals (input)
  "Convert input into groups of calorie sums"
  (let ((cals (sort
                (mapcar (lambda (nums) (reduce #'+ (mapcar #'parse-integer nums)))
                        (split-sequence:split-sequence "" input :test #'equal))
                #'>)))
      cals))

(run-day1)

Evaluating (run-day1) results in

; --------------------------------------------------------------------------------
; 127.0.0.1:4005 (connected)
; --------------------------------------------------------------------------------
; eval (buf): ...s/advent-of-code/2022/lisp/src/day01.lisp
#<PACKAGE "ADVENT-OF-CODE-2022">
; --------------------------------------------------------------------------------
; eval (current-form): (run-day1)
; The function COMMON-LISP-USER::RUN-DAY1 is undefined.

[russtoku]

The behavior that you are experiencing is due to the Common-Lisp client not being able to switch the current package in the REPL when an (in-package ...) form has been evaluated. When you evaluate (run-day1), the form is being evaluated in the default COMMON-LIST-USER package. That's the message you see in the Conjure log.

I've made a note of this in https://github.com/Olical/conjure/issues/489#issuecomment-1639015736. In that comment, I suggested a work-around which is to evaluate a (defpackage ...) form.

More work is needed to improve the Common-Lisp client so hopefully someone will help.

[iicurtis]

Thanks for the feedback. Unfortunately I think we have separate issues. Adding a (defpackage :advent-of-code-2022) did nothing, and the log still reports #<PACKAGE "ADVENT-OF-CODE-2022"> for a buffer evaluation.

This may be related to how I start the REPL. I'm not terribly familiar with lisp. If I am editing a source file in my project, I am adding --eval '(ql:quickload :advent-of-code-2022)' to enable project loading/scoping.

If I start the REPL without the quickload and add the (defpackage ...), I get the package, but then evaluating individual lines after that results in unbound errors.

; --------------------------------------------------------------------------------
; 127.0.0.1:4005 (connected)
; --------------------------------------------------------------------------------
; eval (buf): ...s/advent-of-code/2022/lisp/src/day01.lisp
#<PACKAGE "ADVENT-OF-CODE-2022">
; --------------------------------------------------------------------------------
; eval (current-form): (in-package #:advent-of-code-2022)
; The variable #:ADVENT-OF-CODE-2022 is unbound.
; --------------------------------------------------------------------------------
; eval (current-form): (defparameter *day1-input* #P"../inputs/d...
; The variable *DAY1-INPUT* is unbound.

[russtoku]

Interesting!

Using Conjure v4.48.0, I tried to duplicate your project layout with:

$ tree iicurtis/
iicurtis/
├── inputs
│   └── day01.txt
└── src
    ├── day01.lisp

Using this code:

(defpackage :advent-of-code-2022                                                           
  (:use :cl))                                                                              

(in-package #:advent-of-code-2022)                                                          

(defparameter *day1-input* #P"../inputs/day01.txt")

And running SBCL like so:

$ cl-repl 
To load "swank":
  Load 1 ASDF system:
    swank
; Loading "swank"
.
;; Swank started at port: 4005.
  ___  __          ____  ____  ____  __
 / __)(  )    ___ (  _ \(  __)(  _ \(  )
( (__ / (_/\ (___) )   / ) _)  ) __// (_/\
 \___)\____/      (__\_)(____)(__)  \____/
cl-repl 0.6.4 on Roswell 23.10.14.114, SBCL 2.3.6
(C) 2017-2018 TANI Kojiro <kojiro0531@gmail.com>

CL-USER>

I get this in the Conjure log buffer:

; --------------------------------------------------------------------------------         
; 127.0.0.1:4005 (connected)                                                               
; --------------------------------------------------------------------------------         
; eval (buf): .../iicurtis/src/day01rt.lisp                                                
#<PACKAGE "ADVENT-OF-CODE-2022">                                                           
; --------------------------------------------------------------------------------         
; eval (current-form): (in-package #:advent-of-cod...                                       
#<PACKAGE "ADVENT-OF-CODE-2022">                                                           
; --------------------------------------------------------------------------------         
; eval (current-form): (defparameter *day1-input*...                                       
*DAY1-INPUT* 
; --------------------------------------------------------------------------------         
; eval (selection): *day1-input*                                                           
#P"../inputs/day01.txt"

BTW, you should be able to say, (in-package :advent-of-code-2022), instead of (in-package #:advent-of-code-2022).

[iicurtis]

Can I ask what is in your /.replrc, ~/.roswell/config, and ~/.roswell/init.el (or ~/.sbclrc if you aren't using roswell)?

I just created a minimal example to match yours,

❯ tree
.
├── inputs
│   └── day01.txt
└── src
    └── day01.lisp

❯ cl-repl
  ___  __          ____  ____  ____  __
 / __)(  )    ___ (  _ \(  __)(  _ \(  )
( (__ / (_/\ (___) )   / ) _)  ) __// (_/\
 \___)\____/      (__\_)(____)(__)  \____/
cl-repl 0.6.4 on Roswell 23.10.14.114, SBCL 2.4.1
(C) 2017-2018 TANI Kojiro <kojiro0531@gmail.com>

CL-USER> (ql:quickload :swank)
To load "swank":
  Load 1 ASDF system:
    swank
; Loading "swank"
.
[OUT]: (:SWANK)
CL-USER> (swank:create-server :dont-close t)
;; Swank started at port: 4005.

[OUT]: 4005
CL-USER>

:ConjureEvalBuf yields

; Sponsored by @campbellr ❤
; --------------------------------------------------------------------------------
; 127.0.0.1:4005 (connected)
; --------------------------------------------------------------------------------
; eval (buf): ...s/language-playground/lisp/src/day01.lisp
#<PACKAGE "ADVENT-OF-CODE-2022">

using Conjure v4.50

    ● conjure 4.35ms  start
        dir     /home/icurtis/.local/share/nvim/lazy/conjure
        url     https://github.com/olical/conjure
        version 4.50.0
        tag     v4.50.0
        branch  master
        commit  f50d4db

[russtoku]

I'm using Roswell.

~/.replrc:

;; File:         ~/.replrc
;; Description:  init file for cl-repl (Run SBCL Common Lisp)
;; Date:         07/14/2023
;;

;; Start a Swank server on start-up.
(in-package :repl-user)
(ql:quickload :swank)
(swank:create-server :dont-close t)

~/.roswell/config:

setup.time  0   3898355134
sbcl-bin.version    0   2.3.6
default.lisp    0   sbcl-bin

No ~/.roswell/init.el.

[iicurtis]

Thanks for sharing, other than the (in-package :repl-user), that should be pretty similar.

Wait, I just re-read your output. Maybe I am mistaken here, but shouldn't your first buf eval return *DAY1-INPUT*? It looks like you are seeing the same issue. Or I am just not understanding the intention of ConjureEvalBuf. I can go through my file expression by expression and use <leader>er and it will produce the expected output.

(defpackage :advent-of-code-2022
  (:use :cl))

(in-package #:advent-of-code-2022)

(defparameter *day1-input* #P"../inputs/day01.txt")
(print *day1-input*)

; --------------------------------------------------------------------------------
; 127.0.0.1:4005 (connected)
; --------------------------------------------------------------------------------
; eval (buf): ...s/language-playground/lisp/src/day01.lisp
#<PACKAGE "ADVENT-OF-CODE-2022">
; --------------------------------------------------------------------------------
; eval (root-form): (in-package #:advent-of-code-2022)
#<PACKAGE "ADVENT-OF-CODE-2022">
; --------------------------------------------------------------------------------
; eval (root-form): (defparameter *day1-input* #P"../inputs/d...
*DAY1-INPUT*
; --------------------------------------------------------------------------------
; eval (root-form): (print *day1-input*)
; 
; #P"../inputs/day01.txt" 
#P"../inputs/day01.txt"

[russtoku]

I used eval-selection to get the value of the parameter, *day1-input*, instead using (print ...).

[iicurtis]

I used wireshark to capture the packets on any/tcp port 4005 for an event. This has me even more confused, as the entire string is there and its as if swank is actually just ignoring all expressions after the first one.

ConjureEvalBuf:

(:emacs-rex (swank:eval-and-grab-output "(defpackage :advent-of-code-2022\x0a
                                           (:use :cl))\x0a\x0a(in-package
                                           #:advent-of-code-2022)\x0a\x0a(defparameter
                                           *day1-input*
                                           #P\"../inputs/day01.txt\")\x0a(print
                                           *day1-input*)")
            ":advent-of-code-2022" t
            8)

(:return (:ok ("" "#<PACKAGE \"ADVENT-OF-CODE-2022\">")) 8)

ConjureEvalCurrentForm

(:emacs-rex (swank:eval-and-grab-output "(print *day1-input*)")
            ":advent-of-code-2022" t 7)\x0a
(:return (:ok ("\x0a#P\"../inputs/day01.txt\" " "#P\"../inputs/day01.txt\"")) 7)

[Olical]

This may mean we either need to send one line at a time to the REPL or do some parsing to pull out each form and send each one on their own, which isn't ideal since that involves parsing common lisp flawlessly inside Lua.

On Fri, 9 Feb 2024, 21:42 Isaac Curtis, @.***> wrote:

I used wireshark to capture the packets on any/tcp port 4005 for an event. This has me even more confused, as the entire string is there and its as if swank is actually just ignoring all expressions after the first one.

ConjureEvalBuf:

(:emacs-rex (swank:eval-and-grab-output "(defpackage :advent-of-code-2022\x0a (:use :cl))\x0a\x0a(in-package #:advent-of-code-2022)\x0a\x0a(defparameter day1-input #P\"../inputs/day01.txt\")\x0a(print day1-input)") ":advent-of-code-2022" t 8)

(:return (:ok ("" "#<PACKAGE \"ADVENT-OF-CODE-2022\">")) 8)

ConjureEvalCurrentForm

(:emacs-rex (swank:eval-and-grab-output "(print day1-input)") ":advent-of-code-2022" t 7)\x0a (:return (:ok ("\x0a#P\"../inputs/day01.txt\" " "#P\"../inputs/day01.txt\"")) 7)

— Reply to this email directly, view it on GitHub https://github.com/Olical/conjure/issues/555#issuecomment-1936642761, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACM6XJKUBQ4LVZJ6VFRAH3YS2J47AVCNFSM6AAAAABC466PWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZWGY2DENZWGE . You are receiving this because you commented.Message ID: @.***>

[iicurtis]

Okay, so I took a look at how emacs SLY was working, and it seemed to use interactive-eval-region when you activate sly-eval-buffer. I replaced eval-and-grab-output with interactive-eval-region and it seems to work?

; --------------------------------------------------------------------------------
; 127.0.0.1:4005 (connected)
; --------------------------------------------------------------------------------
; eval (buf): ...s/language-playground/lisp/src/day01.lisp
; => #P"../inputs/day01.txt"

Any idea if there are downsides to using interactive-eval-region? The main thing I see is that it only grabs the output of the last expression instead of forming a list for each one. https://github.com/slime/slime/blob/master/swank.lisp

[russtoku]

Using interactive-eval-region should be OK because the Emacs side of Slime uses it to implement slime-eval-buffer. slime-eval-buffer calls slime-eval-region which calls swank:interactive-eval-region. That is the interactive-eval-region function in the Swank server running in the Common Lisp REPL process).

I don't know why eval-and-grab-output was used.

[iicurtis]

I guess you find out eventually. Without the grab-output part, we do not get print for format expressions to show up in the log area, which makes it difficult to work with. Common Lisp is interesting, but its tooling could sure use some love.

[iicurtis]

I still think interactive-eval-region is ultimately the way to go, as that is what emacs does. However, to get the stdout emacs opens a 'channel' at the beginning of communication. The slynk server then responds with a bunch of features, which is very long to which emacs just sends an ACK. Then from then out, slynk sends stdout as "channel" info. We would need to parse these :channel-send commands.

000041(:emacs-rex (slynk-mrepl:create-mrepl 1) "common-lisp-user" t 2)
; snip - feature response from server

An example from emacs for the above discussion example:

000392(:emacs-rex (slynk:interactive-eval-region ";;;; Day 1: Calorie Counting\x0a(in-package :advent-of-code-2022)\x0a\x0a(defparameter *day1-input* #P\"../inputs/day01.txt\")\x0a(defparameter *day1-test-input* \"1000\x0a2000\x0a3000\x0a\x0a4000\x0a\x0a5000\x0a6000\x0a\x0a7000\x0a8000\x0a9000\x0a\x0a10000\")\x0a\x0a(defun run-day1 ()\x0a  (let ((input (uiop:read-file-lines *day1-input*)))\x0a    (time (format t \"Part 1: ~a~%\" (day01-part1 input)))\x0a    (time (format t \"Part 2: ~a~%\" (day01-part2 input)))))\x0a\x0a(defun day01-part1 (input)\x0a  (reduce #'max (read-cals input)))\x0a\x0a(defun day01-part2 (input)\x0a  (reduce #'+ (read-cals input) :end 3))\x0a\x0a(defun read-cals (input)\x0a  \"Convert input into groups of calorie sums\"\x0a  (let ((cals (sort\x0a               (mapcar (lambda (nums) (reduce #'+ (mapcar #'parse-integer nums)))\x0a                       (split-sequence:split-sequence \"\" input :test #'equal))\x0a               #'>)))\x0a    cals))\x0a\x0a(run-day1)\x0a") ":advent-of-code-2022" t 85)

D00005F(:channel-send 1 (:write-string "WARNING: redefining ADVENT-OF-CODE-2022::RUN-DAY1 in DEFUN\x0a"))

TCP ACK

F000062(:channel-send 1 (:write-string "WARNING: redefining ADVENT-OF-CODE-2022::DAY01-PART1 in DEFUN\x0a"))

TCP ACK

F000062(:channel-send 1 (:write-string "WARNING: redefining ADVENT-OF-CODE-2022::DAY01-PART2 in DEFUN\x0a"))

TCP ACK

G0001AD(:channel-send 1 (:write-string "Part 1: 69501\x0aEvaluation took:\x0a  0.000 seconds of real time\x0a  0.000112 seconds of total run time (0.000097 user, 0.000015 system)\x0a  100.00% CPU\x0a  392,525 processor cycles\x0a  98,256 bytes consed\x0a  \x0aPart 2: 202346\x0aEvaluation took:\x0a  0.000 seconds of real time\x0a  0.000109 seconds of total run time (0.000094 user, 0.000015 system)\x0a  100.00% CPU\x0a  377,160 processor cycles\x0a  65,504 bytes consed\x0a  \x0a"))

TCP ACK

H000018(:return (:ok "NIL") 85)

TCP ACK

[russtoku]

@iicurtis , Are you aware of https://github.com/Olical/conjure/pull/400? It was created as an improvement of the Common Lisp client (Slynk) and was written by the same person that created the current Common Lisp client (Swank).

It may provide a way to accomplish what you need for your Common Lisp development. My Common Lisp expertise is zero but I am willing to help out a bit.

[iicurtis]

I hadn't seen that yet, it looks like a good update to split out some of the output parsing. I actually think SLIME/swank is more active than SLY/slynk from the looks these days. We'd be losing out on stickers but we already don't use those. I swapped out SLY for SLIME in emacs and the new result will be slightly easier to implement.

(:emacs-rex (swank:connection-info) ":advent-of-code-2022" t 1)\n
; snip

(:emacs-rex (swank:swank-require '(swank-indentation swank-trace-dialog swank-package-fu swank-presentations swank-macrostep swank-fuzzy swank-fancy-inspector swank-c-p-c swank-arglists swank-repl)) "COMMON-LISP-USER" t 2)\n

(:emacs-rex (swank:init-presentations) ":advent-of-code-2022" t 3)\n
(:return (:ok swank::present-repl-results) 3)

(:emacs-rex (swank-repl:create-repl nil :coding-system "utf-8-unix") ":advent-of-code-2022" t 4)\n
(:return (:ok ("COMMON-LISP-USER" "CL-USER")) 4)

(:emacs-rex (swank:interactive-eval-region ";;;; Day 1: Calorie Counting\n(in-package :advent-of-code-2022)\n\n(defparameter *day1-input* #P\"../inputs/day01.txt\")\n(defparameter *day1-test-input* \"1000\n2000\n300
(:write-string "WARNING: redefining ADVENT-OF-CODE-2022::RUN-DAY1 in DEFUN\n" nil 3)
(:write-done 3)\n
(:write-string "WARNING: redefining ADVENT-OF-CODE-2022::DAY01-PART1 in DEFUN\n" nil 3)
(:write-done 3)\n
(:write-string "WARNING: redefining ADVENT-OF-CODE-2022::READ-CALS in DEFUN\n" nil 3)
(:write-done 3)\n
e-string "Part 1: 69501\nEvaluation took:\n  0.000 seconds of real time\n  0.000216 seconds of total run time (0.000216 user, 0.000000 system)\n  100.00% CPU\n  748,475 processor cycles\n  98,256 bytes consed\
(:write-done 3)\n
(:return (:ok "=> NIL") 5)

[russtoku]

See https://github.com/Olical/conjure/discussions/557

[Olical]

Just out of interest, is this any better now? We changed how the in-package stuff works so you can have multiple in-package calls in a file now and your evals should be contextual based on the closest one.

[Olical]

I'm going to close this under the assumption that the CL changes we did improved things. Please re-open if not!