call to comint-send-region in racket-send-last-sexp losing spaces in macros?

[tgbugs]

It seems that something strange is going on with an interaction between comint-send-region and syntax->string inside of racket-send-last-sexp. When I run racket-run-module-at-point with the cursor in test-py everything works as expected, but when I run racket-send-last-sexp where indicated below, all of the spaces in the input are removed, apparently during the call to #:with block-source (datum->syntax #'(body ...) (syntax->string #'(body ...))) because the syntax object looks like it is correct and has spaces. I'm not entirely sure this is a racket-mode issue, but it manifests as one.

Given that the call to pretty-print only triggers for racket-send-last-sexp I'm wondering if this has to do with the fact that there is some compilation phase that is missing in racket-send-last-sexp that is handled correctly when compiling a whole file? This was because I had compiled the file first, if the file is not compiled then the pretty-print fires in both cases.

#lang racket/base
(require
 racket/pretty
 (only-in racket/system system*)
 (only-in racket/string string-split non-empty-string?)
 (for-syntax racket/base
             racket/pretty
             syntax/parse
             syntax/to-string
             syntax/location
             (only-in racket/list last)
                     ))
(define-syntax (python stx)
  (syntax-parse stx
  #:datum-literals (python > <)
  [(python (~optional (configuration-keyword-expr ...)) >
           body ...
           < (~optional (symbol-or-expression ...)) python)
   #:with block-source (datum->syntax #'(body ...) (syntax->string #'(body ...))) ; <<<<<<<<<<<<<<<<<<<<<<<
   #:with block-file (datum->syntax #'(body ...) (syntax-source #'(body ...)))
   #:with block-span (datum->syntax #'(body ...) (syntax-span #'(body ...)))
   #:with block-end (datum->syntax #'(body ...) (syntax-span (last (syntax-e #'(body ...)))))
   #:do [(pretty-print #'(body ...))] ; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
   #'(eval-python block-source
                  #:file block-file
                  #:span block-span
                  #:end block-end
                  (~? (~@ configuration-keyword-expr ...))
                  (~? (~@ #:return-values '(symbol-or-expression ...))))]))

(define (eval-python src
                     #:file block-file
                     #:span block-span
                     #:end block-end
                     #:interpreter [interpreter "/usr/bin/env python3"]
                     #:return-values [return-values null])
  (pretty-display (format "py-src:\n~a" src))
  (pretty-display (format "py-res:\n~a" return-values))
  (apply system* (append (string-split interpreter) (list "-c" src))))

(module+ test-py
  (python > import os
          print('lol', os.name) < python)

;; C-x C-e directly above this line <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<  
  )

Results Ok.

hrm.rkt/test-py> 
#<syntax:/tmp/test.rkt:23:25 (import os print ((quote lol) (quote (unquote os.name))))>
py-src:
import os
print('lol', os.name)
py-res:
()
lol posix
#t

Error.

hrm.rkt/test-py> 
#<syntax:/tmp/test.rkt:23:25 (import os print ((quote lol) (quote (unquote os.name))))>
py-src:
importosprint('lol',os.name)
py-res:
()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'importosprint' is not defined
#f

((alist-get 'racket-mode package-alist)
 #s(package-desc racket-mode
         (20210202 2145)
         "Racket editing, REPL, and more"
         ((emacs
           (25 1))
          (faceup
           (0 0 2))
          (pos-tip
           (20191127 1028)))
         nil nil "/home/tom/.emacs.d/elpa/racket-mode-20210202.2145"
         ((:url . "https://www.racket-mode.com/")
          (:maintainer "Greg Hendershott")
          (:authors
           ("Greg Hendershott"))
          (:commit . "3e63bdc5bdc5a9d0a28f3568fa994a5792c447c5"))
         nil))
((emacs-version "28.0.50")
 (system-type gnu/linux)
 (x-gtk-use-system-tooltips t)
 (major-mode racket-mode)
 (racket--el-source-dir "/home/tom/.emacs.d/elpa/racket-mode-20210202.2145/")
 (racket--rkt-source-dir "/home/tom/.emacs.d/elpa/racket-mode-20210202.2145/racket/")
 (racket-program "racket")
 (racket-command-timeout 10)
 (racket-xp-after-change-refresh-delay 1)
 (racket-xp-highlight-unused-regexp "^[^_]")
 (racket-repl-buffer-name-function nil)
 (racket-memory-limit 2048)
 (racket-error-context medium)
 (racket-history-filter-regexp "\\`\\s *\\'")
 (racket-images-inline t)
 (racket-images-keep-last 100)
 (racket-images-system-viewer "display")
 (racket-images-system-viewer "display")
 (racket-use-repl-submit-predicate nil)
 (racket-pretty-print t)
 (racket-indent-curly-as-sequence t)
 (racket-indent-sequence-depth 0)
 (racket-pretty-lambda nil)
 (racket-smart-open-bracket-enable nil)
 (racket-module-forms "\\s(\\(?:module[*+]?\\|library\\)")
 (racket-logger-config
  ((cm-accomplice . warning)
   (GC . info)
   (module-prefetch . warning)
   (optimizer . info)
   (racket/contract . error)
   (sequence-specialization . info)
   (* . fatal)))
 (racket-show-functions
  (racket-show-pseudo-tooltip)))
(enabled-minor-modes
 (auto-composition-mode)
 (auto-compression-mode)
 (auto-encryption-mode)
 (auto-fill-mode)
 (auto-save-mode)
 (blink-cursor-mode)
 (electric-indent-mode)
 (file-name-shadow-mode)
 (font-lock-mode)
 (global-eldoc-mode)
 (global-font-lock-mode)
 (hs-minor-mode)
 (line-number-mode)
 (menu-bar-mode)
 (mouse-wheel-mode)
 (semantic-minor-modes-format)
 (tool-bar-mode)
 (tooltip-mode)
 (transient-mark-mode))
(disabled-minor-modes
 (abbrev-mode)
 (auto-fill-function)
 (auto-save-visited-mode)
 (buffer-read-only)
 (button-mode)
 (cl-old-struct-compat-mode)
 (column-number-mode)
 (compilation-minor-mode)
 (compilation-shell-minor-mode)
 (completion-in-region-mode)
 (defining-kbd-macro)
 (eldoc-mode)
 (electric-layout-mode)
 (electric-quote-mode)
 (global-prettify-symbols-mode)
 (global-semantic-highlight-edits-mode)
 (global-semantic-highlight-func-mode)
 (global-semantic-show-parser-state-mode)
 (global-semantic-show-unmatched-syntax-mode)
 (global-semantic-stickyfunc-mode)
 (global-visual-line-mode)
 (horizontal-scroll-bar-mode)
 (ido-everywhere)
 (isearch-mode)
 (jit-lock-debug-mode)
 (next-error-follow-minor-mode)
 (overwrite-mode)
 (paragraph-indent-minor-mode)
 (prettify-symbols-mode)
 (racket-smart-open-bracket-mode)
 (racket-xp-mode)
 (semantic-highlight-edits-mode)
 (semantic-highlight-func-mode)
 (semantic-mode)
 (semantic-show-parser-state-mode)
 (semantic-show-unmatched-syntax-mode)
 (semantic-stickyfunc-mode)
 (sh-electric-here-document-mode)
 (size-indication-mode)
 (tab-bar-history-mode)
 (tab-bar-mode)
 (temp-buffer-resize-mode)
 (unify-8859-on-decoding-mode)
 (unify-8859-on-encoding-mode)
 (url-handler-mode)
 (use-hard-newlines)
 (view-mode)
 (visible-mode)
 (visual-line-mode)
 (window-divider-mode)
 (xref-etags-mode))

[greghendershott]

comint-send-region is effectively copying from the source buffer and pasting into the REPL buffer.

In a Racket Mode REPL, typing:

> (require syntax/to-string)
> (syntax->string #'(foo bar baz))
"foobarbaz"

This seems to be because the TCP input port for the Racket Mode REPL has not had port-count-lines! called on it. Therefore things like syntax-column are #f. Quick glance, syntax->string uses syntax->column.

Possibly this broke when doing "The Big Switcheroo", and the REPL input port changed from stdin to TCP.

I'll push a commit to add port-count-lines!.

[greghendershott]

Thanks again for reporting!

Merged.

If you get Racket Mode from MELPA, it might take a couple hours for that to refresh.

You'll want to wait until it says the version is > 20210211.0000.

As usual be sure to do a package-refresh-contents in Emacs before updating.

[tgbugs]

Confirming working on my end now. Thanks!

[greghendershott]

Reopening because while looking at #556 I realized that commit 7f12cb1 regressed this.