Using C-u C-M-x to instrument individual definitions for debugging is unreliable

First of all, thanks for your awesome work!

I noticed that in racket-mode I cannot step debug across a dict-set! call. There are probably other examples too, but this is the one I encountered. Example:

(define (demo-dict-debugger)
  (define some-dict (make-hash))
  (dict-set! some-dict "key" "value")
  (dict-ref some-dict "key"))

Instrument the function for debugging and then try to step over each line. When reaching dict-set! the program fails with

; lifted/2.2: undefined;
;  cannot reference an identifier before its definition
;   in module: "/tmp/racket-mode.rkt"

((alist-get 'racket-mode package-alist))
((emacs-version "27.1")
 (system-type gnu/linux)
 (x-gtk-use-system-tooltips nil)
 (major-mode racket-mode)
 (racket--el-source-dir "/home/felixl/.emacs.d/.local/straight/build-27.1/racket-mode/")
 (racket--rkt-source-dir "/home/felixl/.emacs.d/.local/straight/build-27.1/racket-mode/racket/")
 (racket-program "racket")
 (racket-command-timeout 10)
 (racket-path-from-emacs-to-racket-function identity)
 (racket-path-from-racket-to-emacs-function identity)
 (racket-browse-url-function racket-browse-url-using-temporary-file)
 (racket-documentation-search-location "https://docs.racket-lang.org/search/index.html?q=%s")
 (racket-xp-after-change-refresh-delay 1)
 (racket-xp-mode-lighter
  (:eval
   (racket--xp-mode-lighter)))
 (racket-xp-highlight-unused-regexp "^[^_]")
 (racket-repl-buffer-name-function nil)
 (racket-submodules-to-run
  ((test)
   (main)))
 (racket-memory-limit 2048)
 (racket-error-context medium)
 (racket-repl-history-directory "~/.emacs.d/.local/etc/racket-mode/")
 (racket-history-filter-regexp "\\`\\s *\\'")
 (racket-images-inline t)
 (racket-imagemagick-props nil)
 (racket-images-keep-last 100)
 (racket-images-system-viewer "display")
 (racket-pretty-print t)
 (racket-use-repl-submit-predicate nil)
 (racket-pretty-print t)
 (racket-indent-curly-as-sequence t)
 (racket-indent-sequence-depth 0)
 (racket-pretty-lambda nil)
 (racket-smart-open-bracket-enable nil)
 (racket-module-forms "\\s(\\(?:module[*+]?\\|library\\)")
 (racket-logger-config
  ((cm-accomplice . warning)
   (GC . info)
   (module-prefetch . warning)
   (optimizer . info)
   (racket/contract . error)
   (sequence-specialization . info)
   (* . fatal)))
 (racket-show-functions
  (racket-show-pseudo-tooltip)))
(enabled-minor-modes
 (+popup-mode)
 (async-bytecomp-package-mode)
 (auto-composition-mode)
 (auto-compression-mode)
 (auto-encryption-mode)
 (auto-fill-mode)
 (auto-save-mode)
 (better-jumper-local-mode)
 (better-jumper-mode)
 (column-number-mode)
 (company-mode)
 (delete-selection-mode)
 (diff-hl-margin-mode)
 (display-line-numbers-mode)
 (doom-modeline-mode)
 (dtrt-indent-mode)
 (electric-indent-mode)
 (file-name-shadow-mode)
 (flycheck-mode)
 (flycheck-popup-tip-mode)
 (font-lock-mode)
 (gcmh-mode)
 (general-override-mode)
 (global-company-mode)
 (global-eldoc-mode)
 (global-flycheck-mode)
 (global-font-lock-mode)
 (global-git-commit-mode)
 (global-hl-line-mode)
 (global-so-long-mode)
 (global-undo-fu-session-mode)
 (highlight-numbers-mode)
 (highlight-quoted-mode)
 (hl-line-mode)
 (hl-todo-mode)
 (hs-minor-mode)
 (ivy-mode)
 (ivy-rich-mode)
 (ivy-rich-project-root-cache-mode)
 (line-number-mode)
 (mouse-wheel-mode)
 (org-roam-bibtex-mode)
 (org-roam-db-autosync-mode)
 (override-global-mode)
 (persp-mode)
 (projectile-mode)
 (racket-smart-open-bracket-mode)
 (racket-xp-mode)
 (rainbow-delimiters-mode)
 (recentf-mode)
 (save-place-mode)
 (savehist-mode)
 (semantic-minor-modes-format)
 (shell-dirtrack-mode)
 (show-paren-mode)
 (size-indication-mode)
 (smartparens-global-mode)
 (smartparens-mode)
 (solaire-global-mode)
 (transient-mark-mode)
 (undo-fu-mode)
 (undo-fu-session-mode)
 (vi-tilde-fringe-mode)
 (volatile-highlights-mode)
 (which-key-mode)
 (whitespace-mode)
 (window-divider-mode)
 (winner-mode)
 (ws-butler-global-mode)
 (ws-butler-mode)
 (yas-global-mode)
 (yas-minor-mode))
(disabled-minor-modes
 (+emacs-lisp-ert-mode)
 (+org-pretty-mode)
 (+popup-buffer-mode)
 (abbrev-mode)
 (ace-window-display-mode)
 (ace-window-mode)
 (amx-debug-mode)
 (amx-mode)
 (anzu-mode)
 (auto-fill-function)
 (auto-revert-mode)
 (auto-revert-tail-mode)
 (auto-save-visited-mode)
 (avy-linum-mode)
 (bibtex-completion-notes-global-mode)
 (bibtex-completion-notes-mode)
 (blink-cursor-mode)
 (buffer-face-mode)
 (buffer-read-only)
 (bug-reference-mode)
 (bug-reference-prog-mode)
 (cl-old-struct-compat-mode)
 (company-search-mode)
 (compilation-minor-mode)
 (compilation-shell-minor-mode)
 (completion-in-region-mode)
 (counsel-mode)
 (cursor-intangible-mode)
 (cursor-sensor-mode)
 (dash-fontify-mode)
 (defining-kbd-macro)
 (diff-auto-refine-mode)
 (diff-hl-dir-mode)
 (diff-hl-dired-mode)
 (diff-hl-margin-local-mode)
 (diff-hl-mode)
 (diff-minor-mode)
 (dired-hide-details-mode)
 (dired-isearch-filenames-mode)
 (dired-omit-mode)
 (diredfl-global-mode)
 (diredfl-mode)
 (drag-stuff-global-mode)
 (drag-stuff-mode)
 (dtrt-indent-global-mode)
 (edit-indirect--overlay)
 (eldoc-mode)
 (electric-layout-mode)
 (electric-quote-mode)
 (general-override-local-mode)
 (git-commit-mode)
 (git-gutter-mode)
 (global-anzu-mode)
 (global-auto-revert-mode)
 (global-dash-fontify-mode)
 (global-diff-hl-mode)
 (global-display-line-numbers-mode)
 (global-git-gutter-mode)
 (global-hide-mode-line-mode)
 (global-hl-todo-mode)
 (global-prettify-symbols-mode)
 (global-semantic-highlight-edits-mode)
 (global-semantic-highlight-func-mode)
 (global-semantic-show-parser-state-mode)
 (global-semantic-show-unmatched-syntax-mode)
 (global-semantic-stickyfunc-mode)
 (global-vi-tilde-fringe-mode)
 (global-visual-line-mode)
 (global-whitespace-mode)
 (global-whitespace-newline-mode)
 (goto-address-mode)
 (goto-address-prog-mode)
 (helm--minor-mode)
 (helm--remap-mouse-mode)
 (helm-adaptive-mode)
 (helm-autoresize-mode)
 (helm-display-line-numbers-mode)
 (helm-ff--delete-async-modeline-mode)
 (helm-migemo-mode)
 (helm-popup-tip-mode)
 (hide-mode-line-mode)
 (horizontal-scroll-bar-mode)
 (ibuffer-auto-mode)
 (ido-everywhere)
 (image-minor-mode)
 (isearch-mode)
 (ispell-minor-mode)
 (jit-lock-debug-mode)
 (macrostep-mode)
 (magit-auto-revert-mode)
 (magit-blame-mode)
 (magit-blame-read-only-mode)
 (magit-blob-mode)
 (magit-gitflow-mode)
 (magit-popup-help-mode)
 (magit-todos-mode)
 (magit-wip-after-apply-mode)
 (magit-wip-after-save-local-mode)
 (magit-wip-after-save-mode)
 (magit-wip-before-change-mode)
 (magit-wip-initial-backup-mode)
 (magit-wip-mode)
 (mail-abbrevs-mode)
 (markdown-live-preview-mode)
 (menu-bar-mode)
 (mml-mode)
 (next-error-follow-minor-mode)
 (org-capture-mode)
 (org-cdlatex-mode)
 (org-fancy-priorities-mode)
 (org-indent-mode)
 (org-list-checkbox-radio-mode)
 (org-src-mode)
 (org-superstar-mode)
 (org-table-follow-field-mode)
 (org-table-header-line-mode)
 (orgtbl-mode)
 (outline-minor-mode)
 (overwrite-mode)
 (paragraph-indent-minor-mode)
 (paredit-mode)
 (pcre-mode)
 (prettify-symbols-mode)
 (racket-debug-mode)
 (rst-minor-mode)
 (rxt--read-pcre-mode)
 (rxt-global-mode)
 (rxt-mode)
 (semantic-highlight-edits-mode)
 (semantic-highlight-func-mode)
 (semantic-mode)
 (semantic-show-parser-state-mode)
 (semantic-show-unmatched-syntax-mode)
 (semantic-stickyfunc-mode)
 (server-mode)
 (sh-electric-here-document-mode)
 (shell-command-with-editor-mode)
 (show-smartparens-global-mode)
 (show-smartparens-mode)
 (smartparens-global-strict-mode)
 (smartparens-strict-mode)
 (smerge-mode)
 (so-long-minor-mode)
 (solaire-mode)
 (tab-bar-history-mode)
 (tab-bar-mode)
 (table-fixed-width-mode)
 (table-mode-indicator)
 (temp-buffer-resize-mode)
 (text-scale-mode)
 (toc-org-mode)
 (tool-bar-mode)
 (tooltip-mode)
 (transient-resume-mode)
 (unify-8859-on-decoding-mode)
 (unify-8859-on-encoding-mode)
 (url-handler-mode)
 (use-hard-newlines)
 (vc-parent-buffer)
 (view-mode)
 (visible-mode)
 (visual-line-mode)
 (which-function-mode)
 (whitespace-newline-mode)
 (with-editor-mode)
 (xref-etags-mode))

With this file:

#lang racket/base

(require racket/dict)

(define (demo-dict-debugger)
  (define some-dict (make-hash))
  (dict-set! some-dict "key" "value")
  (dict-ref some-dict "key"))

(demo-dict-debugger)

With C-c C-c it runs and prints "value" in the REPL.
With C-u C-u C-c C-c it runs stopping at the open ( for (demo-dict-debugger) (position 174).
As I keep pressing SPACE it steps. The REPL ends up looking like:

; 
; Welcome to Racket v8.3.0.11 [cs].
; 
> 
"value"
issue-612.rkt> 
; Debug annotate #<path:/var/tmp/issue-612.rkt>
[issue-612.rkt:174]> 
[issue-612.rkt:43]> 
[issue-612.rkt:92]> 
[issue-612.rkt:102]> 
[issue-612.rkt:107]> 
[issue-612.rkt:141]> 
[issue-612.rkt:145]> 
"value"
[issue-612.rkt:193]> 
issue-612.rkt>

So I can't reproduce this.

However maybe your actual complete source program is different?

Or maybe you're using different steps (no pun intended)?

Or maybe there is indeed some bug but due to something else we need to figure out about how your situation differs from mine.

p.s. To double check, another way you might have tried this:

I can also step successfully by entering (demo-dict-debugger) in the REPL (after having done the C-u C-u C-c C-c to annotate the program for step debugging, and run it, as in the first example).

In that case the first break is the open ( for the define of the function. Thereafter it's the same breaks.

So I can't reproduce this.

I think I might have found the difference.

I just tried to debug with C-u C-u C-c C-c and it works just fine. But when I run the program with C-c C-c, place the cursor within the function and press C-u C-M-x to instrument the function I receive an error (I don't know how the error could have slipped my attention before):

breakable-positions: contract violation
  expected: path?
  given: #f
  in: the keys of
      (hash/c
       path?
       (set/c exact-positive-integer? #:cmp 'eq))
  contract from: 
      (definition breakable-positions)
  blaming: <my-home-directory>/.emacs.d/.local/straight/build-27.1/racket-mode/racket/debug.rkt
   (assuming the contract is correct)
  at: <my-home-directory>/.emacs.d/.local/straight/build-27.1/racket-mode/racket/debug.rkt:40:17

After pressing C-u C-M-x a second time, I receive the message Now you can call the function in the REPL to step debug it, but as soon as I do I get the lifted/2.2: undefined; error initially described.

To be sure I updated to the latest commit #7121e993c, which is why I include the details again.

((alist-get 'racket-mode package-alist))
((emacs-version "27.1")
 (system-type gnu/linux)
 (x-gtk-use-system-tooltips nil)
 (major-mode racket-mode)
 (racket--el-source-dir "/home/felixl/.emacs.d/.local/straight/build-27.1/racket-mode/")
 (racket--rkt-source-dir "/home/felixl/.emacs.d/.local/straight/build-27.1/racket-mode/racket/")
 (racket-program "racket")
 (racket-command-timeout 10)
 (racket-path-from-emacs-to-racket-function UNDEFINED)
 (racket-path-from-racket-to-emacs-function UNDEFINED)
 (racket-browse-url-function racket-browse-url-using-temporary-file)
 (racket-documentation-search-location "https://docs.racket-lang.org/search/index.html?q=%s")
 (racket-xp-after-change-refresh-delay 1)
 (racket-xp-mode-lighter
  (:eval
   (racket--xp-mode-lighter)))
 (racket-xp-highlight-unused-regexp "^[^_]")
 (racket-repl-buffer-name-function nil)
 (racket-submodules-to-run
  ((test)
   (main)))
 (racket-memory-limit 2048)
 (racket-error-context medium)
 (racket-repl-history-directory "~/.emacs.d/.local/etc/racket-mode/")
 (racket-history-filter-regexp "\\`\\s *\\'")
 (racket-images-inline t)
 (racket-imagemagick-props nil)
 (racket-images-keep-last 100)
 (racket-images-system-viewer "display")
 (racket-pretty-print t)
 (racket-use-repl-submit-predicate nil)
 (racket-pretty-print t)
 (racket-indent-curly-as-sequence t)
 (racket-indent-sequence-depth 0)
 (racket-pretty-lambda nil)
 (racket-smart-open-bracket-enable nil)
 (racket-module-forms "\\s(\\(?:module[*+]?\\|library\\)")
 (racket-logger-config
  ((cm-accomplice . warning)
   (GC . info)
   (module-prefetch . warning)
   (optimizer . info)
   (racket/contract . error)
   (sequence-specialization . info)
   (* . fatal)))
 (racket-show-functions
  (racket-show-pseudo-tooltip)))
(enabled-minor-modes
 (+popup-mode)
 (async-bytecomp-package-mode)
 (auto-composition-mode)
 (auto-compression-mode)
 (auto-encryption-mode)
 (auto-fill-mode)
 (auto-save-mode)
 (better-jumper-local-mode)
 (better-jumper-mode)
 (column-number-mode)
 (company-mode)
 (delete-selection-mode)
 (diff-hl-margin-mode)
 (display-line-numbers-mode)
 (doom-modeline-mode)
 (dtrt-indent-mode)
 (electric-indent-mode)
 (eros-mode)
 (file-name-shadow-mode)
 (flycheck-mode)
 (flycheck-popup-tip-mode)
 (font-lock-mode)
 (gcmh-mode)
 (general-override-mode)
 (global-company-mode)
 (global-eldoc-mode)
 (global-flycheck-mode)
 (global-font-lock-mode)
 (global-git-commit-mode)
 (global-hl-line-mode)
 (global-so-long-mode)
 (global-undo-fu-session-mode)
 (highlight-numbers-mode)
 (highlight-quoted-mode)
 (hl-line-mode)
 (hl-todo-mode)
 (hs-minor-mode)
 (ivy-mode)
 (ivy-rich-mode)
 (ivy-rich-project-root-cache-mode)
 (line-number-mode)
 (mouse-wheel-mode)
 (org-roam-db-autosync-mode)
 (override-global-mode)
 (persp-mode)
 (projectile-mode)
 (racket-smart-open-bracket-mode)
 (racket-xp-mode)
 (rainbow-delimiters-mode)
 (recentf-mode)
 (save-place-mode)
 (savehist-mode)
 (semantic-minor-modes-format)
 (shell-dirtrack-mode)
 (show-paren-mode)
 (size-indication-mode)
 (smartparens-global-mode)
 (smartparens-mode)
 (solaire-global-mode)
 (transient-mark-mode)
 (undo-fu-mode)
 (undo-fu-session-mode)
 (vi-tilde-fringe-mode)
 (volatile-highlights-mode)
 (which-key-mode)
 (whitespace-mode)
 (window-divider-mode)
 (winner-mode)
 (ws-butler-global-mode)
 (ws-butler-mode)
 (yas-global-mode)
 (yas-minor-mode))
(disabled-minor-modes
 (+emacs-lisp-ert-mode)
 (+org-pretty-mode)
 (+popup-buffer-mode)
 (abbrev-mode)
 (ace-window-display-mode)
 (ace-window-mode)
 (amx-debug-mode)
 (amx-mode)
 (auto-fill-function)
 (auto-revert-mode)
 (auto-revert-tail-mode)
 (auto-save-visited-mode)
 (avy-linum-mode)
 (blink-cursor-mode)
 (buffer-face-mode)
 (buffer-read-only)
 (cl-old-struct-compat-mode)
 (company-search-mode)
 (compilation-minor-mode)
 (compilation-shell-minor-mode)
 (completion-in-region-mode)
 (counsel-mode)
 (cursor-intangible-mode)
 (cursor-sensor-mode)
 (dash-fontify-mode)
 (defining-kbd-macro)
 (diff-auto-refine-mode)
 (diff-hl-dir-mode)
 (diff-hl-dired-mode)
 (diff-hl-margin-local-mode)
 (diff-hl-mode)
 (diff-minor-mode)
 (dired-hide-details-mode)
 (dired-isearch-filenames-mode)
 (dired-omit-mode)
 (diredfl-global-mode)
 (diredfl-mode)
 (dtrt-indent-global-mode)
 (edit-indirect--overlay)
 (eldoc-mode)
 (electric-layout-mode)
 (electric-quote-mode)
 (general-override-local-mode)
 (git-commit-mode)
 (git-gutter-mode)
 (global-auto-revert-mode)
 (global-dash-fontify-mode)
 (global-diff-hl-mode)
 (global-display-line-numbers-mode)
 (global-git-gutter-mode)
 (global-hide-mode-line-mode)
 (global-hl-todo-mode)
 (global-prettify-symbols-mode)
 (global-semantic-highlight-edits-mode)
 (global-semantic-highlight-func-mode)
 (global-semantic-show-parser-state-mode)
 (global-semantic-show-unmatched-syntax-mode)
 (global-semantic-stickyfunc-mode)
 (global-vi-tilde-fringe-mode)
 (global-visual-line-mode)
 (global-whitespace-mode)
 (global-whitespace-newline-mode)
 (helm--minor-mode)
 (helm--remap-mouse-mode)
 (helm-autoresize-mode)
 (helm-display-line-numbers-mode)
 (helm-ff--delete-async-modeline-mode)
 (helm-ff-icon-mode)
 (helm-migemo-mode)
 (helm-popup-tip-mode)
 (hide-mode-line-mode)
 (horizontal-scroll-bar-mode)
 (ibuffer-auto-mode)
 (image-dired-minor-mode)
 (image-minor-mode)
 (isearch-mode)
 (jit-lock-debug-mode)
 (macrostep-mode)
 (magit-auto-revert-mode)
 (magit-blame-mode)
 (magit-blame-read-only-mode)
 (magit-blob-mode)
 (magit-gitflow-mode)
 (magit-popup-help-mode)
 (magit-todos-mode)
 (magit-wip-after-apply-mode)
 (magit-wip-after-save-local-mode)
 (magit-wip-after-save-mode)
 (magit-wip-before-change-mode)
 (magit-wip-initial-backup-mode)
 (magit-wip-mode)
 (mail-abbrevs-mode)
 (markdown-live-preview-mode)
 (menu-bar-mode)
 (mml-mode)
 (next-error-follow-minor-mode)
 (org-capture-mode)
 (org-cdlatex-mode)
 (org-fancy-priorities-mode)
 (org-indent-mode)
 (org-list-checkbox-radio-mode)
 (org-src-mode)
 (org-superstar-mode)
 (org-table-follow-field-mode)
 (org-table-header-line-mode)
 (orgtbl-mode)
 (outline-minor-mode)
 (overwrite-mode)
 (paragraph-indent-minor-mode)
 (pcre-mode)
 (prettify-symbols-mode)
 (racket-debug-mode)
 (rst-minor-mode)
 (rxt--read-pcre-mode)
 (rxt-global-mode)
 (rxt-mode)
 (semantic-highlight-edits-mode)
 (semantic-highlight-func-mode)
 (semantic-mode)
 (semantic-show-parser-state-mode)
 (semantic-show-unmatched-syntax-mode)
 (semantic-stickyfunc-mode)
 (server-mode)
 (sh-electric-here-document-mode)
 (shell-command-with-editor-mode)
 (show-smartparens-global-mode)
 (show-smartparens-mode)
 (smartparens-global-strict-mode)
 (smartparens-strict-mode)
 (smerge-mode)
 (so-long-minor-mode)
 (solaire-mode)
 (tab-bar-history-mode)
 (tab-bar-mode)
 (temp-buffer-resize-mode)
 (text-scale-mode)
 (toc-org-mode)
 (tool-bar-mode)
 (tooltip-mode)
 (transient-resume-mode)
 (unify-8859-on-decoding-mode)
 (unify-8859-on-encoding-mode)
 (url-handler-mode)
 (use-hard-newlines)
 (vc-parent-buffer)
 (view-mode)
 (visible-mode)
 (visual-line-mode)
 (which-function-mode)
 (whitespace-newline-mode)
 (with-editor-mode)
 (xref-etags-mode))

Aha -- thanks! I can definitely reproduce this. I'll investigate.

I found and fixed the dumb, obvious layer of this: The error message as a result of supplying #f to a dictionary that expects a path?.

However, after fixing that, the function is annotated with zero breakable positions. In other words, if you try to step through it, you just step over the whole thing. It's still not the result you would want or expect.

Interestingly, if you change the function not to use something from racket/dict -- say the function body is just a few (println 42) expressions -- then it does get annotated with breakable positions.

racket/dict guards things like dict-set! with contracts. When contracts are involved, macro expansion becomes more significantly more complicated. Apparently the annotator can handle the entire module expansion just fine, but not the attempt to re-define just the function.

Two paths here:

Try to figure this out (assuming it's even possible, for some reasonable definition of possible).
Add this to the caveats that already exist for instrumenting individual functions.

The entire step debugger is labeled "experimental". I admit that's kind of a cop-out, but it's also honest. And within that, the instrument-just-one-function feature is even hackier -- it's an experiment within an experiment. I liked having that kind of thing in Emacs for the Elisp debugger. It occurred to me that maybe I could do something similar. And it kind of works... for simple cases... but....

In contrast to Emacs Lisp, Racket is a module-centric language, especially wrt macro expansion. The more sophisticated the macros (like the contract system), the less likely you can just "monkey-patch" some individual definition without updating other things that happened while expanding the whole module. At least, at my pay grade, that's my understanding of the situation.

I'll keep thinking about this, but I wanted to go ahead and share that early assessment.

Thanks for the apt and detailed analysis! I read somewhere that the debugging feature is considered experimental, yet I am eternally grateful for it. Especially when exploring foreign code that does not adhere to best practices (e.g. small one-purpose functions), I find the debugger invaluable.

I think the happy path (or least the more well beaten path) is instrumenting the entire module. That uses the same technique as does Dr Racket.

My attempt to let you instrument individual definitions, like you can in an environment like Emacs, is the trickier part. Maybe it is possible to improve this, but I'm not sure I would find it worth the effort, just personally for my own use.

By the way, I continued thinking about this. My analysis above isn't really satisfying. It doesn't really explain why you can redefine the function normally in the REPL (plain C-M-x) and it works, but not when it is first expanded and instrumented for step debugging breakpoints. Something else is going on. I even have some changes, already, with which the example works and would ostensibly close this issue per se. Before declaring victory, I want to take some more time to feel like I truly understand why.

I've pushed a few commits to an issue-612 topic branch.

As you can see from the commit messages and code comments, the situation is improved. However there's still a gotcha: After using C-u C-M-x to successfully instrument a definition, you can't then use plain C-M-x to un-instrument it. I mean, you can, and it redefines the demo-dict-debugger function -- but not the two extra lifted/N definitions resulting from the contract wrapper macros, so you'll get an undefined error message about those. Instead you need to re-run the whole file without instrumentation (e.g. C-c C-c).

What is this "lifted" stuff?

The example program

#lang racket/base

(require racket/dict)

(define (demo-dict-debugger)
  (define some-dict (make-hash))
  (dict-set! some-dict "key" "value")
  (dict-ref some-dict "key"))

Fully macro-expands to something like

(module issue-612 racket/base
  (#%module-begin
   (module configure-runtime '#%kernel
     (#%module-begin (#%require racket/runtime-config) (#%app configure (quote #f))))
   (#%require racket/dict)
   (define-values (lifted/2:19)
     (begin:26
       (with-continuation-mark:26
        contract-continuation-mark-key:26
        (#%app:27 cons idB42 'no-negative-party)
        (let-values:28 ()
          (#%app:29
           idX39:18
           (#%app:31
            module-name-fixup:30
            (#%app:33 variable-reference->module-source/submod:32 (#%variable-reference:32))
            (#%app:34 list:30)))))))
   (define-values (lifted/3:24)
     (begin:35
       (with-continuation-mark:35
        contract-continuation-mark-key:35
        (#%app:36 cons:21 idB34:22 'no-negative-party:21)
        (let-values:37 ()
          (#%app:38
           idX31:23
           (#%app:40
            module-name-fixup:39
            (#%app:42 variable-reference->module-source/submod:41 (#%variable-reference:41))
            (#%app:43 list:39)))))))
   (define-values (demo-dict-debugger)
     (lambda ()
       (let-values (((some-dict) (#%app make-hash)))
         (#%app:20 lifted/2:19 some-dict (quote "key") (quote "value"))
         (#%app:25 lifted/3:24 some-dict (quote "key")))))))

Alas the more I look into this, the more problems I find.

The latest: The instrumentation code, as well as the breakable positions data structure, are designed assuming whole-file instrumentation. They don't attempt to track which code and positions are associated with which function name. As a result, if you instrument a single function, and it has already been instrumented (either by whole-file or single-function instrumentation), the old breakable positions data remains. So if you're at a break and use e.g. the n and p to navigate among breakable positions, it can include both old garbage positions and new accurate positions.

Currently step-debugging assumes: You edit a file. You re-run it instrumented. The whole file will be coherent for step debugging.

In hindsight, "What the heck, let's add instrument-function on top of instrument-whole-file, as an experiment" was a bad idea.

I think the list of caveats is becoming so long I should just deprecate this, for now.

It might be possible to loop back later and do individual-function instrumentation as its own, distinct feature. Obviously at the lowest level it would use the same technique for instrumenting code to be breakable. But it would get its own UX and data structures.

Over the last few days I spent a fair amount of time thinking about this. I seem to have stopped cycling between problems and possible work-arounds. I seem to have arrived at a fixed point -- the same, negative conclusion as my previous comment. :disappointed:

I really think the least-worst thing I could do, short-term, is change C-u C-M-x to do... nothing. Not instrument. Just print some advisory message ~= "This was a super-experimental feature that turned out to be unreliable and was removed. Instead please C-u C-u C-c C-c to instrument and run entire files."

Having said that, I could imagine some UX where you:

C-u C-u C-c C-c to expand/instrument/run a whole file for debugging. This is reliable even for things like contract macros.
Use commands to toggle one or more regions (like function definitions) to pay attention to the breakpoints, or not.
If you make any edits to the source file, you do need to go back to step 1.

In other words the whole file would be instrumented. It might just ignore certain regions of breakpoints.

Maybe by default (configurable) all breakpoints would be ignored, so you'd need to toggle regions on.

Maybe that experience would feel similar enough to how you imagined you wanted to use C-u C-M-x? Maybe??

If you have any thoughts about this, or your ideal workflow ("debugflow"?) I'd still love to hear about them.

First of all, thanks a lot for your detailed investigation. Regardless of the outcome it was really informative and insightful. The debugging experience you sketched (with instrumenting the whole file and using some sort of breakpoints) would definitely work for me. As I work mostly in IDEs like IntelliJ or Visual Studio Code it is actually the debugging experience that is closest to what I am used to. Even the instrumenting and the "run to here" feature took me a while to get used to as normally I debug with breakpoints. That is, breakpoints that stay active across multiple debugging sessions.

Having said that, I am really unexperienced in Racket (and Lisp in general) and, therefore, maybe my ideas of how debugging Racket programs should look like are completely off. I use debugging not just as a means of finding bugs, but also for program understanding. Most of the time I have to read other people's code and often the code is convoluted, imperative (i.e. modifies state) and the functions are large. As a result, exploring the program in the REPL by trying out the functions does not really help and inserting loads of printfs or using https://docs.racket-lang.org/debug/index.html becomes tedious after a while.

To summarize, I like your described UX ideas but I am also fine with anything else that helps me to make sense of complex programs.

Thanks for the feedback!

The debugging experience you sketched (with instrumenting the whole file and using some sort of breakpoints) would definitely work for me. As I work mostly in IDEs like IntelliJ or Visual Studio Code it is actually the debugging experience that is closest to what I am used to. Even the instrumenting and the "run to here" feature took me a while to get used to as normally I debug with breakpoints. That is, breakpoints that stay active across multiple debugging sessions.

Prior to Racket and Emacs I spent many years with C/C++ and IDEs like Borland or Microsoft. I was a big fan of the old book Writing Solid Code, and its advice that, for any new code, run it the first time by stepping through it in the debugger. Early in my time with Racket, I used the Dr Racket debugger like that. Although I don't do that for Racket, anymore, now I think I understand where you're coming from based on my own experience.

I had thought maybe you'd gravitated toward instrumenting a single function because you had a lisp background and/or were used to that from debugging Emacs Lisp code. My UX idea, above, was trying to preserve some of that spirit, on top of whole-file instrumentation.

But instead IIUC it seems like your main request would be for persistent breakpoints between sessions. Cool. It's actually been some years since I used that kind of debugger, so, I'll try to get some hands-on refresher, then think about how that could apply here.

(The underlying instrumentation stuff is pretty flexible, and would support even things like conditional breakpoints; the conditions could be arbitrary Racket code. But. Walk before run. The initial hurdle is probably just persistent, unconditional breakpoints, with an Emacs front end UI for managing those in the back end.)

I made a fresh issue #614 for a new feature.

I pushed a commit with a sketch of an initial implementation. (I'm not necessarily asking you to try it. Unless it's super convenient for you to try and you're really interested. Otherwise I'll refine it and dogfood it a bit more.)

Awesome, thank you. I will try it as soon as possible.

greghendershott / racket-mode

Using C-u C-M-x to instrument individual definitions for debugging is unreliable #612