Internal error on every key press when using a custom collections path in #lang r6rs

Bjarno commented 3 years ago

Description

While working with R6RS in DrRacket, I have encountered the following issue. When working in some code that calls a library, every time I enter a key press in the text editor, an internal error is displayed at the top of the DrRacket window.

The full error is as follows:

internal error: standard-module-name-resolver: collection not found
  for module path: (lib "my-library/foo-lib.rkt")
  collection: "my-library"
  in collection directories:
   /home/bjarno/.racket/7.9/collects
   /usr/share/racket/collects
   ... [161 additional linked and package directories]
  context...:
   /usr/share/racket/collects/syntax/private/id-table.rkt:454:11: free-id-table-ref
   /usr/share/racket/pkgs/drracket-tool-lib/drracket/private/syncheck/traversals.rkt:1467:0: add-id
   /usr/share/racket/pkgs/drracket-tool-lib/drracket/private/syncheck/traversals.rkt:415:11: for-loop
   /usr/share/racket/pkgs/drracket-tool-lib/drracket/private/syncheck/traversals.rkt:183:2: level+tail+mod-loop
   /usr/share/racket/pkgs/drracket-tool-lib/drracket/private/syncheck/traversals.rkt:47:10: expanded-expression
   /usr/share/racket/pkgs/drracket/drracket/private/syncheck/online-comp.rkt:43:0: build-trace
   /usr/share/racket/pkgs/drracket/drracket/private/syncheck/online-comp.rkt:97:13
   /usr/share/racket/pkgs/drracket/drracket/private/syncheck/online-comp.rkt:91:5: loop

There is no error when the code is executed (by pressing Run). The error is only present when using DrRacket as the IDE. The error can be closed (by clicking Hide), but it appears again once a key is pressed in the editor pane. And, even more annoying, each key press already closes the error, but after the syntax checker fails, the error shows up again.

It only happens when I'm...

Working with R6RS code (R6RS being the language that uses a library, it doesn't occur when using Racket*); and
The library is installed in a custom collects folder (as configured in the Choose Language window), it doesn't occur when I place my library in one of the default collect folders (e.g., ~/.racket/7.9/collects on my system); and
This is the weird thing: it only happens when I use a variable (as defined by the library in the custom collect folder) twice...

* I have not checked any other languages besides Racket and R6RS

How to reproduce:

Download collects-bug-library.zip; which contains a simple Racket library (the same error would occur when using an R6RS library!) that exports a single procedure foo (its source code can be summarised as (define (foo) 42) (provide foo)). Extract it somewhere where Racket doesn't find it by default (e.g., your Downloads folder).
Open DrRacket, Make sure that "Determine language from source" is selected, and open the "Choose Language" menu. Add the collects-bug-library folder (not the my-library inside that folder) to the collection paths (below the default one).

Enter the following code in the DrRacket IDE

#lang r6rs
(library
(bug-test)
(export d)
(import (only (rnrs base) define let) ; or (rnrs base), doesn't matter too much here
     (my-library foo-lib))
(define d (let () foo)))

Press some keys (e.g., add a comment or some whitespace): no internal error pops up.
Change line 8 to (define d (let () foo foo))) (just add another reference to foo).
Oh no! An internal error (see above)!
Every time when you press a new key (add another comment, or some whitespace) the internal error disappears and reappears again a few moments later.

Restarting DrRacket after changing the collection paths doesn't solve the issue.

As mentioned earlier (but now is the opportune moment to test it), the same issue is not present when using Racket. When I replace my test code to the following, no internal error pops up (despite the fact that both implementations in Racket and R6RS are almost equivalent).

#lang racket
(require my-library/foo-lib)
(provide d)
(define d (let () foo foo))

What I have found so far

By checking the stack trace of the internal error, I have found that the issue lies somewhere in the syntax checker (syncheck). In share/pkgs/drracket-tool-lib/drracket/private/syncheck/traversals.rkt the add-id procedure adds a new identifier to the so-called free-id-table. The first time the syntax checker encounters the identifier foo, the binding does not yet exist, and the execution of free-id-table-ref succeeds without an error, the result is saved by add-id back into the free-id-table. The second time when the syntax checker encounters the identifier foo, the binding already exists and the id-table-ref is called again. This means that the id-table-ref procedure in collects/syntax/private/id-table.rkt is called, which checks if the id-table already contains a mapping for foo, which it has. However, it still needs to check whether both identifiers are the same free variable (by using free-variable=?) to make sure that the old result can be used again. At least, that's my understanding from reading the stack trace, and the two files. So if I'm wrong, feel free to correct me!

Just to check, I have modified collects/syntax/private/id-table.rkt to contain some displays (I have no experience with debugging DrRacket code, so that seemed to be the easiest way for me to test some things :P ) and have found that during the execution of free-variable=? that internal error is thrown (all displays before the execution of free-variable=? are performed, but none afterwards, at least while syntax checking the second occurrence of foo). I was however not able to find an easy-to-debug definition of free-variable=?, so that's where my debugging efforts ended.

From what I can tell (from reading the documentation of free-variable=?), I think that free-variable=? needs to know information about the modules (and thus, the collection directories) to determine whether two syntactic objects are referring to the binding expression. I expect that (and now I'm really guessing), when using R6RS, free-variable=? is not aware that the collection paths are non-default, which in turn results in free-variable=? not finding the module/file where foo is defined which results in the internal error: standard-module-name-resolver: collection not found. Or at least, there is some kind of incompatibility where free-variable=? loses information concerning the use of collections/modules/libraries in when using #lang R6RS.

What now?

If this description alone does not contain sufficient information to identify and fix the bug, I'm willing to delve deeper into this issue. But as I have no prior experience with (Dr)Racket's source code, I do not know what I can try next. So if you have any advice/ideas that might help to troubleshoot this issue, also feel free to share them!

rfindler commented 3 years ago

I have not yet investigated deeply, but probably the bug has to do with check syntax using the settings in the language dialog (where you added a collection) properly. If you can use raco pkg to set up a package then you should be able to avoid this problem. If you use a name for your pkg that's a directory (and include a / in the raco pkg install command) you should be able to work around this problem.

Bjarno commented 3 years ago

Indeed, packaging the library as a raco package seems to work fine and it also seems to support our use case. The library in which we first encountered this problem with was one used by students as part of an undergraduate course (in which they have to make changes to it as part of the exercise sessions/assignments). We used the ability to modify the collection paths, as that seemed to be the easiest solution for them. I was not yet aware of the functionality to install/link local packages using raco pkg, so I hadn't thought about trying that yet. Thanks for the tip/workaround!

rfindler commented 3 years ago

For students it may make sense to remain entirely within drr. You can put a pkg at a url or on the pkg server and they can remain within drr to install it (use File | Install Package...). Then, to access the source code they would use the File|Open Require Path... menu item.

If your students are comfortable with the command line there are also ways to set up packages to cooperate with git.

racket / drracket