stumped by this bytecode-caching problem

[mbutterick]

If Pollen generates bytecode for #lang pollen files, builds are about 20% faster. But there is a bytecode-caching bug I can’t figure out, illustrated by the sample code below.

This sample simulates a render of a Pollen source "doc.rkt" that relies on "pollen.rkt". After "pollen.rkt" is changed and the bytecode regenerated for both files, only "pollen.rkt" can be re-evaluated correctly. When "doc.rkt" is re-evaluated, it shows the older value.

Why is this so?

At first, I thought the dependency manager was failing to notice that "doc.rkt" relies on "pollen.rkt" (and thus should be recompiled when "pollen.rkt" changes). But the tests using module-recorded-dependencies seem to disprove this theory.

Then I thought that the bytecode for "doc.rkt" wasn’t being refreshed on the second call to managed-compile-zo. But even if I erase the first set of bytecode files, the problem persists.

My only remaining theory is that managed-compile-zo has some deeper cache mechanism that needs to be overcome, but if so how?

#lang at-exp debug racket
(require compiler/cm compiler/depend rackunit)

(display-to-file @string-append{
 #lang racket/base
 (provide id)
 (define id "first")} "pollen.rkt" #:exists 'replace)

(display-to-file @string-append{#lang pollen
 ◊id doc} "doc.rkt" #:exists 'replace)

(managed-compile-zo "pollen.rkt")
(managed-compile-zo "doc.rkt")

;; proves that dependency manager notices that "doc.rkt" relies on "pollen.rkt"
(check-true (and (member (path->complete-path (string->path "pollen.rkt")) (module-recorded-dependencies "doc.rkt")) #true))

(parameterize ([current-namespace (make-base-namespace)])
  (check-equal? (dynamic-require "pollen.rkt" 'id) "first"))

(parameterize ([current-namespace (make-base-namespace)])
  (check-equal? (dynamic-require "doc.rkt" 'doc) "first doc"))

(display-to-file @string-append{
 #lang racket/base
 (provide id)
 (define id "second")} "pollen.rkt" #:exists 'replace)

;; we erase the bytecode!
;; we erase the bytecode!
(for ([bc '("compiled/pollen_rkt.zo" "compiled/doc_rkt.zo")])
     (when (file-exists? bc)
       (delete-file bc)))

(managed-compile-zo "pollen.rkt")
(managed-compile-zo "doc.rkt")

(check-true (and (member (path->complete-path (string->path "pollen.rkt")) (module-recorded-dependencies "doc.rkt")) #true))

(parameterize ([current-namespace (make-base-namespace)])
  (check-equal? (dynamic-require "pollen.rkt" 'id) "second"))

(parameterize ([current-namespace (make-base-namespace)])
  (check-equal? (dynamic-require "doc.rkt" 'doc) "second doc")) ; huh?!?!

[MichaelMMacLeod]

This is indeed weird. I'm not sure what's going on here, but it's worth noting that v7.7 [cs] passes these tests, while v7.7 [non-cs] doesn't.

[mbutterick]

Ah yes thanks, I should’ve tried that. Well, that tells me it’s a Racket bug (perhaps even longstanding).

[mbutterick]

For now, I put in the extra bytecode patch for Racket CS (and will extend it to BC later, if someone smarter figures out a patch). Still, given the comparatively slower performance of CS, the performance improvement is more valuable there.

[mbutterick]

Turns out it is not a bug in Racket, but rather a side effect of undefined behavior.