Builds - Reuse of `*.pkg` files ?

[callendorph]

In this project:

https://github.com/callendorph/lbstanza-tidy

I have two build targets - main and tidy-tests https://github.com/callendorph/lbstanza-tidy/blob/36cebf5516b0668bbdf6b3ef5c8f57418d81df43/stanza.proj#L11

They both should generate the *.pkg artifacts in the pkgs directory.

What I find is that even if I run stanza build main before running stanza build tidy-tests - The tidy-tests target will rebuild all of the *.pkg files from scratch. Example:

Here is the commands I run in the command line:

$> make clean
rm ./pkgs/*.pkg
$> time stanza build

real    0m10.422s
user    0m10.203s
sys     0m0.125s

$> time stanza build tidy-tests

real    0m12.659s
user    0m11.078s
sys     0m0.391s

Here is a log of filesystem events generated when this sequence ran:

https://gist.github.com/callendorph/39c27a2655939ae7774fc1fecf9a4a50

Here is a zip file of the strace output for stanza build tidy-tests

strace -f stanza build tidy-tests &> strace.list

stanza_build_tidy-tests_strace.zip

In this trace you will see the following:

1. Around lines 1552 - 1935 - it starts reading the *.pkg files for the Enums as O_RDONLY. This is what I expect it to do
2. At around 14541 -16143 - It starts opening these same *.pkg files as O_WRONLY|O_CREAT|O_TRUNC which I believe will clobber the existing file.

Line: 15150

[pid  6995] open("pkgs/tidy$Enums$TidyAttrId.pkg", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 6

My expectation would be that for these packages, when they get built by stanza build that they would be reused in stanza build tidy-tests. Is this not the right expectation ? Is that not what is happening here ?

[m-hilgendorf]

Hi! I think I can provide some context to this one in case anyone else runs into it -

The build-test command will add a TESTING define to the compiler invocation, which invalidates the cached .pkg files.

Consider this code :

defpackage my-package :
  import core
  import collections

public-when(TESTING) defn my-private-function () :
  "hello!"

defpackage my-package/tests :
  import core
  import collections

deftest test-my-private-function :
  #ASSERT(my-private-function() == "hello!")

The cached .pkg file of my-package will need to know: what are my functions, and what are their visibility? In this case, the TESTING flag will invalidate the .pkg, since it changes the visibility of a function in the package, and needs to be recompiled. At the moment the compiler can't guarantee that the cached .pkg files compiled with one set of flags is the same as another, so it makes a conservative choice to recompile everything. It would be great to see more incrementalism in the future, though!

One way to work around this is to have two pkg directories in your build folder, one for artifacts compiled normally, and one for those compiled during testing.

[callendorph]

OK - thank you!

[CuppoJava]

Thanks for the explanation Mike. That is exactly what's happening.