Non-determinism in target task

khardenstine commented 7 years ago

`build.boot` content (if applicable)

(set-env!
  :resource-paths #{"resources"})

(deftask aux []
         (merge-env! :source-paths #{"src"}
                     :resource-paths #{"resources-aux"}))

Description

Resources are not being copied deterministically into the target directory, when running boot target

Steps to reproduce

See this repo for a complete demo: https://github.com/khardenstine/boot-target-resource-reproduction The two scripts ./build-with-deps.sh and ./build-no-deps.sh both are simply invoking: boot aux target. The build-with-deps script installs a few node modules in the resource directory first. Running boot aux target with those additional files results in none of the resources from "resources-aux" being copied over.

Interestingly a subtle change of moving the source-paths up into the global set-env! fixes this (see works.boot as opposed to broken.boot).

martinklepsch commented 6 years ago

I tried to minimize the repro a bit but there seem to be a few gotchas:

it seems important that the set-env! command also sets :source-paths
it seems important that there is a large number of files in the directory that is added during the second merge-env! call

I'm using the following build.boot for reproduction:

(when (= "true" (System/getenv "BOOT_DOUBLE_SET_ENV"))
  (prn 'set-env :resource-paths #{"resources"})
  (set-env! :resource-paths #{"resources"}
            :source-paths #{"src"}))

(prn 'set-env :resource-paths #{"resources" "resources-aux"})
(set-env! :resource-paths #{"resources" "resources-aux"}
          :source-paths #{"src"})

(prn :env (select-keys (get-env) [:resource-paths :source-paths]))

Then running it like this (fish shell syntax):

# working
env BOOT_DOUBLE_SET_ENV=false boot target; ll target/; find target/ -type f | wc -l

# failing
env BOOT_DOUBLE_SET_ENV=true boot target; ll target/; find target/ -type f | wc -l

Interestingly the number of files that ends up in target/ when calling set-env! twice varies significantly. Sometimes it's one file, other times it's complete, most often it seems to be a few hundred files (of the 2400 in the node_modules dir).

Some verbose logs

Below are some interesting potions of the -vv output for the failing command from a situation where the target directory ended up having just one file in it.

set-env :resource-paths #{"resources"}
registering resources [:create :modify :delete]
sending change event
Acquired java.util.concurrent.Semaphore@772d4c88[Permits = 0]...
Syncing project dirs to temp dirs...
Filesystem: copying a-sanity-check.edn...
Sync complete.
Released java.util.concurrent.Semaphore@772d4c88[Permits = 1]...
watch service closed
registering src [:create :modify :delete]
registering src/someapp [:create :modify :delete]
registering resources [:create :modify :delete]
sending change event
Acquired java.util.concurrent.Semaphore@772d4c88[Permits = 0]...
Syncing project dirs to temp dirs...
Filesystem: copying someapp/app.clj...
Sync complete.
Released java.util.concurrent.Semaphore@772d4c88[Permits = 1]...
set-env :resource-paths #{"resources-aux" "resources"}
Acquired java.util.concurrent.Semaphore@772d4c88[Permits = 0]...
Syncing project dirs to temp dirs...
watch service closed
registering src [:create :modify :delete]
registering src/someapp [:create :modify :delete]
registering resources-aux [:create :modify :delete]
...
registering resources-aux/node_modules/winston/test [:create :modify :delete]
registering resources-aux/node_modules/winston/test/transports [:create :modify :delete]
registering resources [:create :modify :delete]
sending change event
:env {:resource-paths #{"resources-aux" "resources"}, :source-paths #{"src"}}
Acquired java.util.concurrent.Semaphore@72d5ae44[Permits = 0]...
Committing fileset...
Commit: adding   d41d8cd98f00b204e9800998ecf8427e.1513764590000 a-sanity-check.edn...
Commit: adding   9c14bf3504b15c6bb3b12b7e76f6b846.1513764590000 someapp/app.clj...
Commit complete.
Released java.util.concurrent.Semaphore@72d5ae44[Permits = 1]...
Writing target dir(s)...
Filesystem: copying node_modules/lodash/package/_compareAscending.js...
Filesystem: copying node_modules/lodash/package/fp/flatMap.js...
Filesystem: copying node_modules/lodash/package/_hashGet.js...
... 
Filesystem: copying node_modules/lodash/forOwnRight.js...
Filesystem: linking a-sanity-check.edn...
Filesystem: copying node_modules/lodash/fp/thru.js...

The only file in target/ was a-sanity-check.edn which was linked in the second to last line of the verbose output while other files were still being copied. The boot process terminated pretty much immediately after that file was linked.

nimaai commented 4 years ago

I think I have a problem which might be related to this issue:

(deftask uberjar
  "Build an uberjar of the application."
  []
  (comp (with-pass-thru _
          (set-env! :resource-paths #{"resources/all" "resources/prod"}))
        (aot)
        (uber)
        (sift :add-resource #{"resources/all" "resources/prod"})
        (jar)
        (target)))

Without the sift only the resources from resources/all are there in the uberjar file, so set-env! seems not to be working properly in this case or I understand it wrongly.

boot-clj / boot