NoSuchFileException related to Intellij __jb_tmp__ safe write files

kennyjwilli commented 8 years ago

I seem to be inconsistently getting this error when writing the target dirs after a reload. It goes away if I restart my boot process.

Writing main.cljs.edn...
Compiling ClojureScript...
• main.js
Writing target dir(s)...
java.util.concurrent.ExecutionException: java.nio.file.NoSuchFileException: target/some/project/myns.cljs___jb_tmp___
      java.nio.file.NoSuchFileException: target/some/project/myns.cljs___jb_tmp___
    file: "target/some/project/myns.cljs___jb_tmp___"
sun.nio.fs.UnixException.translateToIOException               UnixException.java:   86
  sun.nio.fs.UnixException.rethrowAsIOException               UnixException.java:  102
  sun.nio.fs.UnixException.rethrowAsIOException               UnixException.java:  107
   sun.nio.fs.UnixFileSystemProvider.implDelete      UnixFileSystemProvider.java:  244
   sun.nio.fs.AbstractFileSystemProvider.delete  AbstractFileSystemProvider.java:  103
                                            ...                                       
                        boot.filesystem/delete!                   filesystem.clj:  163
                         boot.filesystem/patch!                   filesystem.clj:  178
                                            ...                                       
                clojure.core/apply/invokeStatic                         core.clj:  652
                        clojure.core/partial/fn                         core.clj: 2534
                                            ...                                       
      boot.core/fileset-syncer/fn/iter/fn/fn/fn                         core.clj:  930
            clojure.core/binding-conveyor-fn/fn                         core.clj: 1938
                                            ...                                       
Elapsed time: 0.162 sec

From the discussion on #boot on Slack it seems the __jb_tmp__ is an IntelliJ file that is semi-atomically created (using SafeFileOutputStream according to cfleming).

arichiardi commented 8 years ago

~~This would also solve the issue with emacs' lock files.~~

kennyjwilli commented 8 years ago

Example project with error below. https://github.com/kennyjwilli/boot-files-issue

Writing main.cljs.edn...
Compiling ClojureScript...
• main.js
Writing target dir(s)...
java.util.concurrent.ExecutionException: java.nio.file.NoSuchFileException: target/boot_files/core.cljs___jb_tmp___
      java.nio.file.NoSuchFileException: target/boot_files/core.cljs___jb_tmp___
    file: "target/boot_files/core.cljs___jb_tmp___"
sun.nio.fs.UnixException.translateToIOException               UnixException.java:   86
  sun.nio.fs.UnixException.rethrowAsIOException               UnixException.java:  102
  sun.nio.fs.UnixException.rethrowAsIOException               UnixException.java:  107
   sun.nio.fs.UnixFileSystemProvider.implDelete      UnixFileSystemProvider.java:  244
   sun.nio.fs.AbstractFileSystemProvider.delete  AbstractFileSystemProvider.java:  103
                                            ...                                       
                        boot.filesystem/delete!                   filesystem.clj:  163
                         boot.filesystem/patch!                   filesystem.clj:  178
                                            ...                                       
                clojure.core/apply/invokeStatic                         core.clj:  652
                        clojure.core/partial/fn                         core.clj: 2534
                                            ...                                       
      boot.core/fileset-syncer/fn/iter/fn/fn/fn                         core.clj:  930
            clojure.core/binding-conveyor-fn/fn                         core.clj: 1938
                                            ...                                       
Elapsed time: 0.101 sec

kennyjwilli commented 8 years ago

It appears that the exception is thrown in every boot process (e.g. I have boot web-dev running in one terminal and boot repl running in another. When the exception is thrown in the boot web-dev instance the exception also gets thrown in the boot repl terminal).

kennyjwilli commented 6 years ago

This is still occuring for me. The exception is slightly different though. Related problem?

Uncaught exception in thread Thread-29:
                                  java.lang.Thread.run                   Thread.java:  745
                                                   ...                                    
                               boot.core/watch-dirs/fn                      core.clj:  751
                    boot.core/set-user-dirs!/on-change                      core.clj:  194
                             boot.core/sync-user-dirs!                      core.clj:  139
                                                   ...                                    
                                      boot.core/patch!                      core.clj:  724
                                   clojure.core/reduce                      core.clj: 6704
                           clojure.core.protocols/fn/G                 protocols.clj:   13
                             clojure.core.protocols/fn                 protocols.clj:   75
                    clojure.core.protocols/iter-reduce                 protocols.clj:   49
                               boot.core/patch!/merge'                      core.clj:  722
                                                   ...                                    
                                boot.filesystem/mktree                filesystem.clj:  113
                                                   ...                                    
                              boot.file/walk-file-tree                      file.clj:   59
                      java.nio.file.Files.walkFileTree                    Files.java: 2706
                     java.nio.file.FileTreeWalker.next           FileTreeWalker.java:  372
                    java.nio.file.FileTreeWalker.visit           FileTreeWalker.java:  276
            java.nio.file.FileTreeWalker.getAttributes           FileTreeWalker.java:  225
                    java.nio.file.Files.readAttributes                    Files.java: 1737
     sun.nio.fs.LinuxFileSystemProvider.readAttributes  LinuxFileSystemProvider.java:   99
      sun.nio.fs.UnixFileSystemProvider.readAttributes   UnixFileSystemProvider.java:  144
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes   UnixFileAttributeViews.java:   55
         sun.nio.fs.UnixException.rethrowAsIOException            UnixException.java:  107
         sun.nio.fs.UnixException.rethrowAsIOException            UnixException.java:  102
       sun.nio.fs.UnixException.translateToIOException            UnixException.java:   86
java.nio.file.NoSuchFileException: src/compute/ui_frontend/re_frame.cljs___jb_old___
    file: "src/compute/ui_frontend/re_frame.cljs___jb_old___"

martinklepsch commented 6 years ago

@kennyjwilli thanks for reporting!

Is the repro (https://github.com/kennyjwilli/boot-files-issue) still intact and can you reproduce the issue there?
Could you please add some minimal information what to run to reproduce the issue?

kennyjwilli commented 6 years ago

I'm struggling to get it to reproduce outside of our large CLJS project. We have a project with 50+ CLJS namespaces and occasionally while recompiling, that error is thrown. That project takes 2s or more to compile compared to the 0.2s the boot-files-issue repo takes. I'll take a look at Boot's code to see if this is an easy fix.

kennyjwilli commented 6 years ago

It seems to happen if there are rapid file changes while it is compiling the CLJS.

kennyjwilli commented 6 years ago

It seems like these files are created from IntelliJ's "safe write" feature: https://stackoverflow.com/questions/23271895/temporary-jb-old-file-in-phpstorm-and-webstorm-causing-errors.

kennyjwilli commented 6 years ago

Disabling "safe write" fixes the issue.

It's tough to debug when I can't get the Cursive debugger to break on the NoSuchFileException. I'd really like to know what params are being passed to each function so I at least can replicate the behavior in the REPL.

It's also interesting that I can only reliably reproduce the error when a CLJS REPL is running.

kennyjwilli commented 6 years ago

Best guess at what is happening:

File is saved in editor with "safe write" enabled
IntelliJ writes the changed file to its temp file (e.g. src/compute/ui_frontend/re_frame.cljs___jb_tmp___)
Boot creates the filesystem tree which includes the jb_tmp file
IntelliJ safely writes to the original file and deletes its temporary file.
Boot attempts to walk the file tree it created in 1 and a java.nio.file.NoSuchFileException is thrown because the jb_tmp file was included in the file tree but deleted before the file tree could be walked.

This is a race condition which would explain why it only happens occasionally.

kennyjwilli commented 6 years ago

I have added the regex .*\_\_\_$ to my .bootignore which should cause Boot to ignore any files ending with ___ but the issue still occurs.

martinklepsch commented 6 years ago

There’s an outstanding PR (which will be part of the next release) that makes bootignore behavior more consistent: https://github.com/boot-clj/boot/pull/663

You could try merging those changes and rebuilding boot locally to see if that makes bootignore work as expected.

I’m also wondering if there are ways to make IntelliJ write those files elsewhere? I roughly remember similar issues related to emacs storing temporary files in watched directories. (I think there’s some stuff in the wiki about this, will update with a link later.)

martinklepsch commented 6 years ago

It seems there are other people also having issues with IntelliJ's safe write feature: https://github.com/cgrand/enlive/issues/143 — the universal solution seems to be disabling safe write 😏

martinklepsch commented 6 years ago

@kennyjwilli I updated the name if this ticket, feel free to change if you feel this does not accurately reflect the nature of this problem.

martinklepsch commented 6 years ago

Rereading #663 it seems that this won't affect the issue at hand since .bootignore really just takes care of ignoring directories, not individual files.

martinklepsch commented 6 years ago

Ideas for changes that could help future users avoid this kind of trouble:

Document it in the Cursive page in the wiki
Catch NoSuchFileExceptions occuring in boot.file/walk-file-tree and print additional information if the filepath contains __jb_old__ or similar

witek commented 6 years ago

I have the same problem using Spacemacs:

Exception in thread "Thread-15" java.nio.file.NoSuchFileException: src/tao/.#tests_spa.cljs at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.nio.file.Files.readAttributes(Files.java:1737) at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225) at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372) at java.nio.file.Files.walkFileTree(Files.java:2706) at boot.file$walk_file_tree.doInvoke(file.clj:59) at clojure.lang.RestFn.invoke(RestFn.java:425) at boot.filesystem$mktree.doInvoke(filesystem.clj:113) at clojure.lang.RestFn.invoke(RestFn.java:439) at boot.core$patch_BANG_$merge_SINGLEQUOTE___728.invoke(core.clj:722) at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:49) at clojure.core.protocols$fn__7839.invokeStatic(protocols.clj:75) at clojure.core.protocols$fn__7839.invoke(protocols.clj:75) at clojure.core.protocols$fn__7781$G__7776__7794.invoke(protocols.clj:13) at clojure.core$reduce.invokeStatic(core.clj:6748) at clojure.core$reduce.invoke(core.clj:6730) at boot.core$patch_BANG_.doInvoke(core.clj:724) at clojure.lang.RestFn.invoke(RestFn.java:464) at boot.core$sync_user_dirs_BANG_.invoke(core.clj:139) at boot.core$set_user_dirs_BANG_$on_change__459.invoke(core.clj:194) at boot.core$watch_dirs$fn__737.invoke(core.clj:751) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748)

larkery commented 5 years ago

Whilst waiting for this problem to go away I did this hack:

(require 'boot.filesystem)
(in-ns 'boot.filesystem)
(let [mkvisitor-0 mkvisitor]
  (defn mkvisitor [^Path root tree & {:keys [ignore]}]
    (let [^SimpleFileVisitor v0 (mkvisitor-0 root tree :ignore ignore)]
      (proxy [SimpleFileVisitor] []
        (postVisitDirectory [t e] (.postVisitDirectory v0 t e))
        (preVisitDirectory [path attr] (.preVisitDirectory v0 path attr))
        (visitFile [path attr] (.visitFile v0 path attr))
        (visitFileFailed [path exc] FileVisitResult/CONTINUE)))))
(in-ns 'boot.user)

This appears to make this class of issue go away for me.

boot-clj / boot

NoSuchFileException related to Intellij __jb_tmp__ safe write files #471