dired-async and tramp: dired-get-file-for-visit: File no longer exists; type g to update Dired buffer [5 times]

[aagon]

Hello,

When copying a file asynchronously to a remote server via tramp, the file cannot be opened right away. When trying to open it from the dired buffer of the distant directory, I get the error :

dired-get-file-for-visit: File no longer exists; type g to update Dired buffer [5 times]

I have to revert the buffer multiple times, and wait several seconds. The file is several hundred bytes (it definitely does not take this long to copy).

The full debug log, when pressing Enter on the file in the remote directory buffer :

Debugger entered--Lisp error: (error #("File no longer exists; type g to update Dired buffer" 28 29 (font-lock-face help-key-binding face help-key-binding)))
  error(#("File no longer exists; type g to update Dired buffer" 28 29 (font-lock-face help-key-binding face help-key-binding)))
  dired-get-file-for-visit()
  dired-find-file()
  funcall-interactively(dired-find-file)
  command-execute(dired-find-file)

The steps to reproduce are quite straightforward in emacs -Q :

(package-initialize)
(load "/home/rico/.emacs.d/elpa/async-1.9.7.0.20230802.104021/async.elc")
(load "/home/rico/.emacs.d/elpa/async-1.9.7.0.20230802.104021/dired-async.elc")
(dired-async-mode)

Then opening two dired buffers, one for a local directory, one for a distant one, and copying a file with dired-do-copy (now advised, ofc) from the local to the remote, and trying to open the remote file immediately.

I am curious as to whether you can reproduce this.

Thank you for this very useful package.

Best,

Aymeric Agon-Rambosson

[thierryvolpiatto]

aagon @.***> writes:

1. ( ) text/plain (*) text/html

Hello,

When copying a file asynchronously to a remote server via tramp, the file cannot be opened right away. When trying to open it from the dired buffer of the distant directory, I get the error :

dired-get-file-for-visit: File no longer exists; type g to update Dired buffer [5 times]

I have to revert the buffer multiple times, and wait several seconds. The file is several hundred bytes (it definitely does not take this long to copy).

The full debug log, when pressing Enter on the file in the remote directory buffer :

Debugger entered--Lisp error: (error #("File no longer exists; type g to update Dired buffer" 28 29 (font-lock-face help-key-binding face help-key-binding))) error(#("File no longer exists; type g to update Dired buffer" 28 29 (font-lock-face help-key-binding face help-key-binding))) dired-get-file-for-visit() dired-find-file() funcall-interactively(dired-find-file) command-execute(dired-find-file)

The steps to reproduce are quite straightforward in emacs -Q :

(package-initialize) (load "/home/rico/.emacs.d/elpa/async-1.9.7.0.20230802.104021/async.elc") (load "/home/rico/.emacs.d/elpa/async-1.9.7.0.20230802.104021/dired-async.elc") (dired-async-mode)

Then opening two dired buffers, one for a local directory, one for a distant one, and copying a file with dired-do-copy (now advised, ofc) from the local to the remote, and trying to open the remote file immediately.

You have to wait the process end up, look at the dired-async message in mode-line.

-- Thierry

[aagon]

The problem happens definitely after the modeline shows "Asynchronous Copy of 1 on 1 file done", which I assume means that the child process as ended.

[thierryvolpiatto]

aagon @.***> writes:

1. ( ) text/plain (*) text/html

The problem happens definitely after the modeline shows "Asynchronous Copy of 1 on 1 file done", which I assume means that the child process as ended.

So probably you have to revert dired buffer (if it persists report an emacs bug, I am using dired-async only from Helm, I don't use dired personally).

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.*Message ID: @.***>

-- Thierry

[thierryvolpiatto]

Just tried to reproduce with your recipe and couldn't, works fine here. I used scp for the remote side.

[aagon]

I could reproduce this on another machine, with pure upstream emacs (not the debian version).

More specifically : the function file-exists-p with the new remote file as argument fails after I've had the notification that the asynchronous copy is done. This is very likely a bug, since I can do find-file on the remote file directly without errors (and this is coherent with what you told me regarding Helm, which is probably why you could not reproduce).

So, an updated recipe :

(package-initialize)
(load "/home/rico/.emacs.d/elpa/async-1.9.7.0.20230802.104021/async.elc")
(load "/home/rico/.emacs.d/elpa/async-1.9.7.0.20230802.104021/dired-async.elc")
(dired-async-mode)
(dired-do-copy) ; with point on "file", and setting the destination to remote directory "/scp:remote_server:"
(file-exists-p "/scp:remote_server:file") ; Eval this only after getting the notification. This fails several times (3 generally)

But as you noted, this works :

(package-initialize)
(load "/home/rico/.emacs.d/elpa/async-1.9.7.0.20230802.104021/async.elc")
(load "/home/rico/.emacs.d/elpa/async-1.9.7.0.20230802.104021/dired-async.elc")
(dired-async-mode)
(dired-do-copy) ; with point on "file", and setting the destination to remote directory "/scp:remote_server:"
(find-file "/scp:remote_server:file") ; This works right away (after getting the notification)

You could not reproduce with helm because the function you used does not run file-exists-p. This does not seem to be related to dired directly, since find-file-read-only fails as well, because it runs file-exists-p.

I've tried ssh and scp for the remote side, does not seem to change anything.

I'm not necessarily saying this is a bug in async. Before I go bother emacs-devel with this, could you try and reproduce with the updated recipe ?

Best,

Aymeric

[thierryvolpiatto]

aagon @.***> writes:

1. ( ) text/plain (*) text/html

I could reproduce this on another machine, with pure upstream emacs (not the debian version).

More specifically : the function file-exists-p with the new remote file as argument fails after I've had the notification that the asynchronous copy is done. This is very likely a bug, since I can do find-file on the remote file directly without errors (and this is coherent with what you told me regarding Helm, which is probably why you could not reproduce).

So, an updated recipe :

(package-initialize) (load "/home/rico/.emacs.d/elpa/async-1.9.7.0.20230802.104021/async.elc") (load "/home/rico/.emacs.d/elpa/async-1.9.7.0.20230802.104021/dired-async.elc") (dired-async-mode) (dired-do-copy) ; with point on "file", and setting the destination to remote directory "/scp:remote_server:" (file-exists-p "/scp:remote_server:file") ; Eval this only after getting the notification. This fails several times (3 generally)

Can't reproduce either, sorry.

But as you noted, this works :

(package-initialize) (load "/home/rico/.emacs.d/elpa/async-1.9.7.0.20230802.104021/async.elc") (load "/home/rico/.emacs.d/elpa/async-1.9.7.0.20230802.104021/dired-async.elc") (dired-async-mode) (dired-do-copy) ; with point on "file", and setting the destination to remote directory "/scp:remote_server:" (find-file "/scp:remote_server:file") ; This works right away (after getting the notification)

You could not reproduce with helm because the function you used does not run file-exists-p.

Of course it does internally.

-- Thierry

[aagon]

I am happy to report that I cannot reproduce anymore (this is unrelated to the last commits, note).

Thank you for your patience, closing.

[aagon]

I just managed to reproduce this exact issue, albeit only with a tramp connection requiring a password (scp as well as ssh methods). Not sure if this is a sufficient criterion for reproducing, just bringing it to your attention.

For reference for the unfortunate who can reproduce this, cleaning up the tramp connection using tramp-cleanup-this-connection like so solves the problem :

diff --git a/dired-async.el b/dired-async.el
index d4eb912..a6d46a7 100644
--- a/dired-async.el
+++ b/dired-async.el
@@ -172,7 +172,8 @@ or rename for `dired-async-skip-fast'."
                     do (with-current-buffer b
                          (when (and (not (file-remote-p default-directory nil t))
                                     (file-exists-p default-directory))
-                           (revert-buffer nil t)))))
+                           (revert-buffer nil t))
+                         (tramp-cleanup-this-connection))))
          ;; Finally send the success message.
          (funcall dired-async-message-function
                   "Asynchronous %s of %s on %s file%s done"

The cleaning is done after the copy is finished and the child process has died. The function tramp-cleanup-this-connection can be called inconditionnally, since it takes care of checking whether the dired buffer is a remote buffer.

[aagon]

More informations about this issue :

It cannot be reproduced when remote-file-name-inhibit-cache is set to t, which makes a lot of sense, as its docstring says:

The attributes of remote files are cached for better performance. If they are changed outside of Emacs's control, the cached values become invalid, and must be reread. If you are sure that nothing other than Emacs changes the files, you can set this variable to nil.

Having an asynchronous child emacs process change the file attributes of a directory (by creating a file in it, for instance) outside of the parent emacs' control is liable to cause issues.

According to the value of this variable, tramp maintains a cache of file properties (see file lisp/net/tramp-cache.el).

One solution, other that disabling the remote file attributes file entirely, would be to invalidate the relevant entries of the cache during the callback function. This is less violent than the solution of the previous comment.

[thierryvolpiatto]

aagon @.***> writes:

1. ( ) text/plain (*) text/html

More informations about this issue :

It cannot be reproduced when remote-file-name-inhibit-cache is set to t, which makes a lot of sense, as its docstring says:

The attributes of remote files are cached for better performance.
If they are changed outside of Emacs's control, the cached values
become invalid, and must be reread. If you are sure that nothing
other than Emacs changes the files, you can set this variable to nil.

Having an asynchronous child emacs process change the file attributes of a directory (by creating a file in it, for instance) outside of the parent emacs' control is liable to cause issues.

According to the value of this variable, tramp maintains a cache of file properties (see file lisp/net/tramp-cache.el).

One solution, other that disabling the remote file attributes file entirely, would be to invalidate the relevant entries of the cache during the callback function. This is less violent than the solution of the previous comment.

Isn't this more a dired bug rather than an async bug? Dired is a old UI to browse directories, I personally don't like it, I use my personal counterpart which is helm-find-files, which doesn't suffer from these problems of dired buffers not updated (I use filenotify internally).

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.*Message ID: @.***>

-- Thierry

[aagon]

Isn't this more a dired bug rather than an async bug?

This is not a dired bug. dired relies on file-exists-p, which itself relies on tramp-file-name-handler when the file is remote. I assume that it is this last function that provides the (wrong) answer, according to the cache tramp maintains.

Any interface that relies on the cache tramp provides, through its file name handler, will have the same issue.

helm dodges this issue because, as you say, it relies on filenotify rather than the tramp cache.

But the fact that the cache is invalid is hardly tramp's fault, since the changes to the remote file system are done outside of his control by the child emacs, and the docstring warns that activating the cache without time expiration is dangerous.

The interaction of dired-async-mode and remote-file-name-inhibit-cache set to nil causes this.

[aagon]

I have a fully non-interactive recipe :

;; Evaluate with (load "~/reproduce.el")

(package-initialize)

(load "~/.emacs.d/elpa/async-1.9.8/async.elc")
(load "~/.emacs.d/elpa/async-1.9.8/dired-async.elc")

;; Needed for dired-do-copy, adjust
(load "~/temp/with-simulated-input/with-simulated-input.elc")

(dired-async-mode)
(setq remote-file-name-inhibit-cache nil)

;; Open the dired buffers first, important
(find-file "/scp:macmini_local:"); This must be a remote location, not mounted in memory (no tmpfs)
(find-file-other-window "~/")

;; Place point on file to copy
(search-forward "finder-inf.el")

;; Copy (make sure the file does not already exist on the other side)
(with-simulated-input
    ((wsi-simulate-idle-time 3) "/scp:macmini_local:" (wsi-simulate-idle-time 3) "RET")
  (dired-do-copy))

;; Wait for copy to finish
(sit-for 5)             ; Adjust according to file size and connection speed

;; At this point, the file has finished being copied to the remote fs

;; Test its existence according to file-exists-p (I see "NOK")
(if (file-exists-p "/scp:macmini_local:finder-inf.el")
    (message "OK")
  (message "NOK"))

;; Test with dired anyway

;; Go to the window holding the remote tramp buffer
(other-window 1)

;; Revert buffer to make the file appear
(revert-buffer t t t)

;; Go to the file
(search-forward "finder-inf.el")

;; Try to open it through dired, and report result (I see "NOK")
(if (ignore-errors (dired-find-file))
    (message "OK")
  (message "NOK"))

;; Now try to open directly (works for me)
(find-file "/scp:macmini_local:finder-inf.el")

It requires the library with-simulated-input, however, I could not make a full non interactive recipe without it. I hope you can reproduce.

It turns out that whether you have a password or passphrase to the remote server does not matter. However, the remote directory must be on a real non-volatile storage (no tmpfs mounted /tmp directory, for instance). I assume tramp keeps no cache of files stored on remote memory.

[aagon]

The interaction of dired-async-mode and remote-file-name-inhibit-cache set to nil causes this.

To be honest, a very specific set of settings is required for this bug to be reproductible, and dired-async cannot be entirely blamed for this. After all, if the user activates the tramp cache without expiration, and then proceeds to change the remote file system without tramp knowing, it could be argued that he only has himself to blame. So I would understand if you did not think dired-async to be the good place to fix this. But warning the user that it is preferable to disable the tramp cache could be a good idea.

For reference, I ended up solving the problem by advising dired-revert like this (my previous solution of advising dired-async-after-file-create did not work) :

(defun my/invalidate-cache(&optional _arg _noconfirm)
  "Invalidate the directory attribute cache that dired gets from
tramp"
  (let ((dir (expand-file-name dired-directory)))
    (when (and (fboundp 'tramp-tramp-file-p)
           (tramp-tramp-file-p dir))
      (tramp-flush-directory-properties (tramp-dissect-file-name dir)
                    (tramp-file-local-name dir)))))

(advice-add #'dired-revert :before #'my/invalidate-cache)

Now, when the dired buffer is reverted, stale cache entries are removed, and dired-find-files works correctly.

But I'm still curious to see whether you can reproduce my last recipe, just to check I am not mad.