Can not do dired on ~/Documents when it is in iCloud on Catalina

[pedz]

emacs-version returns:

GNU Emacs 27.0.50 (build 1, x86_64-apple-darwin18.7.0, NS appkit-1894.00 Version 10.15 (Build 19A583)) of 2019-10-10

^X ^F /~/Documents results in:

Debugger entered--Lisp error: (error "Listing directory failed but ‘access-file’ worked")
  signal(error ("Listing directory failed but ‘access-file’ worked"))
  error("Listing directory failed but `access-file' worked")
  insert-directory("/Users/pedz/Documents/" "-al" nil t)
  dired-insert-directory("/Users/pedz/Documents/" "-al" nil nil t)
  dired-readin-insert()
  dired-readin()
  dired-internal-noselect("~/Documents/" nil)
  dired-noselect("~/Documents/")
  run-hook-with-args-until-success(dired-noselect "~/Documents/")
  find-file-noselect("~/Documents/" nil nil t)
  find-file("~/Documents/" t)
  funcall-interactively(find-file "~/Documents/" t)
  call-interactively(find-file nil nil)
  command-execute(find-file)

This is on Catalina with my Documents (and Desktop) in iCloud. From an outside Terminal, you see:

ls -l@d ~/Documents 
drwx------@ 4 pedz  staff  128 Oct 10 10:52 /Users/pedz/Documents
    com.apple.icloud.desktop      8 

I've tried a few things but have not been able to resolve this yet. This was originally reported on the Emacs help mailing list.

I've tried adding Emacs to the "Full Disk Access" list as well as "Developer Tools" in System Preferences => Security & Privacy => Privacy

I've discovered that Emacs.app and all of its files and directories have the com.apple.quarantine attribute so I've removed that.

And, as we probably all know, this version starts up a ruby shell and then does an exec. I've tried changing that to "command" so that the parent process that the Finder starts is a direct ancestor of the "ls" command that Emacs eventually does.

I've also tried to horse around in the Terminal.app starting sub shell, irb, etc trying to recreate it outside of Emacs and have not been able to.

[XeCycle]

I somehow got it working. Add Emacs.app to "Full Disk Access", then inside Emacs, use ns-open-file-using-panel (bound to Cmd-o by default) to open an arbitrary file directly inside ~/Documents. After that emacs can dired in ~/Documents. Similarly for ~/Downloads, etc.

[pedz]

To help clarify (based upon my experiments -- please verify) There seems to be three of these: ~/Documents, ~/Desktop, and ~/Downloads ... I'm surprised ~/Downloads is in there because its not copied to iCloud.

The ns-open-file-using-panel must be done once for each of these. After that, it appears to work forever and ever. I even rebooted and it still worked. Go figure...

Thank you @XeCycle

[XeCycle]

Add up to that reboot note: it started failing again after a macOS upgrade. Repeat the same steps and it's fixed again.

[josh-berry]

FWIW, the Internet (well, StackExchange) seems to think this issue is related to the launcher script being written in Ruby, yet "Full Disk Access" is granted only to Emacs (and not to Ruby):

https://emacs.stackexchange.com/questions/53026/how-to-restore-file-system-access-in-macos-catalina/53037#53037

I tend to agree. I did a quick test with my Emacs install where I moved the launcher script out of the way and symlinked the appropriate binary in its place, and everything worked fine. I did not need to do ns-open-file-using-panel to be able to see the contents of ~/Desktop and ~/Documents (for example). Specifically, I did this:

$ cd /Applications/Emacs.app/Contents/MacOS
$ mv Emacs Emacs.old
$ ln -s Emacs-x86_64-10_14 Emacs

Just to be safe (and to verify this is actually the problem), I did the following:

1. Revoked Full Disk Access and uninstalled Emacs (explicitly revoking is required because macOS will remember that FDA was granted even after a reinstall)
2. Reinstalled Emacs
3. Re-granted Full Disk Access
4. Confirmed I once again could not see my Desktop and Documents folders in dired. (Interestingly, I could still see my Downloads folder, which I had previously opened using the ns-open-file... trick.)
5. Repeated the mv/ln procedure above
6. Confirmed I see my Desktop and Documents folders again

So this definitely seems to be tied to the launcher script.

[jangid]

Are these changes in the repository now? I tried to build a fresh and the output is without the ruby script. And it is working great.

[jangid]

Oh! I took directly from Emacs.app and not the .dmg output from combine-and-package. Emacs.app worked fine.

[pedz]

Which kinda confirms that its the start up script. Just to let this side of the world know -- I updated to 10.15.1 and did not hit any problems. Go figure.

[slaanesh]

On a 10.15.1 fresh install I got the problem and had to give full disk access and do the mv / ln trick. Definitely the startup script yes

[a-bangk]

$ cd /Applications/Emacs.app/Contents/MacOS
$ mv Emacs Emacs.old
$ ln -s Emacs-x86_64-10_14 Emacs

I installed emacs for the first time, and I hit this issue within 15 minutes of experimenting. Can this be fixed from install?

[neppramod]

My issue was, I had this custom dired setting, had to comment it out

dired-listing-switches "-AlBGh --group-directories-first"

As mentioned here.

[dcguim]

For Emacs installations running on ARM-based processors (M1s), run the following commands for optimal performance on /Applications/Emacs.app/Contents/MacOS: mv Emacs Emacs.old && mv Emacs.pdmp Emacs.pdmp.old mv Emacs-arm64-11_2 Emacs && mv Emacs-arm64-11_2.pdmp Emacs.pdmp

[caldwell]

@dcguim What is optimal about that?

[dcguim]

Please correct if I am wrong @caldwell, but if the Emacs binary built for x86 processor is ran on an ARM-based processor it will be first converted into a universal binary through apple's Roseta, whereas the Emacs-arm-* binaries run natively on ARM-based processors. I believe arm processors can execute instructions, on average, faster for arm binary than for universal binaries, quoting the developer.apple website; (universal binaries) "... is not a substitute for creating a native version of your app." and "The system prefers to execute an app’s arm64 instructions on Apple silicon."

[caldwell]

The main Contents/MacOS/Emacs executable is not an x86 binary, it's actually just a Ruby script. It explicitly links to /usr/bin/ruby (ie, the system ruby), which is a fat binary on an M1:

$ file /usr/bin/ruby
/usr/bin/ruby: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/usr/bin/ruby (for architecture x86_64):    Mach-O 64-bit executable x86_64
/usr/bin/ruby (for architecture arm64e):    Mach-O 64-bit executable arm64e

The ruby launcher script figures out which OS+architecture the machine is and execs the correct binary. When I launch emacs I get this on the splash page:

This is GNU Emacs 27.2 (build 1, arm-apple-darwin20.3.0, NS appkit-2022.30 Version 11.2.3 (Build 20D91))
 of 2021-03-27

That tells me the correct native architecture is running. I can also verify it in Activity Monitor by searching for "Emacs" and adding the "Architecture" column (right clicking the table header). That shows me it being set to "Apple" which means it's running M1 native.

Renaming away the Ruby script will of course work, but it shouldn't change any outcomes (and if it does, then there's a bug that should be fixed).

[DanLipsitt]

Renaming away the Ruby script will of course work, but it shouldn't change any outcomes (and if it does, then there's a bug that should be fixed).

That appears to be the case. I followed @josh-berry's instructions above and now I can dired ~/Downloads when I couldn't before.

[caldwell]

Can someone who has this issue try the nightly build from 2022-04-05. I believe it can help in this case but I don't use iCloud for ~/Documents so I can't really test it.

[pedz]

Can someone who has this issue try the nightly build from 2022-04-05. I believe it can help in this case but I don't use iCloud for ~/Documents so I can't really test it.

On macOS 12.3.1 on a MBP with the M1 Max chip:

Opening ~/Documents works without any hiccups of oddities. Opening ~/Desktop the very first time pops up a dialog box (which I didn't capture) that the user can click Ok or something positive and it disappears and will not be seen again even after restarting Emacs. The same is true for ~/Downloads -- it pops up once.

I use helm which may affect things but I'd rather doubt it.