Feature request: force download of all files in an iCloud folder, including hidden files

[chchen2021]

Is it possible to force iCloud to download an entire folder, including hidden files? For example, can we include a command in ~/Documents/.profile so all files in a-Shell iCloud folder are automatically downloaded upon opening a-Shell?

When I use macOS and a-Shell to manage the same iCloud folder (e.g. with pdflatex or task), I often encounter the annoyance of iCloud files not yet downloaded locally (so the files have an .icloud suffix). So far the best solution is to select all files and hit copy, which will force the downloading of all visible files in a folder, but this solution has two problems:

1. I need to leave the CLI and use the Files app to select all files and hit copy. I have tried commands like open . but it does not open the current directory in Files. The same command would have opened the current directly in Finder.
2. There is no way to select hidden folders or hidden files in the Files app. I ended up renaming .task to task just so that I can select the files in the folder to force download. Such renaming works with a modification to .taskrc, but would not work in some scenarios (e.g. the .obsidian file within an Obsidian vault).

So I am basically asking for an iCloud download behavior similar to what I see in macOS, in which files are automatically downloaded to the local disk.

Notes added: Interestingly, iSH accomplishes the "automatic" download after executing the ls -a command at the folder of interest. It works for both visible and hidden files/folders. The same trick does not work in a-Shell.

[holzschu]

I see what you mean. I agree that it's really annoying. I'm a bit wary of doing things automatically (after ls -a, for example), as users might not always want to start a download (due to poor network connection, for example). But selecting a folder in pickFolder should definitely result in downloading all the files from that folder.

I'm adding code for that in a-Shell, with a time limit (if the network connection is bad, a-Shell will try downloading files for 5 seconds before giving up).

[chchen2021]

Thanks for being so receptive to the feature requests. I would like to offer some additional considerations:

• If a user does not want to automatically download a folder, they may not want it with pickFolder either. Even worse, if a bookmark is persistent, they may forget about the automatic download in future, and put a really big file in the bookmarked folder that ends up using wireless data.
• It's not ideal to use pickFolder to force download a-Shell's own iCloud folder, ~iCloud (~cloud). It hurts the workflow, and also creates a third bookmark, ~Documents, which can be very confusing.

My recommendation is to have an additional command like "downloadFolder". One can either use it in the command line, or include it in ~/Documents/.profile so an included folder is always downloaded.

I am not sure about the 5-second rule. The problem with non-yet-downloaded iCloud files is that the names of the files are occupied, and a new file will often be created (so "file 1" instead of "file"). I am not experienced in iCloud behavior to make any useful suggestion though.

[holzschu]

You're right. I will add the downloadFolder and downloadFile command. Still, if I pick a folder inside iCloud using pickFolder, I definitely expect the files inside to be available, so I'll also keep the recent addition (not just a-Shell's own iCloud folder, it can be somewhere else on iCloud).

I've had some bad experiments with iCloud over the summer, including the iCloud/Documents folder being unavailable for weeks, so I would advise you to keep local copies of anything important.

[chchen2021]

Thanks for the cautionary note. My ~/Library/Mobile Documents folder is backed up by Time Machine, and the backup has indeed saved me from losing an entire iCloud folder. As long as there is sufficient hard drive space, I think all iCloud files are automatically downloaded by macOS, so Time Machine is a good choice for backup.

[chchen2021]

There is actually a URL scheme to open an iCloud folder from a-Shell, as explained in this blog. For example, the following command will open a-Shell's iCloud folder from the command line: open shareddocuments:///private/var/mobile/Library/Mobile%20Documents/iCloud~AsheKube~app~a-Shell/Documents/

This URL scheme does not directly solve the download problem, but accomplishes the counterpart of open path/to/folder in macOS, which I deemed not possible in the original post.

[holzschu]

Hi, the new commands are now available in TestFlight: https://testflight.apple.com/join/WUdKe3f4

[chchen2021]

Just installed TestFlight on my iPad (not yet on iPhone). I can confirm that pickFolder, downloadFile and downloadFolder are all forcing iCloud downloads.

A minor glitch: it's not obvious how to get help. The help for syntax is revealed by downloadFile and downloadFolder --help, respectively. A third behavior is observed on pickFolder, which ignores the --help option. The inconsistency may be an issue for other a-Shell commands.

@holzschu: TeX (but not LuaTeX) needs to be reenabled on the beta version, so I tested pdflatex again. I must say that the downloadFolder function is a huge improvement. I now appreciate why automatic download needs to be enabled by pickFolder - many duplicate auxiliary TeX files were created just because they were not downloaded locally. Thanks for taking care of the iCloud download issue!

[chchen2021]

I now have downloadFolder ~cloud in ~/Documents/.profile. Almost all my a-Shell work is done in this folder so it makes sense to have this folder completely downloaded.

This setup revealed a limitation, perhaps imposed by iOS. The downloadFolder command will not force download recursively. If I have a subfolder in a-Shell's iCloud folder, its content is only downloaded if I include the subfolder in the profile. iOS has a similar limitation when I use the copy trick to force download (see my initial comment above),

Is there anything I can do to make the entire a-Shell iCloud folder always downloaded to the local disk? Or, perhaps a recursive download can be programmed in a-Shell?

[holzschu]

Yes, that limitation is on purpose, as you don't know what kind of files might be hidden further down the hierarchy.

Someone who really needs a recursive download of all sub-folders could do it with find, e.g.: find . -type d -exec downloadFolder {} \; -print, or find . -type f -name .*.icloud -exec downloadFile {} \; -print.

[chchen2021]

Thanks so much for the additional commands. Exactly what I wanted.

[chchen2021]

Someone who really needs a recursive download of all sub-folders could do it with find, e.g.: find . -type d -exec downloadFolder {} \; -print, or find . -type f -name .*.icloud -exec downloadFile {} \; -print.

Sometime after the iOS 15 update, my a-Shell beta went awry. All commands (such as ls) hung indefinitely after I hit enter, with the exception of exit if it was entered as a fresh command. I reverted to 1.7.5 and things were working, but the update to 1.8.0 brought back the hanging behavior again. A fresh a-Shell reinstallation did not help on either iPad or iPhone.

The culprit is the automatic download command you kindly provided above: find ~cloud -type f -name .*.icloud -exec downloadFile {} \; I commented it out in ~/Documents/.profile and everything is working normally. Not sure why this command does not work in a-Shell versions beyond 1.7.5.

[holzschu]

That seems to be a bug in "find -exec", which definitely should not be there. I'm investigating.

[chchen2021]

The following information may be useful. I remember the hanging problem happened after iOS 15 upgrade, not after the a-Shell beta incorporating texlive 2021. Since a couple of other apps had problems with the enter key in iOS 15, I thought the issue would go away with iOS updates, but it obviously did not. Around that time, a-Shell tried to address some iOS keyboard issues with a couple of beta releases.

[holzschu]

No, I can confirm it's an a-Shell issue, related to more security around commands launched by other commands (to avoid having one command changing the environment and directory of another). It will be fixed in the next build.

[holzschu]

Hi, this should be fixed with build 203 currently on TestFlight ( https://testflight.apple.com/join/WUdKe3f4 ). If you can confirm that it works for you, I will release it for everyone.

[chchen2021]

I can confirm the find -exec bug is gone. Thanks for fixing it.

Also, the original latexmk works, but somehow it is no longer included in the last few builds. I just copied on in my bin folder and it works fine.

[holzschu]

latexmk should also be in the new TestFlight, but you might need to uninstall/reinstall TeX packages to make it appear.

[chchen2021]

latexmk is indeed included.

It was an easy reinstallation on the iPad through Settings, but on the iPhone, I ended up removing the entire a-Shell to avoid repeated crashes. Everything is working fine now.