search

nextcloud / desktop

💻 Desktop sync client for Nextcloud
https://nextcloud.com/install/#install-clients
GNU General Public License v2.0
2.99k stars 788 forks source link

Support FileProvider API (VFS) in macOS #1337

Closed helmut72 closed 10 months ago

helmut72 commented 5 years ago

https://www.apple.com/macos/catalina-preview/features/

Third-party cloud service integration An all-new FileProvider API for cloud storage providers delivers a new way to seamlessly integrate their services into the Finder without requiring a kernel extension, helping to maintain the security of your Mac. Cloud storage providers can now deliver their apps through the Mac App Store.

marcotrevisan commented 1 year ago

Great news! @claucambra I've just installed it and I'm trying to make it sync with my existing Nextcloud server. Do we need to enable some flag in Library/Preferences/Nextcloud macOS VFS Beta/nextcloud-macos-vfs-beta.cfg? If I enable the experimental options, it seems to enable the old sync, not the FIleProvider based one.

Sorry if I'm asking a silly question.

Thanks again!

claucambra commented 1 year ago

Great news! @claucambra I've just installed it and I'm trying to make it sync with my existing Nextcloud server. Do we need to enable some flag in Library/Preferences/Nextcloud macOS VFS Beta/nextcloud-macos-vfs-beta.cfg? If I enable the experimental options, it seems to enable the old sync, not the FIleProvider based one.

Sorry if I'm asking a silly question.

Thanks again!

No hidden config settings need to be enabled, the file provider module should be enabled automatically when your account is configured. Is the Nextcloud extension enabled in the system settings?

If it is still not showing up, there is this point in the big post:

If the Nextcloud macOS VFS beta does not appear in the finder sidebar after setting up your account in the client, restart the desktop client as well as finder, and then it should show up

marcotrevisan commented 1 year ago

You were right, sorry for useless question! I restarted my mac before reading your reply, so I guess it would have worked by only restarting Finder. What fooled me is that I saw the normal folder in my home folder and the sync progress was working just like before, so I thought I was missing something in the configuration. Thanks again and congratulations!

marcotrevisan commented 1 year ago

My first feedback. Before testing I uninstalled the Release NC Client app from the systems under test, just in case.

On a Big Sur 11.7 - iMac - Intel 2014 it's currently working like a charm. I mean: I've set up the account info; specifying that I'm not syncing any subfolder under the root so the legacy sync won't do anything in my home subfolder; closed the account setup and the File Provider share appears under my "Locations" in Finder. I can immediately start browsing there and do whatever I want, it just works! It's very responsive. I have to check if I can configure Finder to [download and] open the preview on space-bar pressed, like it does for files in local folders, but that's another matter - not sure if it's possibile...

On my mac book air (M1, 2020) running MacOS Monterey 12.6.3, after a few clicks opening the root subfolders and some 2nd-level folders, the share won't open subfolders anymore, gets stuck and won't recover (even after reboots). That initial "grace period" has lasted about 30 seconds or so. It may be related to your "First sync will take a while" point, but under BigSur things are going quite better. My colleague did the same under MacOS Monterey 12.4 and same hardware, had the same behaviour. After an initial all-good behaviour, something happens and FInder gets stuck on that share.

On server side, I can see PROPFIND requests arriving in what seems to be a recursive "tree visit", and the user agent is FileProviderExt as I would expect. This holds for both BigSur and Monterey clients. I guess this is how Finder works.

Also, I further aggravated my situation by adding another account (to my home NC 25.0.4 server): Finder got completely stuck, fileproviderctl listproviders was also stuck. After waiting for a good 30 minutes I had to do a force shutdown.

I'll let you know more. I'm also gonna try the client on Ventura hosts too very soon!

marcotrevisan commented 1 year ago

Sorry I didn't point out that the share is quite populated (>250K files), I guess this is a factor.

claucambra commented 1 year ago

My first feedback. Before testing I uninstalled the Release NC Client app from the systems under test, just in case.

On a Big Sur 11.7 - iMac - Intel 2014 it's currently working like a charm. I mean: I've set up the account info; specifying that I'm not syncing any subfolder under the root so the legacy sync won't do anything in my home subfolder; closed the account setup and the File Provider share appears under my "Locations" in Finder. I can immediately start browsing there and do whatever I want, it just works! It's very responsive. I have to check if I can configure Finder to [download and] open the preview on space-bar pressed, like it does for files in local folders, but that's another matter - not sure if it's possibile...

Nice!

On my mac book air (M1, 2020) running MacOS Monterey 12.6.3, after a few clicks opening the root subfolders and some 2nd-level folders, the share won't open subfolders anymore, gets stuck and won't recover (even after reboots). That initial "grace period" has lasted about 30 seconds or so. It may be related to your "First sync will take a while" point, but under BigSur things are going quite better. My colleague did the same under MacOS Monterey 12.4 and same hardware, had the same behaviour. After an initial all-good behaviour, something happens and FInder gets stuck on that share.

The recursive scan begins on first remote change detection, and I wasn't kidding when I said it takes a while -- on my personal instance it takes ~15 minutes, but I have no shares on there. Depending on your network connection and the number of files/folders, it can take a lot longer, and the File Provider extension will not do much until the scan is complete

On server side, I can see PROPFIND requests arriving in what seems to be a recursive "tree visit", and the user agent is FileProviderExt as I would expect. This holds for both BigSur and Monterey clients. I guess this is how Finder works.

Yup, we are following Apple's advice here.

Also, I further aggravated my situation by adding another account (to my home NC 25.0.4 server): Finder got completely stuck, fileproviderctl listproviders was also stuck. After waiting for a good 30 minutes I had to do a force shutdown.

That's weird; fileproviderctl should be independent from the extension, and each file provider account should work independently of each other...

I'll let you know more. I'm also gonna try the client on Ventura hosts too very soon!

Cool!

marcotrevisan commented 1 year ago

After reboot of my MacBookAir I was able to watch my home NC share first populating, then getting stuck for some 15 minutes, finally becoming responsive. On the work share, I'm afraid this will require many hours as changes happen quite frequently there.

Two main questions arise in my mind: why other FileProvider implementations (i.e. the most recent versions of G Drive and Dropbox) aren't seemingly suffering this issue, and secondly, why under BugSur it just works perfectly from the start? There seems to be something more in the newer OS versions... can it be a changed defaults in the configuration of the FileProvider framework?

I'm afraid it will be a problem in case of big shares, if a single file change will trigger recursive scans from root with a corresponding blocking of the Finder location (I hope that won't be the case).

About Ventura: just tried on MacBookAir - M1 2020, and it behaved like the Monterey hosts. All wi-fi connected hosts are in internal wireless LAN (5GHz AC). Bandwidth is not usually a problem as we easily reach over 100 Mbps, latency well under 10ms.

I'll keep testing and reporting my findings. I can try to save filtered logs and send you them if you like. Cheers,

marcotrevisan commented 1 year ago

From a end-user perception it's like in BigSur there're more threads doing their job independently (one of them is serving Finder's UI interactions), while in Monterey and Ventura they seem to be synchronized on something, the result being some kind of bottleneck...

marcotrevisan commented 1 year ago

I just went back to the BigSur host. While navigating trough the share is very responsive, I tried to create a second level folder and the result was the parent folder becoming inaccessible for permission reasons (folder icon with the "red spot"). I guess this is due to the background scanning in progress (so perhaps Monterey and Ventura are keeping stuff frozen to avoid the user getting unwanted results in write operations). However, if I move to another share in Finder and then go back to the NC location, the parent folder becomes accessible again and I can find the newly created folder inside, with the default "unnamed" name (localized in my language). I can then rename it: the parent folder becomes inaccessible, same trick to get back, and it shows up renamed. The renaming is visible via web in NC Files.

marcotrevisan commented 1 year ago

In the end of today I'm having:

Maybe the pace of the PROPFIND's can be adjusted. If the whole process is waiting for them to finish, I'll have to wait for a very long time.

Sorry for my many posts. Thanks for your patience :-D

marcotrevisan commented 1 year ago

Update. The big work share is now operational on the macbook air M1 with Monterey 12.6.3. Seems to work pretty smoothly. I tested:

Tomorrow I'll be able to use it under server load and see how it will behave in that scenario. I'll keep you updated.

Kyshman commented 1 year ago

@marcotrevisan am on Monterey on an Intel Mac. Didn't have any trouble setting up and having VFS appear on the sidebar. It's also quite responsive after the initial scan. But am wondering whether you noticed something though -- All the files/directories have a modification date of 1st Jan 1970 at 03:00

image

This is not reflected on the server though where they all maintain their correct modification dates.

Even newly created files have this date too. image

marcotrevisan commented 1 year ago

@Kyshman yes the modification times are shown as 1970-01-01. If I remember correctly there is no direct retriveal of the modification time, so the current implementation may behave differently.

My problems arised on a quite populated and frequently updated share. On a smaller one I hadn't got any major issues so far.

marcotrevisan commented 1 year ago

@claucambra I went back to the office and my FileProvider client must be resyncing now, probably due to my colleagues' updates to files.

In this situation my findings are:

  1. I see a stream of PROPFIND requests arriving from my client, I guess it's "catching up" with modified files
  2. If I create files / folders using the browser under my root or in another subfolder, they are NOT showing up in Finder. Seems like it's not allowing for parallel local updates due to remote changes. The problem is, Finder is not showing any updates to the local folders I'm working on for more than an hour (and counting).
  3. If on the other hand I create a folder under the root (or elsewhere), it puts the folder and I can see it remotely.
  4. If I then copy any file under that folder, Finder shows the "cloud with exclamation mark" on that. The server issues a 404 and the pattern is like: "PUT /remote.php/dav/files/user/new-folder/new-folder/filename HTTP/1.1" 404 265 "-" "Nextcloud-macOS/FileProviderExt", where you can see "new-folder" appearing twice, hence the 404.
  5. On previously synced folders, the file gets uploaded correctly instead.

The above happens on my Monterey 12.6.3 MacbookAir (using my work NC account).

If more arises I'll share it.

marcotrevisan commented 1 year ago

Update to the above after the sync settled (took a couple hours):

  1. PROPFINDS calmed down and were limited to the notify_push response pattern;
  2. Creating/deleting stuff using the Files app on the browser is now reflected in Finder as expected;
  3. (no change);
  4. It works now. Also, the files formerly having the "exclamation mark cloud" icon have been uploaded and now are in sync;
  5. ( no change).

I think we need to find a way to get rid of those long-running syncs, or, to allow for some degree of concurrent local changes. Would it be possible to (if this at all makes any sense to you):

Thank you very much!

claucambra commented 1 year ago

Update to the above after the sync settled (took a couple hours):

  1. PROPFINDS calmed down and were limited to the notify_push response pattern;
  2. Creating/deleting stuff using the Files on the browser now is reflected in Finder as expected;
  3. (no change);
  4. It works now. Also, the files formerly having the "exclamation mark cloud" icon have been uploaded and now are in sync; 5.( no change).

I think we need to find a way to get rid of those long-running syncs, or, to allow for some degree of concurrent local changes. Would it be possible to (if this at all makes any sense to you):

  • restrict the lookups to the folders the user has actually browsed, and:

I implemented this initially and caused some of issues, especially when it came to A) moving unsynced/unexplored folders, and B) it stops us from being able to detect moves from one explored folder to another, unexplored folder. So you could easily run into a situation where you'd move a big, locally materialised file from folderA to unexploredFolderB, it'd report the file as deleted, and you'd be forced to re-download it. So, not ideal

Apple's documentation also says we should report the whole server.

  • have a (maybe configurable) time to live after which folders are "invalidated" (and thus their contents forgotten)?

Something like this (particularly manual unmaterialisation of materialised files) is on the to-do list.

Thank you very much!

Glad to hear no major blow ups have happened yet. But please be careful when running this on your instances, particularly when it comes to concurrent modification of files -- this is still a bit unpredictable

marcotrevisan commented 1 year ago

I think we need to find a way to get rid of those long-running syncs, or, to allow for some degree of concurrent local changes. Would it be possible to (if this at all makes any sense to you):

  • restrict the lookups to the folders the user has actually browsed, and:

I implemented this initially and caused some of issues, especially when it came to A) moving unsynced/unexplored folders, and B) it stops us from being able to detect moves from one explored folder to another, unexplored folder. So you could easily run into a situation where you'd move a big, locally materialised file from folderA to unexploredFolderB, it'd report the file as deleted, and you'd be forced to re-download it. So, not ideal

Apple's documentation also says we should report the whole server.

Case B) can be scary sometimes, but I think it would normally pass unnoticed, as teams usually work on the same sets of folders. I'm not sure how many additional cases are actually hiding behind case A)... But if Apple recommends to report the whole server I can well understand your decision.

Lookups apart, what about allowing for more concurrency on local updates? (like allowing two "worker queues", if this makes any sense for you..)? The FileProviderExt is not flooding the server with PROPFINDs even when it's resyncing, so that's not a problem. The current problem in my humble opinion is that before that resync has finished, Finder is not behaving consistently.

I would prefer the shortcomings of the above mentioned cases A) and B) over long periods of inconistent behaviour on the folders I usually work on...

Thanks again and regards!

claucambra commented 1 year ago

Lookups apart, what about allowing for more concurrency on local updates? (like allowing two "worker queues", if this makes any sense for you..)? The FileProviderExt is not flooding the server with PROPFINDs even when it's resyncing, so that's not a problem. The current problem in my humble opinion is that before that resync has finished, Finder is not behaving consistently.

We process received data concurrently with a specific concurrent dispatch queue, hence why once we have the data from the server the processing is very quick. With network requests we send them one after the other so as to not overload the server; we could send more at once, within limits, but this will not necessarily yield much of a speed up. Especially if several clients are connected at once. This can be explored more.

With regards to Finder being unresponsive, not much we can do about this at the moment. Experimenting with the extension it seems that Finder will always wait for the current file exploration to be finished before proceeding.

I would prefer the shortcomings of the above mentioned cases A) and B) over long periods of inconistent behaviour on the folders I usually work on...

This can always be a setting in the future, with the setting favouring correctness the default.

My recommendation for quicker change detection is to favour deep folder hierarchies over wide folder hierarchies. We do a recursive propfind whenever a folder's ETAG changes -- the etag of a folder changes whenever any of its contents change, or whenever any of its children's content changes. So with deep hierarchies we can do quicker scans of smaller folders and focus only on those that have changed. Obviously it's a lot quicker to scan a folder with 30 children than one with 3000 children.

Hope this helps

marcotrevisan commented 1 year ago

We process received data concurrently with a specific concurrent dispatch queue, hence why once we have the data from the server the processing is very quick. With network requests we send them one after the other so as to not overload the server; we could send more at once, within limits, but this will not necessarily yield much of a speed up. Especially if several clients are connected at once. This can be explored more.

I think we have a few cases worth looking at, i.e. the "create new folder and put a file in that new folder" issue leading to 404's - if we fix it, that may fix more stuff I wasn't able to spot yet. By the way, I also tested updating some unmaterialized files during the sync process, and the materialization works. I was able to edit a .docx file with no issues and saw the "put" in the server upon save.

I would prefer the shortcomings of the above mentioned cases A) and B) over long periods of inconistent behaviour on the folders I usually work on...

This can always be a setting in the future, with the setting favouring correctness the default.

I'd really like to have that setting available, I'd use it eagerly. After all, G Drive and Dbox in their FileProvider clients must be doing some similar tricks too.. we have another > 2TB data in there and I can assure you no deep sync is performed, or we would be dead...

My recommendation for quicker change detection is to favour deep folder hierarchies over wide folder hierarchies. We do a recursive propfind whenever a folder's ETAG changes -- the etag of a folder changes whenever any of its contents change, or whenever any of its children's content changes. So with deep hierarchies we can do quicker scans of smaller folders and focus only on those that have changed. Obviously it's a lot quicker to scan a folder with 30 children than one with 3000 children.

Looks like we already are in a "vertically deep" scenario. Ususally folders don't contain more than 30 items, but they often contain more than 10 subfolders. I guess we're one of the worst cases out there, lol

Hope this helps

Absolutely! Thank you very much for your work! I hope our feedback is helpful too.

marcotrevisan commented 1 year ago

I was trying to make use of fileproviderctl and I'm monitoring the situation using the ls command now. I noticed something weird when I deleted a test folder (namely "test_marco_2") that I created under the root. In Finder it worked apparently without problems, and also remotely it's been removed, everything looks good. The fileproviderctl ls output however is showing some apparently spurious stuff that might interest you @claucambra , here is a snippet:

[...] 19: 📁 test_marco_2 id:__fp/fs/fileID(66120888) size:0 ⬆ (indeterminate) (fp error) 🖥 (no url) oggi, 12:34 (used) 🏁 collection 🥑 gathering completed in 0.028s

collection 🥑 : received updates:

♻️ 19: 📁 test_marco_2 id:__fp/fs/fileID(66120888) size:0 ⬆ (indeterminate) (fp error) 🖥 (no url) oggi, 12:34 (used)

collection 🥑 : received updates:

♻️ 19: 📁 test_marco_2 id:__fp/fs/fileID(66120888) size:0 🖥 (no url) oggi, 12:34 (used)

collection 🥑 : received updates:

♻️ 19: 📁 test_marco_2 id:__fp/fs/fileID(66120888) size:0 ⬆ (indeterminate) (fp error) 🖥 (no url) oggi, 12:34 (used)

collection 🥑 : received updates:

♻️ 19: 📁 test_marco_2 id:__fp/fs/fileID(66120888) size:0 🖥 (no url) oggi, 12:34 (used)

collection 🥑 : received updates:

♻️ 19: 📁 test_marco_2 id:__fp/fs/fileID(66120888) size:0 ⬆ (indeterminate) (fp error) 🖥 (no url) oggi, 12:34 (used)

collection 🥑 : received updates:

♻️ 19: 📁 test_marco_2 id:__fp/fs/fileID(66120888) size:0 🖥 (no url) oggi, 12:34 (used)

collection 🥑 : received updates:

♻️ 19: 📁 test_marco_2 id:__fp/fs/fileID(66120888) size:0 ⬆ (indeterminate) (fp error) 🖥 (no url) oggi, 12:34 (used)

collection 🥑 : received updates:

♻️ 19: 📁 test_marco_2 id:__fp/fs/fileID(66120888) size:0 🖥 (no url) oggi, 12:34 (used)

collection 🥑 : received updates:

♻️ 19: 📁 test_marco_2 id:__fp/fs/fileID(66120888) size:0 ⬆ (indeterminate) (fp error) 🖥 (no url) oggi, 12:34 (used)

collection 🥑 : received updates:

♻️ 19: 📁 test_marco_2 id:__fp/fs/fileID(66120888) size:0 🖥 (no url) oggi, 12:34 (used)

collection 🥑 : received updates:

♻️ 19: 📁 test_marco_2 id:__fp/fs/fileID(66120888) size:0 ⬆ (indeterminate) (fp error) 🖥 (no url) oggi, 12:34 (used)

strozzascotte commented 1 year ago

Hi @claucambra, I was waiting for Virtual Files on Mac and I'm excited to trying it. NC26 still doesn't appear on Stable update channel on my server. Does the Beta VFS Mac Client (v. 3.7.50) new sync engine work with NC25.0.5 too?

marcotrevisan commented 1 year ago

@strozzascotte I'm testing it against 25.0.4/5 servers (with notify_push app enabled). Keep in mind it's a beta so you have to accept a higher risk using it compared to a release version.

claucambra commented 1 year ago

Hi @claucambra, I was waiting for Virtual Files on Mac and I'm excited to trying it.

NC26 still doesn't appear on Stable update channel on my server. Does the Beta VFS Mac Client (v. 3.7.50) new sync engine work with NC25.0.5 too?

I've been testing on both 25 and 23, so yeah. Best experience will be with push notifications enabled, so that will limit how low your NC version is. But the file provider module can also manually poll the server for changes if push notifications are not available.

TL;DR - unless you are running something positively ancient it'll work, and the module has been actively tested on NC23+

marcotrevisan commented 1 year ago

Just for the records, some features that I noticed to be missing in Finder:

About Big Sur: the mount is showing the following issues on write operations:

So Mac OS 11 Big Sur perhaps needs specific tests. I'll try on another Big Sur host to see if it shares that same behaviour.

OS 13 Ventura: on one machine (MBA 2020 M1) it seems to work exactly the same like mine under OS 12, on another (MBA 2020 M1) it seems stuck (I mean it doesn't even seem to contact the server). We'll try to delete and recreate the mount. Is there a best practice to do that? Looks like I'm unable to delete a FileProvider mount...

Thanks!

claucambra commented 1 year ago

OS 13 Ventura: on one machine (MBA 2020 M1) it seems to work exactly the same like mine under OS 12, on another (MBA 2020 M1) it seems stuck (I mean it doesn't even seem to contact the server). We'll try to delete and recreate the mount. Is there a best practice to do that? Looks like I'm unable to delete a FileProvider mount...

Open terminal and do fileproviderctl domain remove <name of domain>

Each of the entries in the sidebar is a domain. You can see which domains are available by running fileproviderctl domain list

marcotrevisan commented 1 year ago

Big Sur logs, the issue on the firsst level folder becoming unaccessible upon file materialization. Hoping they can be of help:

debug   12:22:51.620259+0100    fileproviderd   [DEBUG] ┏4e9a Item at <private> losing presenter <private>
debug   12:22:51.620357+0100    fileproviderd   [DEBUG] ┗4e9a
debug   12:22:51.621203+0100    FileProviderExt [DEBUG] Attempting to restore persona in -[FPXEnumerator _invalidate]: (null)
debug   12:22:51.621316+0100    FileProviderExt (504) dropping voucher due to nil personaUniqueString
debug   12:22:51.621397+0100    FileProviderExt [DEBUG] ┏2e7 <private>: invalidating vendor enumeration: <private>
debug   12:22:51.621471+0100    FileProviderExt Enumerator is being invalidated for item with identifier: 0074594472f651ffb41de
debug   12:22:51.620773+0100    fileproviderd   [DEBUG] ┣4e9d dispatching to <private>
debug   12:22:51.621562+0100    FileProviderExt [DEBUG] ┗2e7
debug   12:22:51.621632+0100    FileProviderExt [DEBUG] Attempting to restore persona in -[FPXEnumerator _invalidate]: (null)
debug   12:22:51.621071+0100    fileproviderd   [DEBUG] ┳4e9d continuing on <private>
debug   12:22:51.621716+0100    FileProviderExt (504) dropping voucher due to nil personaUniqueString
debug   12:22:51.621196+0100    fileproviderd   [DEBUG] ┣4e9f dispatching to <private>
debug   12:22:51.621830+0100    FileProviderExt [DEBUG] Attempting to restore persona in -[FPXEnumerator _invalidate]: (null)
debug   12:22:51.621336+0100    fileproviderd   [DEBUG] ┗4e9d
debug   12:22:51.622037+0100    FileProviderExt (504) dropping voucher due to nil personaUniqueString
debug   12:22:51.621611+0100    fileproviderd   [DEBUG] ┳4e9f continuing on <private>
debug   12:22:51.622477+0100    FileProviderExt [DEBUG] Attempting to restore persona in -[FPXEnumerator dealloc]: (null)
debug   12:22:51.621794+0100    fileproviderd   [DEBUG] ┗4e9f
debug   12:22:51.622659+0100    FileProviderExt (504) dropping voucher due to nil personaUniqueString
debug   12:22:51.622884+0100    FileProviderExt [DEBUG] <private>: dealloc
TomGem commented 1 year ago

First, thanks a lot! It's great to to see Nextcloud virtual files for macOS coming! 🙏

The beta setup on a Ventura Silicon Mac was fine. Although at first I didn't get I have to check the location 'Nextcloud macOS VFS Beta', not the folder I configured during the setup.. 8)

It took a while to initialise but now the VFS response times are quite ok. Btw, the server is NC25 with notify_push enabled.

My first question (it won't be the last :), the modification times are set to SkyNet' Big Bang. I've read about this above. However, is it a known issue or a misconfiguration on my side?

Screenshot 2023-03-26 at 01 30 19
strozzascotte commented 1 year ago

Thank you very much for this great feature. I've noticed that if you configure an account on the desktop client, delete it, and configure another account from the same Nextcloud server, the label of the virtual drive doesn't get updated right away. In the attached image you can see that on the left panel of Finder the selected item is Nextcloud macOS VFS Beta - admin@... when in fact it is connected to another account as correctly reported in the status bar of the right panel. Restarting Finder resolve the issue. It's definitely just a minor issue, but I hope it's worth reporting it.

Captura de pantalla 2023-03-26 a las 16 19 00
marcotrevisan commented 1 year ago

About my above comment: https://github.com/nextcloud/desktop/issues/1337#issuecomment-1482743744

I'm seeing this still happening. The folder is among the enumerated ones and the file provider is still trying to sync on that.

claucambra commented 1 year ago

Thank you very much for this great feature. I've noticed that if you configure an account on the desktop client, delete it, and configure another account from the same Nextcloud server, the label of the virtual drive doesn't get updated right away. In the attached image you can see that on the left panel of Finder the selected item is Nextcloud macOS VFS Beta - admin@... when in fact it is connected to another account as correctly reported in the status bar of the right panel. Restarting Finder resolve the issue. It's definitely just a minor issue, but I hope it's worth reporting it.

Unfortunately not much we can do about this as this is handled entirely by Finder

claucambra commented 1 year ago

Big Sur logs, the issue on the firsst level folder becoming unaccessible upon file materialization. Hoping they can be of help:

debug 12:22:51.620259+0100    fileproviderd   [DEBUG] ┏4e9a Item at <private> losing presenter <private>
debug 12:22:51.620357+0100    fileproviderd   [DEBUG] ┗4e9a
debug 12:22:51.621203+0100    FileProviderExt [DEBUG] Attempting to restore persona in -[FPXEnumerator _invalidate]: (null)
debug 12:22:51.621316+0100    FileProviderExt (504) dropping voucher due to nil personaUniqueString
debug 12:22:51.621397+0100    FileProviderExt [DEBUG] ┏2e7 <private>: invalidating vendor enumeration: <private>
debug 12:22:51.621471+0100    FileProviderExt Enumerator is being invalidated for item with identifier: 0074594472f651ffb41de
debug 12:22:51.620773+0100    fileproviderd   [DEBUG] ┣4e9d dispatching to <private>
debug 12:22:51.621562+0100    FileProviderExt [DEBUG] ┗2e7
debug 12:22:51.621632+0100    FileProviderExt [DEBUG] Attempting to restore persona in -[FPXEnumerator _invalidate]: (null)
debug 12:22:51.621071+0100    fileproviderd   [DEBUG] ┳4e9d continuing on <private>
debug 12:22:51.621716+0100    FileProviderExt (504) dropping voucher due to nil personaUniqueString
debug 12:22:51.621196+0100    fileproviderd   [DEBUG] ┣4e9f dispatching to <private>
debug 12:22:51.621830+0100    FileProviderExt [DEBUG] Attempting to restore persona in -[FPXEnumerator _invalidate]: (null)
debug 12:22:51.621336+0100    fileproviderd   [DEBUG] ┗4e9d
debug 12:22:51.622037+0100    FileProviderExt (504) dropping voucher due to nil personaUniqueString
debug 12:22:51.621611+0100    fileproviderd   [DEBUG] ┳4e9f continuing on <private>
debug 12:22:51.622477+0100    FileProviderExt [DEBUG] Attempting to restore persona in -[FPXEnumerator dealloc]: (null)
debug 12:22:51.621794+0100    fileproviderd   [DEBUG] ┗4e9f
debug 12:22:51.622659+0100    FileProviderExt (504) dropping voucher due to nil personaUniqueString
debug 12:22:51.622884+0100    FileProviderExt [DEBUG] <private>: dealloc

Unfortunately this is of no use, I'd need the logs retrieved as per the release post as these contain logs about what the file provider extension is doing

claucambra commented 1 year ago

First, thanks a lot! It's great to to see Nextcloud virtual files for macOS coming! 🙏

❤️

The beta setup on a Ventura Silicon Mac was fine. Although at first I didn't get I have to check the location 'Nextcloud macOS VFS Beta', not the folder I configured during the setup.. 8)

Folder configuration in the setup wizard is only for the traditional sync method, not for File Provider. This is still a WIP. 🙂

It took a while to initialise but now the VFS response times are quite ok. Btw, the server is NC25 with notify_push enabled.

My first question (it won't be the last :), the modification times are set to SkyNet' Big Bang. I've read about this above. However, is it a known issue or a misconfiguration on my side?

Known issue, I can replicate on all my files too. Looking into a fix now.

claucambra commented 1 year ago

Going back to answer some more of these, @marcotrevisan

Just for the records, some features that I noticed to be missing in Finder:

  • Thumbnails on the icons of unmaterialized files, I see in iCloud they're available (they also show up at space-bar-pressed, not great resolution but enough to recognize the file contents), I guess the API allows for that;

Yup, this is not implemented yet -- on the todo-list

  • When I right-click on a folder containing unmaterialized stuff, the info panel is not showing the total size (a "--" is shown instead).

We generally are not properly handling full folder sizing yet -- also on the todo-list ;)

About Big Sur: the mount is showing the following issues on write operations:

  • Wen a file / folder is created / updated / deleted, the first level folder containing it will suddenly close and show up as "access denied" (i.e. the folder icon with the red symbol on it). After a second or so, it will become accessible again.

Hmm, strikes me as incorrect error handling in the code, will look into it

  • If you were creating a subfolder, you'll find it with the default "unnamed folder" name, and when you rename it, the same thing will happen (the rename will be successful though).

AFAIK macOS by default immediately creates a folder with this name -- any name you type is just renaming the already created folder

Thanks!

Thanks for your thorough testing and feedback! :)

melMass commented 1 year ago

Is there a PR to follow progress or something like that? I've been searching for a while now with answers from 2018 to an hour ago. Do we still need showExperimentalOptions=true ? Are there real issues to be aware of?

Or is it isVfsEnabled? 😅

image

Thanks

marcotrevisan commented 1 year ago

Is there a PR to follow progress or something like that?

Up until now: https://github.com/nextcloud/desktop/pull/5527

Cheers

marcotrevisan commented 1 year ago

I found another glitch:

So basically looks as if it missed a "trigger" to show up a newly shared folder...

Hope this is reproducible on your systems too. Notify_push is enabled in the NC server instance under test.

sakura-ikeda commented 1 year ago

Please excuse my poor English. Thanks for the great development.

In the case of LDAP/AD, the URL of Webdav becomes UUID from the user name. https://nextcloud.com/remote.php/dav/files/XXXXXXXX-XXXX-XXXXX-XXXX-XXXXXXXXXXXX An error occurred in the Finder in this case. image

Console app logs The domain name and username have changed.

marcotrevisan commented 1 year ago

I found another glitch:

* Added my user to a group;

* The new group membership adds another folder to the visible ones (that folder was shared by another user to that group, read/write enabled);

* The newly discovered folder shows up in FIles (web app) as expected;

* The FileProvider started issuing PROPFINDs on the whole new folder hierarchy;

* After the full tree visit was finished (PROPFINDs finished) the folder didn't show up in Finder (and `fileproviderctl ls` didn't log that folder addition).

* Only when I made a change via web browser in that folder (deleted an empty sub folder), `fileproviderctl ls` finally logged the incoming change, and Finder finally showed up the new 1st level folder.

So basically looks as if it missed a "trigger" to show up a newly shared folder...

Hope this is reproducible on your systems too. Notify_push is enabled in the NC server instance under test.

Sorry I forgot to mention: my macbook was rebooted in the middle of the tree visit process. The process restarted after the reboot (PROPFINDs restarted from where they were interrupted).

Cheers!

d235j commented 1 year ago

Seems to be working well for me, once I set up notify_push and fixed its configuration. Some better error reporting for when notify_push is not set up correctly would be useful.

Separately — can the ability to evict a file/folder be added? At the moment this is not present, which will cause local storage to fill up. Using fileproviderctl to attempt to evict seems to also fail with an error. IMO this is probably the most important feature to have after basic functionality.

marcotrevisan commented 1 year ago

Hi all. @claucambra I found a small file naming glitch and was able to reproduce it. Folders containing "._" in their name get their name truncated once synced (unless there are also spaces in the name, in which case the folder will show up correctly). For example, "5._TEST" will show up in Finder as "5".

To reproduce it I created the following folders:

What happened at first is:

But after a while, those folders in Finder were showing up as follows:

Fortunately, their contents are OK. Updates to their content works as if nothing strange has happened.

on the NC server filesystem side they show up as follows: 

drwxr-xr-x 2 www-data www-data 4096 Apr 12 11:17  5._TEST_NO_SPC
drwxr-xr-x 2 www-data www-data 4096 Apr 12 11:17 '6._TEST W SPC'
drwxr-xr-x 2 www-data www-data 4096 Apr 12 11:18  7._TEST_NO_SPC2

and here is the difference: the one with spaces is deliminted by ' characters which seems to stop the harmful name processing. Hoping this helps. Best regards!

marcotrevisan commented 1 year ago

More info on the above.

  1. The same happens if I "touch" a regular file.
  2. In a terminal window, 'ls -l' shows the correct file and folder names, so this seems to be a Finder issue somehow. 
ls -lc
total 0
-rw-------@ 1 marco  staff  799 12 Apr 15:35 3._TESTFILE1.drawio
-rw-------@ 1 marco  staff  797 12 Apr 15:35 4._TEST FILE.drawio
drwx------@ 3 marco  staff   96 12 Apr 12:10 5._TEST_NO_SPC
drwx------@ 2 marco  staff   64 12 Apr 15:37 6._TEST W SPC
drwx------@ 2 marco  staff   64 12 Apr 15:37 7._TEST_NO_SPC2
-rw-r--r--@ 1 marco  staff    0 12 Apr 16:07 8._PROVAFILE2
Schermata 2023-04-12 alle 17 45 10
claucambra commented 1 year ago

Please excuse my poor English. Thanks for the great development.

Your English is great, no apologies needed

In the case of LDAP/AD, the URL of Webdav becomes UUID from the user name. https://nextcloud.com/remote.php/dav/files/XXXXXXXX-XXXX-XXXXX-XXXX-XXXXXXXXXXXX An error occurred in the Finder in this case. image

Console app logs The domain name and username have changed.

Indeed LDAP authentication has not yet been tested, this will require some changes on the desktop client->file provider communication to get working, but thanks for the logs -- it's on the list of things to get right

claucambra commented 1 year ago

I found another glitch:

  • Added my user to a group;
  • The new group membership adds another folder to the visible ones (that folder was shared by another user to that group, read/write enabled);
  • The newly discovered folder shows up in FIles (web app) as expected;
  • The FileProvider started issuing PROPFINDs on the whole new folder hierarchy;
  • After the full tree visit was finished (PROPFINDs finished) the folder didn't show up in Finder (and fileproviderctl ls didn't log that folder addition).
  • Only when I made a change via web browser in that folder (deleted an empty sub folder), fileproviderctl ls finally logged the incoming change, and Finder finally showed up the new 1st level folder.

So basically looks as if it missed a "trigger" to show up a newly shared folder...

Do you have any logs of when this initial PROPFIND takes place? Either it's not discovering it on lookup or it's ignoring it for some reason, would be critical to figure out which one is happening and why

EDIT: just saw your follow-up comment:

Sorry I forgot to mention: my macbook was rebooted in the middle of the tree visit process. The process restarted after the reboot (PROPFINDs restarted from where they were interrupted).

My hunch is that the propfind for the folder in question happened and was recorded in the database, but when the restart happened these changes had not yet been notified through the file provider API -- so when the rescan happened after reboot this folder was not notified as since it was in the database already it was considered unchanged. Will need to look into this, and this is just a hunch, but it makes sense

claucambra commented 1 year ago

Seems to be working well for me, once I set up notify_push and fixed its configuration. Some better error reporting for when notify_push is not set up correctly would be useful.

Notify_push not being enabled is not really an "error" as the file provider module works fine without push notifications, it's just more efficient with them available.

Separately — can the ability to evict a file/folder be added? At the moment this is not present, which will cause local storage to fill up. Using fileproviderctl to attempt to evict seems to also fail with an error. IMO this is probably the most important feature to have after basic functionality.

Yes, this is on the to-do list

wrench12435678 commented 1 year ago

Does anyone have a link for the build for this? Can only find an app image on the pull request

(sorry if this is the wrong place to ask, couldn't find it in github actions or a post via the build bot)

marcotrevisan commented 1 year ago

Does anyone have a link for the build for this? Can only find an app image on the pull request

(sorry if this is the wrong place to ask, couldn't find it in github actions or a post via the build bot)

It's linked in the announcement comment: https://github.com/nextcloud/desktop/issues/1337#issuecomment-1478386641

Cheers!

marcotrevisan commented 1 year ago

Sorry I forgot to mention: my macbook was rebooted in the middle of the tree visit process. The process restarted after the reboot (PROPFINDs restarted from where they were interrupted).

My hunch is that the propfind for the folder in question happened and was recorded in the database, but when the restart happened these changes had not yet been notified through the file provider API -- so when the rescan happened after reboot this folder was not notified as since it was in the database already it was considered unchanged. Will need to look into this, and this is just a hunch, but it makes sense

I did a new test using a small folder so the whole process could complete quickly. It worked properly. I did it on a group folder, making it appear and disappear by means of both sharing it directly with the current user, unsharing, or adding the user to a group that has visibility on that folder. In all cases it worked properly: the folder showed up in Finder when shared and disappeared when unshared.

My issue was therefore caused by the reboot of the system in the middle of the process. I wouldn't underestimate the frequency of this use case, it may occur regularly when large folders are involved in the share/unshare function. I'm not sure wether a simple "stop" of the system (instead of a reboot) may lead to the same result. If it does, the propbability of that issue arising will be higher...

Thanks and regards!

marcotrevisan commented 1 year ago

Hi @claucambra, in your opinion is this a good time for releasing a second beta build? I saw some stuff and fixes in the commit comments, so I was wondering if it could be worth doing...

Thanks and regards!

marcotrevisan commented 1 year ago

Hi @claucambra ,

I've noticed that sometimes, at least under these circumstances:

when I navigate into a subfolder it doesn't update its contents. It stays stuck for many seconds, then after a while it shows only partial (seemingly old) content. The log only shows rows like below: 

"Set up enumerator for user: marco.trevisan https://nextcloud.<...>"
"Providing enumerator for item with identifier: <...>"
"Enumerator is being invalidated for item with identifier: <...>"
"Received item request for item with identifier: <...>"

Output of fileproviderctl ls <...> for that subfolder:

`

id:__fp/fs/fileID(70701226) ⬇ (indeterminate) ieri, 17:43 (used) ` where all the other subfolders in that directory are not `(indeterminate)`. I tried to add and remove files/folders under that subfolder, and the push notifications arrive and trigger PROPFINDs as usual, the updates are also logged in `fileproviderctl`, but they're not having the desired effect. Is there a way to tell it to "resync" a sub folder? Thanks and regards
misku commented 1 year ago

@claucambra Thank you for the great work! 👍