Support FileProvider API (VFS) in macOS

[helmut72]

https://www.apple.com/macos/catalina-preview/features/

Third-party cloud service integration An all-new FileProvider API for cloud storage providers delivers a new way to seamlessly integrate their services into the Finder without requiring a kernel extension, helping to maintain the security of your Mac. Cloud storage providers can now deliver their apps through the Mac App Store.

[pgauret]

Now that Catalina is out may be a good opportunity to look into FileProvider. This would allow to modernise the macOS desktop client and bring equivalent functionality to the virtual drive for Windows client.

[theJian]

Catalina is out but I can't find a way to create FileProviderExtension on macOS. Does anyone notice that FileProvider is actually removed from Catalina features page?

[mikelynch]

It's strange that it's not listed as a feature in the release, but the Developer documentation does say it's available in macOS 10.15+. I'm afraid I don't know anything about macOS development, but there are some First Steps topics there. @theJian, do those provide any useful guidance for getting started?

It's also just occured to me that this is already implemented in the iOS app, and hopefully that component should also work on macOS. Would that give an easier route to using this API?

One consideration is that presumably the current logic will have to be kept for the moment, in order to support older versions of macOS – I imagine it wouldn't be desirable to require Catalina for a while yet. It'd be great to be support this API as an option though!

[theJian]

@mikelynch I didn't find those useful. Luckily once you created a FileProviderExtension, Xcode will generate template code for you. It's a working simple demo and full of comments. Getting started by reading those code is much easy in my opinion.

[pointlessone]

Documentation is a little inconsistent. On the File Provider Framework page it states availability to be macOS 10.15+ but on the individual protocol pages (e.g. NSFileProviderReplicatedExtension) it's macOS 11.0+. I guess this feature didn't make it into Catalina but since Big Sur is around the corner maybe it's time to revisit this issue?

[leiless]

FYI, File Provider - macOS Support seems available by now.

[aniqueta]

Is this something being worked on? With Big Sur changes, it seems safe to assume that in the next macOS version using the FileProvider API will become mandatory and the Finder extension approach will be deprecated. Thanks.

[leiless]

Is this something being worked on?

Don't know.

It seems that FileProvider not ready yet, and Apple hasn't given any sample code on it.

[hemancuso]

@leiless we actually just shipped Nextcloud support in our FileProvider product (Strongsync). Definitely is ready. Happy to answer questions.

[leiless]

@leiless we actually just shipped Nextcloud support in our FileProvider product (Strongsync). Definitely is ready. Happy to answer questions.

Does there any sample code on FileProvider API? I've checked Apple Developer docs, haven't seeing it yet.

How you guys support FileProvider for Strongsync?

[hemancuso]

I believe the very latest XCode betas generate a template project

[helmut72]

@leiless we actually just shipped Nextcloud support in our FileProvider product (Strongsync). Definitely is ready. Happy to answer questions.

Target: common Apache2 WebDAV

Tasks:

1. Created test folder on macOS: ok, also on the server
2. Copied my Notes (a lot of Markdown files in private and work with subfolders): failed. Work and all subfolders are on the server, no subfolder in Private. On both no single file was uploaded!
3. Deleted test folder: still on the server (empty trashcan on macOS doesn't helped)
4. Result: uninstalled Strongsync

[andreasley]

Xcode 12.5, which was released today, contains a File Provider Extension target template to help getting started with macOS File Provider extensions.

[leiless]

Xcode 12.5, which was released today, contains a File Provider Extension target template to help getting started with macOS File Provider extensions.

FYI: https://developer.apple.com/documentation/xcode-release-notes/xcode-12_5-release-notes

[aniqueta]

Glad to see the activity here. Looks like the API is maturing in macOS Monterey. FYI: Box has adopted it now: https://blog.box.com/get-more-secure-seamless-box-drive-experience-across-macs

[solracsf]

See https://github.com/nextcloud/desktop/blob/master/doc/architecture.rst#virtual-files for how to enable experimental support on v3.4.0.

[pszypowicz]

See https://github.com/nextcloud/desktop/blob/master/doc/architecture.rst#virtual-files for how to enable experimental support on v3.4.0.

Hi, is this coded with a FileProviderAPI? Could you point to a commit/PR implementing this?

[aniqueta]

Note that OneDrive [1] and Dropbox [2] are stating there are changes coming in macOS 12.3 (March?) that is making them fully migrate over to the FileProvider API, with macOS 12.2 being the last version they support with their current virtual files / online files implementation. I can't find anything from Apple indicating what these changes are, but I suspect we'll find out soon...

[1] https://techcommunity.microsoft.com/t5/microsoft-onedrive-blog/inside-the-new-files-on-demand-experience-on-macos/bc-p/3066588 [2] https://help.dropbox.com/installs-integrations/desktop/macos-12-monterey-support

[jeffmancuso]

They both have exceptions to the no kernel extension policy allowing them to use their previous solutions. I imagine this exception is finally going away in a late-cycle point release rather than in macOS 13. This increases their pressure to actually migrate over to FPE.

[marcotrevisan]

Just for the records: https://developer.apple.com/documentation/fileprovider/macos_support/syncing_files_on_macos contains an example XCode project that uses NSFileProviderReplicatedExtension.

Would be great to have the system take care of most of the synchronization work for you, and you only have to focus on the client calls and the extra features.

[claucambra]

Adding support for File Provider is not exactly straightforward with the desktop client as all of our networking and synchronisation code is written in Qt/C++ and therefore not easily usable by a File Provider extension. It seems to me that an easier way of going about this would be to make the File Provider extensions independently network with the server, and therefore making them mostly independent.

Since we have already have a great iOS app that supports the File Provider API on iOS, we should borrow some of the code from there. Unfortunately, there are also a lot of small differences in how some APIs work on iOS vs macOS (including File Provider itself) so this is not exactly straightforward either.

Here is a basic list of the things that need to be done before the extension is ready:

• [X] Get the Nextcloud iOS communication library to build on macOS
• [x] Test the Nextcloud iOS communication library on macOS and ensure nothing is borked
• [X] Graft iOS networking and service code onto the File Provider Extension in the desktop client
• [X] Get the File Provider extension with iOS code to compile on macOS
• [X] Get adding per-account File Provider domains to work
• [x] Correctly acquire desktop client user account data from the side of the File Provider extension
• [x] Test each iOS component to ensure there is no borkage
• [x] Securely connect to the server through the File Provider extension
• [x] Successfully list user files and folders
• [x] Successfully fetch file contents in File Provider
• [x] Successfully sync local changes in File Provider
• [x] Successfully sync server-side changes in File Provider

[marcotrevisan]

@claucambra in my humble opinion this is a very important improvement, seems very reasonable as you described it and should be kept in high priority, as the current Mac OS implementation is struggling to keep up with the everyday user needs, especially when many files and folders are involved. This change would resolve "by design" many important issues and shortcomings (just today a colleague's local disk went out of space because he downloaded too much stuff and the NC folder was over 100GB even if he's using virtual files.. a FileProvider implementation would have freed up space).

As a user, I have tried other clients (both paid and free) that perform solutions similar to the prospected one, but they all fall short in using only a "provider agnostic" set of the Webdav API. For example most don't establish a websocket for push notifications (assuming the Client Push app is up and running), or don't use the NC chunked I/O api, etc etc... Or, they don't use File Provider API and kinda implement a local proxied network share... while that's good for space saving and generally works (except for Spotlight searches), when they run into exceptions the behaviour is often unpredictable and the cause is obscure to the user.

Thanks!

[alex0411]

Hi, are there any updates when this feature will be pulled into the development / released? At the moment this missing feature is holding us back to use Nextcloud in a productive environment, since a lot of our users are using macos based clients and using the experimental option in the client is not an option for a productive environment.

Thanks a lot!

[alex0411]

Hi, are there any updates when this feature will be pulled into the development / released? At the moment this missing feature is holding us back to use Nextcloud in a productive environment, since a lot of our users are using macos based clients and using the experimental option in the client is not an option for a productive environment.

Thanks a lot!

Push

[gaodahai]

Hi, are there any updates when this feature will be pulled into the development / released? At the moment this missing feature is holding us back to use Nextcloud in a productive environment, since a lot of our users are using macos based clients and using the experimental option in the client is not an option for a productive environment.

Thanks a lot!

Hi, we also just saw the mentioned troubles happen. Is there any possibility to get this fixed? Or how do you work to avoid these troubles? Thank you very much!

[alex0411]

Hi, are there any updates when this feature will be pulled into the development / released? At the moment this missing feature is holding us back to use Nextcloud in a productive environment, since a lot of our users are using macos based clients and using the experimental option in the client is not an option for a productive environment.

Thanks a lot!

Push again. Maybe somebody could give us some feedback here.

[marcotrevisan]

I'm not a NC developer but seems like the team is working on improving the current sync engine quality, even if I was unable to properly check for progress in the FileProvider support (anyway NextcloudKit was created and received merges).

4970, #5016, #5011 are just examples of that.

I'm just guessing but the dev team may be currently preferring focusing on the "SyncEngine", making it better. In this case I'd urge implementing #4464 which in my humble opinion would make it work hugely better on all platforms. I can live with ".nextcloud" files, much less with a sync engine that scans the whole world. Thanks!

[hemancuso]

Only posting again on this thread, just in case people are curious or want some help on an implementation. I had implemented support in a product I make back in 2021 comment and will be soon adding support to ExpanDrive. Whoever on the team ends up taking on this project feel free to ping me if you need assistance.

[alex0411]

Hi, is there any feedback already available? I can just share my oberservation within our use-case: Our mac users are refusing to use nextcloud because of this missing functionalities and to stay on the existing file storage software. So it would be really import to get any insights about this topics and the priorities behind it. Thanks a lot!

[mamboone]

Hi, is there any feedback already available? I can just share my oberservation within our use-case: Our mac users are refusing to use nextcloud because of this missing functionalities and to stay on the file storage software. So it would be really import to get any insights about this topics and the priorities behind it. Thanks a lot!

I can just add to this comment. Eagerly waiting for Nextcloud to support the "on demand" feature right away. Looking on the OneDrive side from Microsoft - there this feature works quite fine for months already with Mac.

[p4t5h3]

Just to throw in my personal experience "two-cents" in working with file providers for the people reading along like me: It is not that simple and might be more complex to wire up to the Nextcloud client code and API (which I am not familiar with). This are not simple adaptions but quite different processes compared to classic synchronization. Implementations from iOS cannot be simply ported to macOS due to the API differences.

[tobiasKaminsky]

We are working on it, but as @peterthomashorn outlined, it is really not that simple, unfortunately. As soon as we have an alpha/beta version, we will inform you here and ask for feedback.

[marcotrevisan]

I'd eagerly alpha/betatest it. Consider me added to the list if you like. Also, if it can be of any help I'd vote for a bare-bones Files-only client implementation, no Talk and no other stuff, just focused on Files operation (including locks and push).

[alex0411]

Hi, as feedback after several months without going forward with this feature. We decided to sunset the productive usage of Nextcloud and will use another platform.

[marcotrevisan]

Just for the records, since one of their latest updates the Dropbox client for mac now shows up as a shared position and the icons for the sync state are pretty much the same as iCloud’s. Looks like it’s now a FileProvider based implementation, even if it was one of the snappiest clients out there… which imho further justifies the efforts for this work ;)

[claucambra]

This is currently top of my to-do list and is being actively worked on. There's no timeline or anything like that, and it's not even close to done at the moment, but we have made some nice progress to get the File Provider app extension to communicate with the desktop client. Lots left to do still 🙂

[piermorel]

For those following what Dropbox and others are doing with File Provider, it seems that it always leads to major loss of functionality : it breaks spotlight, files have to be on the main drive (this one is mind boggling when you see the cost of storage with Apple), and the functionality to always keep a folder local is not very trustworthy and its slower (see dropbox documentation).

So shouldn't the focus be to find solutions to keep on supporting macOS without using file provider until Apple fixes it?

[claucambra]

For those following what Dropbox and others are doing with File Provider, it seems that it always leads to major loss of functionality : it breaks spotlight, files have to be on the main drive (this one is mind boggling when you see the cost of storage with Apple), and the functionality to always keep a folder local is not very trustworthy and its slower (see dropbox documentation).

So shouldn't the focus be to find solutions to keep on supporting macOS without using file provider until Apple fixes it?

We do not have any plans to remove conventional file synchronisation from the desktop client -- like VFS, FileProvider will be an option

[claucambra]

Quick progress report -- file browsing is now working! 🙂

https://user-images.githubusercontent.com/70155116/216428719-3f6601df-66d0-4eea-bc09-f841ab375310.mov

[jesusdeveloper]

Quick progress report -- file browsing is now working! 🙂

Screen.Recording.2023-02-02.at.20.16.24.mov

Oh my god! I really wanted to see this because I've been waiting for it for a long time!!!! Very good work! Hopefully soon we can use it!

Congratulations, good job, keep it up!!!

[mrtn223]

Really happy to hear you are actively working on it!!! This is the only reason our company didn't migrate from Dropbox to Nextcloud yet...

[claucambra]

File content fetching now working

https://user-images.githubusercontent.com/70155116/219470752-1118b138-56bb-4b1d-b9f6-47454a740db4.mov

[xavi-b]

Amazing ! Do you have an estimate on when it would be available in a release ?

[claucambra]

Amazing ! Do you have an estimate on when it would be available in a release ?

When it's finished ;)

[miguelarios]

Take your time! We appreciate the great job you are doing! You can't rush greatness.

[footlooseboss]

File content fetching now working

have you looked into partial content fetching? not many apps seem to be making use of it, given how useful it is

NSFileProviderPartialContentFetching

Support for fetching part of a file’s content.

Adopt this protocol to let the system request only part of a file. Apps that read files provided by your extension can benefit from this feature, either by minimizing the amount of data your file provider needs to download, or by finishing the download quickly, freeing up the reading process.

To trigger a partial download, an app must use POSIX read operations to read part of the file. If you clone the entire file, or read the file using file coordination, the system requests the entire file.

For example, a photo app could read just the metadata from each picture in a large album, without having to completely download all the images. Alternatively, a video streaming app could begin playing the video before reading the whole file, reading chunks of data just before it needs them.

[marcotrevisan]

Since we're in the works, I've just tested the usage of the current client (3.7.3) with the newly released Temporary Files Lock v26.0.0 app enabled.

It seems to be finally working properly in my expected use cases.

To make it easy living with locks on the client it would be great to have a "lock" icon appearing along with the file in Finder (if I'm not mistaken the FileProvider framework makes it possible to add icons and other graphic decorations to files based on provider's custom logic), because the current way of showing it is not-so-user-friendly (you have to right-click, go to Nextcloud menu item and check if the file is locked). The everyday user would easily get frustrated. Thanks!

[claucambra]

Pull request is now open. More details to come...

[marcotrevisan]

I'm sure this will be a huge leap forward for Nextcloud Files on Mac OS. Hopefully it will also help decreasing PROPFIND load on server.
Thanks @claucambra and all the dev team!

[claucambra]

As those of you who watched the NC26 announcement will know, File Provider support for our desktop client is now a thing!

This has entailed writing a new, macOS-specific sync engine built from the ground up around Apple’s APIs, designed to make the best use of the system’s frameworks.

We’re super excited to release this new engine as part of a new Nextcloud macOS VFS Beta application. This app is separate to the standard client. It includes all the characteristics you know and love from the desktop client, with the new virtual files/File Provider module included.

However, there is a reason we are releasing this separately and why we’re labelling it a beta. Coming from the person who worked in this - file sync is hard. This beta release will have bugs, and while we have not encountered any major data loss, it is not impossible this might happen. So let me be emphatic — we do not yet recommend the use of the file provider beta app on production instances. Data loss is possible. Use it on your own data at your own risk!

With that out of the way, have fun (and stay safe!) playing with our beta! You can go ahead and download it from the link below:

https://download.nextcloud.com/desktop/prereleases/virtualdrive/Nextcloud-VFS-beta-macOS.pkg

I would really, really like feedback on any bugs, errors, and crashes that you encounter during use of the beta. The file provider module uses Apple’s unified logging system, so you can extract relevant logs using the Console app or by using the log utility from the terminal. To make it easier to find the Nextcloud File Provider related logs, filter by process FileProviderExt and by subsystem com.nextcloud.cloud.FileProviderExt; if you're using the Console app, make sure to check "Include info messages" and "Include debug messages".

When reporting bugs, please make sure to send us this data as well as the conventional debug archive from the desktop client. Also, the more detailed you are in explaining how the problem arose, the better I will be able to go ahead and fix it :)

A couple of last minute tips:

• First sync will take a while -- we need to scan the entire server first, so be patient...
• For those comfortable using the terminal, the fileproviderctl tool is extremely helpful in providing information on each file provider domain. It can also be used to run diagnostics and basic commands on them. Sending any information provided by this tool will be very helpful
• If the Nextcloud macOS VFS beta does not appear in the finder sidebar after setting up your account in the client, restart the desktop client as well as finder, and then it should show up
• Stability is not the only thing left to work on regarding file provider. We have a bunch of file provider related features we plan to start working on that will make the user experience even better!