Feature: Standalone WinRT server for background operations

[hez2010]

What feature or improvement do you think would benefit Files?

We initially chose Win32 Shell APIs for file operations because we wanted the file operations to continue even after Files was closed, but Win32 Shell APIs are designed for automation, not for the general use by a file explorer. And it comes with its own limitations (e.g. slow, no built-in transfer speed calculation and etc.).

We should consider implementing file operations as an out-of-process WinRT exe server, so that we no longer need to rely on the Win32 Shell APIs, neither facing the performance and maintainability issues due to serialization and inter-process communication. By using WinRT we can expose the APIs with WinRT types directly (such as IAsyncOperation<T> and Stream), and call them directly from another process.

Requirements

It was possible only with C++/WinRT, and CsWinRT still doesn't have built-in support for authoring OOP WinRT exe server for now (see the original feature request in CsWinRT: https://github.com/microsoft/CsWinRT/issues/1177). But recently a project Shmuelie.WinRTServer has this implemented, so I think we can start to investigate it.

Files Version

Planning for post 3.0

Windows Version

10.0.22631.2506

Comments

No response

[gave92]

You mean implementing file ops using Win32 API? E.g CopyFileEx, MoveFileWithProgress, etc?

[hez2010]

Yes. Implementing file ops using Win32 API in an external process and expose those APIs through WinRT ABI in an external process. This is similar with what we did before in fullTrustedProcess but not the same. By authoring a WinRT exe server, we don't need to serialize and dispatch messages by ourselves (so there won't be data serialization overhead), instead we can call the API in WinRT exe server from Files main process and operate those "remote object" directly.

[gave92]

Can a WinRT exe server run as administrator? (without the main app running as admin)

[hez2010]

Can a WinRT exe server run as administrator? (without the main app running as admin)

I have no idea about it now. But I think we might not be able to activate an elevated WinRT type from a non-elevated process.

[yaira2]

Will this help us support third party handlers? #7518

[hez2010]

Will this help us support third party handlers? https://github.com/files-community/Files/issues/7518

Need to investigate. But I think it's possible.

[hez2010]

The investigation and experiment have done. Now I'm going to commit to the implementation. Note this is not a Code Quality work but a Feature work (which allows us to have a standalone background server for long-running tasks)

[JaiganeshKumaran]

Why not simply just run the regular executable but without a window open? It will be much simpler. There are not that many resources about creating an out-of-process WinRT server.

[hez2010]

Why not simply just run the regular executable but without a window open? It will be much simpler. There are not that many resources about creating an out-of-process WinRT server.

Because then you will need to handle all serialization stuff and message dispatching by yourself, which results in major maintainability regression. By using WinRT out-of-proc server you can just call APIs in the server directly like you are calling a method in the same project.

[JaiganeshKumaran]

Because then you will need to handle all serialization stuff and message dispatching by yourself,

No I mean just do it in the same process as the main app. Just hide the window.

[hez2010]

No I mean just do it in the same process as the main app. Just hide the window.

This will still face the concurrency issue. We want all operations to happen in a single process to allow us to update to LiteDB 5.

[yaira2]

The investigation and experiment have done. Now I'm going to commit to the implementation. Note this is not a Code Quality work but a Feature work (which allows us to have a standalone background server for long-running tasks)

That's good to hear, keep us updated!

[JaiganeshKumaran]

We want all operations to happen in a single process

I did mean the same process as the app, meaning hitting X on the window will only hide it and the operations will continue to run.