Open jonwis opened 4 years ago
MarkIngramUK
commented
4 years ago For applications that are already compatible with sandboxing (i.e. only read from %APPDATA% or %PROGRAMDATA%, and only access files explicitly chosen by a user) I don't think a continuous reminder is warranted. This could perhaps be controlled via some sandbox capabilities options that are chosen at development time (similar to the macOS model).
eklipse2k8
commented
4 years ago A good test of this feature would be, I can create an SQLite database in the user's documents folder. I shouldn't need permission to do this if it's a new file. This is such a common scenario that the sandbox and the politics around it have never tried to solve for. An app should never need "broadFileAccess" but they should also make it so users can easily open documents and see their saved file right there.
soumyamahunt
commented
4 years ago A good test of this feature would be, I can create an SQLite database in the user's documents folder.
Why not create in AppData folder itself? A lot of users have complained apps creating random folders in their library folders like Documents, Music etc. introduces clutter to their workspaces and they have more trouble in finding their Work folders. Also, this kind of behavior from developers is what prompted MS to limit storage access in first place.
benstevens48
commented
4 years ago It’s really great that this is being looked into, and that it’s the first issue here. UWP file performance and permissions is the thing I’ve most wanted to see an improvement in since 2015! I imagine that any permissions changes will be difficult to deliver via NuGet, so hopefully something can happen on this as soon as possible in order to ensure we don’t have to wait too long before the users have the necessary OS updates.
Below I’ve put together a collection of scenarios related to file system performance and permissions that are problematic in UWP. These are things I have encountered while building my photo-viewing app. (Maybe it’s a bit selfish of me to concentrate on these problems, but hopefully they cover a wide range of use cases). If this post is too long, feel free to move it to a separate issue. Also, I am very happy to discuss any of these issues/ideas further. The scenarios themselves come first, followed by the problems with them in UWP and some possible solutions. I hope that this project will make time to think more creatively about how to deal with some of the file permissions issues. Some of my suggestions below are just initial ideas, so don’t judge them too harshly. It’s such an important subject that I think it’s worth spending a lot of time thinking about.
I put this in as point 0 as it influences all the others. A key principle of my app is to allow users to organise their photos how they like and not to force them into any patterns. Therefore the app cannot assume that all the user’s images are in their Pictures library, and indeed it’s designed for viewing all images, even for web designers etc who are probably not keeping images in there. Also, I’m going to assume that broadFileSystem access is not used, because that is perhaps too unrestricted and also the current user experience for it is poor.
My app has quite a simple concept of allowing users to select files they want it to display. They can do this by selecting a folder, optionally including subfolders, or selecting individual files etc. They can make multiple selections. It is not uncommon for a folder selected to contain 10,000+ image files. The app needs to maintain a list of the selected files (including the order they were selected in etc), therefore it needs to be able to list files in folders, and the quicker and less resource-intensive this is, the better. Obviously the app needs to be able to access the contents of these files alter on, but at this stage a path and file name is all that’s needed.
My app has the option to auto-refresh the selected files to match the contents of the folders that the user selected to view. This can include modifications to files as well as files added/removed/renamed.
This should ideally be fast. Also, it expects to be able to read properties simultaneously.
This is self-explanatory.
This is self-explanatory. There is a button in the app that allows the user to duplicate the file they are currently viewing in the app. (The file is one of those selected in scenario 1). This should place a copy of the file in the same directory as that file, with the name suffixed by ‘ Copy’ or similar. It should be a one-click operation as it’s a convenience function for the user (e.g. to make a copy to edit).
The user may have selected an individual file to view in scenario 1. The app would like to show a button that would load the remaining files from the same folder in this case, ideally via a 1-click operation.
Users who make heavy use of this list may add hundreds or even thousands of files to it.
Such as slideshow may consist of thousands of images, which were originally selected in multiple different ways. The saved data file would store the file paths of these images, and when the data file is loaded it needs to be able to load the associated images.
The user needs to be able to do the same things with these files that they can do with files selected another way. An important feature of this is that the neighboring files are ordered in the same way as in File Explorer. The neighboring files list should be resilient to problems with the file system indexer.
The challenge here is simply that listing the StorageItems in a StorageFolder is incredibly slow and resource intensive compared to using the standard .Net/Win32 API to list the file paths. A quick test showed that in .Net it takes about 0.003ms per file, whereas with UWP StorageItems it takes about 2ms per file, which is around 700 times slower. Also, looking at the Runtime Broker, UWP uses about 70kB memory per file, which is a huge cost when loading 10,000+ files, whereas .Net uses around 0.1kB per file (that’s a very rough estimate).
UWP has the StorageFileQueryResult.ContentsChanged method for watching for file system changes. However, it has absolutely no configuration options for what sort of changes to watch for, so it will fire on any change, even something like last access time I think, and include subfolders. (I agree that if you have no configuration options that firing on any change is the best option). It also does not tell you what changed. Combined with the speed issues in scenario 2, this makes for a doubly bad experience. If you look at the .Net docs of file system watching, they give an example of watching for changes in “C:\”, which made me laugh. In UWP, assuming that you managed to get permission for this folder, in order to determine what changed, you would first have to enumerate all the contents, which would literally take several days, then on a change you would have to enumerate all the contents again, taking several days, and in fact several weeks if you also want to check the date modified to detect file modifications, and look at the difference. Also, at one stage, I tried adding a separate watching for each of 600 sub-folders that had been added to the app (in order to avoid having to query all the files for changes) and basically it ground to a halt – it seems to create a new thread for each watcher which is not necessarily very efficient.
I added this because I’m not aware of a way to do this easily in .Net without writing interop code. Also, for any alternative to StorageFile in UWP, it needs to be possible to get these properties ideally without the overhead of creating a StorageFile. Also, there are some issues with these in UWP. Firstly, if you call any of the Get..PropertiesAsync methods apart from RetrievePropertiesAsync simultaneously on the same StorageFile instance, they will fail, even though they are read-only as far as I can see so have no reason to. Secondly, fetching System.Size and System.DateModified using RetrievePropertiesAsync after the first time does not return an up-to-date result. Thirdly, there can be issues with scenario 9, which I’ll mention later. Finally, using SorageFile.GetBasicPropertiesAsync to get the file modified data is incredibly slow compared to .Net. in .Net it takes about 0.05ms per file. In UWP, it takes about 5ms per file, which is 100 times slower, and also adds about an extra 60kB per file (if you keep the StorageFile), compared to about 0.05kB per file in .Net.
Currently in UWP if a user drags and drop a file/folder from File Explorer into the app, the file/folder is read-only, so can’t be edited, unless the app has access to the file path via some other means.
This duplicate button is not possible in UWP if the user opened the file individually or by double-clicking in File Explorer, unless the file is in the Pictures library (assuming the Pictures library capability). Worse, the app cannot even open the save file dialog to the current folder, since there is no way to set the starting folder of the save file dialog.
This is a similar problem to scenario 5. If the user opened the file individually this is impossible, unless the file is in the Pictures library.
The only general way to save access to files is via the FutureAccessList. However, this is beset with problems. It has quite a small limit of 1000 files which is easily exceeded. The developer has to write code to handle the case when this happens, which is hampered by the fact that any of the APIs surround the FutureAccessList are quite slow, typically taking several milliseconds, and hence such a function has to be carefully debounced so as not to impact performance of the app. Furthermore, the app has to maintain a list of the tokens it is using and ensure this is in sync with the list, which is awkward. It would be easier if the list could be accessed via file path, with a use count. This could be stored in the metadata currently but for some reason in order to obtain the metadata for a token you have to enumerate the whole list. I find that whenever I add a feature to my app that requires this list, it takes me a couple of days to get the logic right, and even then it’s full of annoying compromises. Compare this with a .Net app where you can simply forget about the whole thing and spend zero days on it.
Due to the fact that the slideshow list would be stored in an external file which could then be deleted without the knowledge of the app, and also because this could exceed 1000 files, the FutureAccessList cannot be used for this purpose, so it is impossible unless the files are in the Pictures library.
There are currently various issues with the neighboring files query, which I believe is related to the way it uses the file system indexer. Firstly, it is quite common for the file system indexer to stop working properly. On my old PC this happened after every major Windows update. In this case, the neighboring files query would contain no files or be missing some files. I found disabling and re-enabling file content indexing fixed the problem. Also, if you use RetrievePropertiesAsync to retrieve properties for files loaded in this way, you won’t be able to access some properties. Also, some properties like title will be limited to a certain number of characters (e.g. around 250 for System.Title). Another problem is that the allowed file extensions does not always include all media files .e.g. for some reason .webm video files are not included in the neighboring files list (as of 1909).
Scenarios 1-3 are performance-related, and in the case of scenario 2 the APIs are simply incomplete for UWP and need to be completed. The performance issues should be covered by the suggestions in your proposal. In my opinion in this case it is important that all APIs involving StorageFile/Folder have versions which accept the lightweight alternative. This should include an alternative for RetrievePropertiesAsync. Also, once you have permission to access a file for the current app session, you should be able to access it via any means, unlike currently where StorageFile.GetFileFromPathAsync(myAlreadySelectedStorageFile.Path) will fail. (Memory usage should be taken into account if it becomes a problem, but hopefully it shouldn’t).
Scenario 4 is a simple matter of change the default behavior to read-write, and I think most people agree with this.
Scenario 9 could be covert by enabling the app to indicate that it doesn’t want the neighboring files query to use the file system indexer, or something like that. It could also be solved by just allowing access to the folder, as long as it was possible to get the current sort order. I know this is an increase in access rights, but it would also solve some other problems which is why I mention it. The issue of some media files being omitted could be solved by Windows having a list of such allowed extensions that could be updated independently of a feature update so we do not have to wait a year for such a simple change (assuming the list was then updated to add files such as .webm).
Scenarios 5-8 are more tricky. Basically all of these require more permissive file access or additional capabilities. I have sketched out a few ideas, some of which you might consider outrageous, but I think this is a very important area that needs exploring.
Finally, my thoughts on periodic warnings for users about apps using capabilities - personally I find this kind of notification annoying, so I would be against it. Better to make it easy for a user to investigate this, without giving them an intrusive notification. Maybe it could go so far as something like the using your location icon in the task bar, although even that is potentially annoying if there are too many.
Thanks for your patience in reading that!
eklipse2k8
commented
4 years ago Why not create in AppData folder itself?
Sorry @soumyamahunt, but this is a limitation. What if the user wants to copy the file to a USB drive or put it in their OneDrive storage? I'm forced to build a file manager UI so they can "export" the document, when the file system provides all of this functionality already.
AppData is for caches and app metadata, but not user documents that they want to access, move around or do as they please with. AppData should be considered off limits to users mucking around with anyway.
soumyamahunt
commented
4 years ago Sorry @soumyamahunt, but this is a limitation. What if the user wants to copy the file to a USB drive or put it in their OneDrive storage? I'm forced to build a file manager UI so they can "export" the document, when the file system provides all of this functionality already.
AppData is for caches and app metadata, but not user documents that they want to access, move around or do as they please with. AppData should be considered off limits to users mucking around with anyway.
You don't have to build a file manager UI, user can go to the AppData folder of your application and copy it. If you are insistent in creating folder/file anywhere other than that you can ask user to create a folder with file picker and then add the folder/file in FutureAccessList. Automatically creating folder anywhere is huge no for me.
MarkIvanDev
commented
4 years ago These are the limitations I see in UWP file-access APIs besides the ones that are pointed out above
broadFileSystemAccess capability limitationsStorageLibraryAPIs. You still cannot get the user's libraries when calling StorageLibrary.GetLibraryAsync even though technically, Libraries are part of the file system that is available to the user. Since the docs are saying that the app should not declare the capabilities for accessing the libraries (documentsLibrary, musicLibrary, picturesLibrary, videosLibrary) if it already uses the broadFileSystemAccess capability, declaring broad file system access should also grant access to the user's libraries through the StorageLibrary APIs
Removable Storage Access. Although accessing removable storage requires another capability (removableStorage), broadFileSystemAccess should help lessen the restrictions. As of now, the app still needs to declare the file type associations just to access those specific file types in the user's removable storage devices.
UNC folders. As with the previous point, accessing UNC folders needs a combination of capabilities (privateNetworkClientServer, internetClientServer, enterpriseAuthentication) to access them. Combining broadFileSystemAccess should also remove the file type restriction.
...FromApp API although helps with performance is still severely limitedCopyFileEx, although available to UWP, still only accepts a file path to work. If only it has a variant that accepts a file handle, a workaround can be made. Or better yet, expose a FromApp api that can monitor progress.Since these APIs are not universally available, I think UWP should not be given access to these. It would add an added complexity of checking whether the device that the app runs on supports these or not. IMHO, the Windows.Storage is already a major improvement over the Storage APIs available in previous implementations. It just needs to be updated to have more APIs that will better align with the design of UWP
UWP apps already has a robust way of requesting capabilities (i.e. Microphone, Camera) from the user that does not require a restart. I think this design should be implemented when requesting access for ANY capability. As it stands, changing access to the File System requires an app restart that users can mistake for a crash and does not provide a good look for the apps that use these kind of capabilities.
eklipse2k8
commented
4 years ago You don't have to build a file manager UI, user can go to the AppData folder of your application and copy it. If you are insistent in creating folder/file anywhere other than that you can ask user to create a folder with file picker and then add the folder/file in FutureAccessList. Automatically creating folder anywhere is huge no for me.
The problem is, AppData is hidden and I don't believe it's even easy to traverse because the final document would be behind your package folder that could be anything. If anything, it's there, but it's very unfriendly to users. That's why we're writing our software right?
The scenario I see is this,
This is not about letting an app write junk all over the place, this is about a document based application creating a good user experience.
lukeblevins
commented
4 years ago Hi. I'm the developer of Files (UWP).
Our primary pain points with the current experience of integrating the native Win32 file-access APIs into a UWP app were the things the Win32 File APIs lack (at least for UWP apps):
1.) A simpler alternative to ReadDirectoryChangesW()
2.) Fast replacement for WinRT GetThumbailAsync()
3.) Viewing hidden items and shortcuts
Also, if we're planning on keeping the file/folder pickers, they should really have a modern UI to look familiar in our WinUI apps.
jonwis
commented
4 years ago @duke7553 - thank you!
Simpler alternative to ReadDirectoryChangesW
Have you tried out StorageFolder.TryGetChangeTracker ?
Hidden items and shortcuts
Does FindFirstFileExFromApp help here?
lukeblevins
commented
4 years ago @jonwis My app, Files, previously included a purely WinRT file-access implementation. (Meaning, we used StorageFile and StorageFolder) It was highly optimized at that because we retrieved storage items in batches, and prefetched properties for partial storage items, yet users complained this approach was slower than they were familiar with. As of January, we now use the FindFirstFileExFromApp() to load the items themselves, but we do still rely on StorageFile/StorageFolder to return thumbnails and other properties for the items in a directory.
Regarding StorageFolder.TryGetChangeTracker, my understanding of it was that it works only when the user is accessing libraries, so something like ReadDirectoryChangesW was needed here. I implemented it in Files (UWP) with the last update, v0.10.1 .
lukeblevins
commented
4 years ago Sure, we can access items with the hidden attribute this way, but they are invisible to all of the WinRT file operation methods (i.e. Cut, Copy, Delete)
P.S. I will admit my case is kind of extreme since I built a file explorer. :)
jonwis
commented
4 years ago @duke7553 - There's a whole raft of ...FromApp APIs like CopyFileFromApp, DeleteFileFromApp, and more. I get not wanting to re-implement StorageFile yourself from these primitives, of course. :)
Hey @smaillet-ms - does StorageFolder's change tracker work on any arbitrary storagefolder, or only on libraries?
michael-hawker
commented
4 years ago There's also some weird restrictions on existing capabilities, like access to the Downloads folder only lets you create a sub-directory, see doc issue here. If you're trying to make an alternate browser, this is a weird experience for anyone using any other browser. Maybe there needs to be a AllDownloads capability that's slightly broader?
(Similar to @benstevens48 Scenario 8 above), I think there's a challenge in having any sort of project system at all (and loading these types of file sets from any other source). If a file references the absolute or relative path of another file (or files think '*' wildcard in a csproj); you can't open those references in a UWP today unless you ask the user to select a parent folder that has access to all references. It does seem like a hard challenge to solve though with figuring out an access pattern for detecting and allowing this without needing the full BroadFileAccess capability though... I mean the fallback in UWP is have a folder based project system; however, that doesn't track well with consuming any external formats to your app.
kmgallahan
commented
4 years ago A StorageFolder.MoveAsync method would be highly useful. The current recommendation is... 🤷♂️
There is not currently a "MoveAsync" or similar method. One simple implementation of moving a folder might be to get the desired folder, copy it to the desired location, and then delete the original folder.
soumyamahunt
commented
4 years ago @jonwis ...FromApp apis doesn't work reliably either. I tried to implement GetFileAttributesExFromApp to check whether a file is read only when user opens file via picker or file activation, but this doesn't work when app opens the file for the first time (subsequent query works).
shresthasource
commented
4 years ago @duke7553 - There's a whole raft of
...FromAppAPIs like CopyFileFromApp, DeleteFileFromApp, and more. I get not wanting to re-implementStorageFileyourself from these primitives, of course. :)Hey @smaillet-ms - does StorageFolder's change tracker work on any arbitrary storagefolder, or only on libraries?
From what I remember, it is supposed to work on any folder. However, my last attempt to implement it failed. The issue I faced was that it did not trigger the event right after the changes. I remember reading something along the lines of ...it will wait till something else (don't remember what exactly) before the event is triggered.
shresthasource
commented
4 years ago P.S. I will admit my case is kind of extreme since I built a file explorer. :)
Considering WIndows Store has a dedicated subsection titled "File Managers", I would argue, this is not an uncommon scenario.
ptorr-msft
commented
4 years ago There's also some weird restrictions on existing capabilities, like access to the
Downloadsfolder only lets you create a sub-directory, see doc issue here.
This is a missing feature (downloads folder capability) -- please add another issue for it. Thanks!
brabebhin
commented
4 years ago As a developer of media applications, I find the existing file system access capabilities for UWP are horrendous, and are probably the biggest reason the platform not actually taking off at all. Any kind of serious application will require decent levels of file system access, and decent performance for those APIs. The broad file system access capability does solve the problem to some extent, but this does not address the issue of performance.
Not only are the search / access / property APIs stupidly slow, but they are complimented by equally slow read & write stream APIs. It is unacceptable to wait for several seconds to open and parse 25 files, just because of some unknown limitation imposed at API design time. What is the point in having those APIs as async if the user has to wait ridiculous amounts of time just to perform basic operations? Better have the UI freeze for 1 second than wait 10 seconds staring at a loading bar.
An additional problem is the assumption that every use will put their music / video / pictures / documents in their respective libraries. Truth to be told, I have never seen a single user who actually follows the library pattern completely. Not a single one. Broad file system access should in no way be a restricted capability: it should be top and front. Simply restrict access to obvious danger places like C>Windows and other system folders, and you're golden. Any app that passes through the store should be considered safe, i don't see the point in adding additional security impediments.
These APIs need to go back to the drawing board and be designed for speed and scalability. If Storage.* APIs get deleted tomorrow, I will go out and dance on the recycle bin. They are the biggest reason no productivity app is currently following the UWP model.
jtorjo
commented
4 years ago As a developer of media applications, I find the existing file system access capabilities for UWP are horrendous, and are probably the biggest impediment on the platform not actually taking off at all.
This is definitely the biggest impediment. The whole "sandboxed" concept is a half baked idea of someone that probaly listened to some buzzwords at some point.
But as a developer that reimplemented pretty much all of his app in UWP, there's sooo many issues that will bring you to the "curse master" level.
An additional problem is the assumption that every use will put their music / video / pictures / documents in their respective libraries.
I've been saying the same thing for months. Having said all that, while I do hope this will be fixed, I'm almost 100% sure this will be dragged for years before MS will do anything about it (if they will do something ever). The whole "sandbox" is soooo broken, that no one at MS seeems to want to work on it. Just look at compile times - we've been promised they would be improved for close to a year. Promised again they would start working on compile times 2 months ago, but nothing to show for. But nothing is happening - just promises and delays.
soumyamahunt
commented
4 years ago Also, adding System.IO.File and other System.IO APIs support to UWP should be considered. While back System.IO.MemoryMappedFiles (may be System.IO.Pipes too) support was added by using ...FromApp methods explicitly for UWP, similar approach can be taken for other APIs. This will help dotnet developers bringing their existing projects to UWP without making any significant effort in learning new APIs.
ZodmanPerth
commented
4 years ago Hi folks. I'm the lead developer for the Drawboard PDF UWP app. Our application contains a file explorer that allows the user to pick a root folder using the filepicker and navigate up and down folders within that root. We surface a filtered view of the filesystem; only folders and (mainly) PDF files, for selection for loading.
I'm currently rewriting this section of our app and have discovered the pain of UWP filesystem APIs. I've tried various methods to improve the current experience, and have explored a number of approaches to the problem. At the end of the day, the only stable method is to continue to use GetItemsAsync which is the slowest file access I've ever seen (so I agree with others about performance, I average ~4 seconds for a folder of 1000 items, ~45 seconds for 10000 items).
The UWP APIs are terribly complicated and unfriendly. After spending a lot of time with them and believe I understand how they are intended to be used, I have still failed to get the features promising better performance to work. I don't believe they actually work at all. Further, I don't believe the exposing the file indexer is worth the maintenance cost for Microsoft; it's probably better the Reunion API monitor and decide whether or not it should use the indexer or not as an internal implementation detail.
In the end I've had to create my own folder wrapper class that performs two functions:
Changed event for changes to the folder. This kicks off a background process to re-gather all (filtered) items in the folder using GetItemsAsync, then compare them to the cached list of items, to determine what the changes are. I surface the results through my own event using a lightweight item class. I use locks to ensure I only queue one of these processes at a time, no matter how many times the Changed fires.This allows me to wrap up all the logic I need to process folders at the snail's pace the API provides, while keeping the UI somewhat responsive.
ASIDE: I'm working on a standalone prototype, which I am planning to share publicly when I've finished my investigation. Please let me know it would be helpful for this discussion and I'll post it here when it's done.
I've tried using FindFirstFileExFromApp which is blindingly fast in comparison, and more than acceptable for my needs. However it fails with Win32Exceptions on some folders (C:\ for example), meaning it isn't robust enough to rely on. I have abandoned it as an option.
What I expect from from a v1 Reunion API for filesystem access is:
id+name+type of files/folders within a specified folder the UWP app has rights to, given filter options. I don't care if it's an enumerator, collection, or stream of simple objects, I just want it to be very fast. It should be asynchronous.callback/event to notify me when the contents of a folder has changed, given filter options. The args should include the id+name+type of files/folders that have been added/renamed/removed, and grouped as such. It could also provide some id to pass to an async function to get results and allow the callback/event to complete.id of a file/folder to obtain more details. This could be file properties, image data etc.I feel that would cover the needs of many UWP apps; perhaps many non-UWP apps as well.
Having spent significant time analysing the UWP tools for working with the containerised filesystem model (StorageItem, FutureAccessList, AppData folders etc) I have to say I think the model has a lot of merit. The core issue seems to be that engineers have trouble transitioning from a culture where they can access anything in the filesystem to the containerised model. It's not an intuitive leap, and despite numerous articles and documentation I think Microsoft have failed to onboard engineers into right mindset to work with the containerised model.
I don't have an opinion on BroadFileAccess as at present; our app simply hasn't had the need for it.
I agree with what others have said about the upper limit of the FutureAccessList being too small. It probably made sense when UWP was in a much smaller container and was only capable of producing simple apps. If we want to allow UWP the capability to create serious tools then performant filesystem access is key, whether it's using containerised permission or broad access.
With WinUI 3 on the verge of being delivered it may be that engineers will gravitate towards Win32/Winforms/WPF apps using WinUI 3, and UWP would remain only able to successfully compete with a subset of apps.
I feel resolving filesystem access with a Reunion API will mostly flatten the playfield of what can be achieved on UWP and give it the competitive support it needs and deserves.
brabebhin
commented
4 years ago Having spent significant time analysing the UWP tools for working with the containerised filesystem model (StorageItem, FutureAccessList, AppData folders etc) I have to say I think the model has a lot of merit. The core issue seems to be that engineers have trouble transitioning from a culture where they can access anything in the filesystem to the containerised model. It's not an intuitive leap, and despite numerous articles and documentation I think Microsoft have failed to onboard engineers into right mindset to work with the containerised model.
The container itself is fine. The fact that it takes 100+ms to open a file on a SSD is not (while arguably using native code, when a managed application can achieve 100 times better performance). Other OSes managed to get containers and file access at a decent speed, yet somehow UWP failed to do that.
The StorageFile idea has a lot of merit, especially since it can also abstract away a lot of problems (like files on network drives or streamed files), but this has come at the expense of performance. Microsoft needs to address the performance problem, otherwise nobody will bother with the container.
With WinUI 3 on the verge of being delivered it may be that engineers will gravitate towards Win32/Winforms/WPF apps using WinUI 3, and UWP would remain only able to successfully compete with a subset of apps.
I personally plan to migrate my UWP app to a desktop app + winUI just to get decent file performance and implement some features which are simply not feasible with the current performance of the Storage API. I can continue using the Window.Storage API for most of the application, but performance critical sections need better APIs
ZodmanPerth
commented
4 years ago I've completed my prototype and have added it to a public repository at uwpfileexplorer. It serves as a case study into what UWP engineers have to do to achieve performant file access in the context of a file explorer.
The performance is good enough for an in-app file explorer, but not good enough for filesystem-crawling file processing utilities. The code includes a PowerShell script to create large folders that can be used to highlight performance issues with the API. In particular, note how long it takes to refresh a large folder when changes are made to the folder outside of the app.
It also highlights the inadequacies of the UWP filesystem APIs, and how much work needs to be done to wrap them into something useful.
shresthasource
commented
4 years ago To my experience, UWP apps also do not get access to various items such as URL shortcuts or App shortcuts, and hidden files/folders via StorageSpace API. Access to such items would help.
DrusTheAxe
commented
4 years ago @shresthasource > UWP apps also do not get access to various items such as URL shortcuts or App shortcuts
What sort of access are you interested in?
What
Who
Where
There's gaps between shortcuts and tiles. Some of these are old features that may not make much sense these days (Run as Windows95 etc compatibility options?) and some are functional e.g. Secondary Tiles have no RunAsAdministrator property. Do you have specific gaps you'd find helpful?
Is it the functional behavior you're interested in or the physical shortcut itself i.e. a .LNK file?
There's also some gaps between 'primary' and secondary tiles. The latter are created/manipulated via runtime APIs, the former are automagically created when a package containing <Application>s is registered for a user and deleted when the package is removed, an <Application> only gets 1 primary tile e.g. you can't create a tile to launch an app and another tile with uap10:Parameters="--safemode" etc to create a similar tile with different arguments.
Are there particular gaps you're interested in?
On the primary/secondary tile front, would it be helpful if you could specify tiles in appxmanifest.xml e.g.
<Application Id="App" Executeable="catvideos.exe" ...>
...
<Extensions>
<Extension Category="windows.tile" uap10:Parameters="--type=kittens" uap10:CurrentDirectoryPath="$(knownfolders.videolibrary)\Cats">
<Tile Logo="kitty.png">to produce 2 tiles. The 'primary' today for the <Application> runs "...pkgdir...catvideos.exe" with currentdirectory=pkgdir, and a 2nd tile that runs "...pkgdir...\catvideos.exce --type=kittens" with currentdirectory=c:\users\username\videos\Cats
shresthasource
commented
4 years ago This should clarify most of the answers:
Use Case: A user browses to the Desktop Folder using a UWP file manager and do not see the Application shortcuts that launches Applications such as Edge Browser. A URL shortcut is created from Context Menu -> New -> Shortcut and puts a URL or other paths.
Issue: A user does not see such shortcuts in a folder and might end up deleing the folder thinking it's empty. But some URL shortcuts or other such items might have useful/important contents. A user blames developer for a file manager that did not show items that were there.
What sort of access are you interested in?
What
- create shortcuts
- update shortcuts
- delete shortcuts
- other?
All are relevant/useful. I tend to think the StorageSpace API does offer something to create a link/shortcut to a different folder (just like copy/cut/paste, there was something along the line of link, haven't tried it though). At very least such links/shortcuts should be visible. Best case scenario, they should have similar methods as other storageitems. Also, I am not sure if storage item type of neither folder nor file is actually implemented or usable for anything as of now.
Who
- shortcuts in user's profile
- shortcuts for all users
- other?
Again, I am looking mostly at from a location where shortcuts are rather than build in shortcuts to something. However, another issue I had is accessing various folders via StorageSpace. As of now, we can access Music, Videos, Documents, and Pictures. However, places such as Desktop and Downloads are not accessible from the API. So, best a developer can do is to try to do some magic with user name and generate a default path (e.g. C:\Users\username). But, such workaround wouldn't work for all the cases.
Where
- shortcuts on Start Menu
- shortcuts on desktop
- shortcuts on tray
- other?
Shortcut items wherever they are located (in StorageSpace). Desktop definitely. StartMenu maybe. Tray, probably not important unless tray shortcuts are also stored somewhere. Others -> Wherever user decides to create shortcuts to files/folders/applications.
There's gaps between shortcuts and tiles. Some of these are old features that may not make much sense these days (Run as Windows95 etc compatibility options?) and some are functional e.g. Secondary Tiles have no RunAsAdministrator property. Do you have specific gaps you'd find helpful?
Probably not much relevant answer in my context here.
Is it the functional behavior you're interested in or the physical shortcut itself i.e. a .LNK file?
Not entirely sure about the question, but I tend to think I am talking about physical shortcuts.
benstevens48
commented
4 years ago @DrusTheAxe I think at a minimum the physical shortcut itself (.lnk file) should be listed when listing the items in its parent folder. I suppose this could be optional but I don't see a reason not to include it. Maybe there could be an option to also resolve shortcuts when listing files in a folder, but this wouldn't work if just file names were returned and I wouldn't want it to impact too much on performance, so I don't think this is essential in a low-level API. There also needs to be a way to get the target file path of the shortcut. Should permission to access the target file automatically be granted? I'm not sure, but it would be difficult to do anything useful if not.
Qws
commented
4 years ago People don't want apps to dump files on random system locations behind the scene.
Only Devs think it's nice to dump their files wherever they want, it's a selfish use case and cause horrible Windows user experience (maybe not immediately, but it will in the long run).
Many people find Windows a confusing system, compared to iOS and even Android. Downloading and installing legacy Win32 app is even a more scarier and confusing process, but on iOS and even Android it's a fun process.
The reason why UWP is considered a fail? Let's just look how you get UWP apps installed on the PC in the first place.
The Windows Store itself a horrible app, look at the Windows Store review, people calling out how terrible the app is. The Windows Store is slow, freezing, horribly designed has boring UI for years. The whole Store needs to be revamped. It feels like it's a sluggish Surface advertisement board + some games.
Please don't blame UWP when the place where you get UWP apps is barely functional.
Another problem is, building an UWP app is still confusing, you don't need to know XAML and C#. Let people know how to make UWP apps with other tech stack, and further broaden the ways to build UWP apps.
UWP has limitation, sure, some of them are problematic, sure. But the file access is not one of them, it's one of the better limitations. Once you break this limitation, it will be abused... then you get back to the old Win32 legacy chaotic software and worst case literally spyware/malware. You just end up walking circles.
Keep UWP tightly locked and safe for the users to use... broaden the ways to build UWP apps, and important one, fix the store.
jtorjo
commented
4 years ago UWP has limitation, sure, some of them are problematic, sure. But the file access is not one of them, it's one of the better limitations. Once you break this limitation, it will be abused... then you get back to the old Win32 legacy chaotic software and worst case literally spyware/malware. You just end up walking circles.
Bro, did you just try to find a place to vent? You clearly have no idea what it's been talked about here. You clearly haven't used UWP storage file/folder API, so please do some tests before you open your mouth.
Qws
commented
4 years ago Bro, did you just try to find a place to vent? You clearly have no idea what it's been talked about here. You clearly haven't used UWP storage file/folder API, so please do some tests before you open your mouth.
What do I need to know about Windows.Storage? You should be more detailed instead of encouraging people to be quiet with just baseless assumptions.
I'm responding on the demand (by few devs) where they want to store Sqlite database (and other app related files) in user's PERSONAL documents or other places where user will never know of, doesn't know what to do with it and most importantly doesn't get properly deleted.
Seriously, no normal user wants to have an Sqlite database in their personal folders. You're literally just spreading garbage on their harddisk if you do that. It's unwanted. Like I said, just selfish dev usecases, which leads to worse user experience for Windows.
If I delete an app, everything related to the app and has no purpose for the user should be deleted, sadly, lots of time that just doesn't get happened with legacy Win32 software and its chaotic nature.
That's why all the app files should be stored in the ApplicationData.Current.LocalFolder, and if the user dislikes the app, they just uninstall it, and nothing of the app should be left on the PC, unless the user explicitly allows it. But that's how simple it should work.
And yes, I have used Windows.Storage apis like FileIO, StorageFolder, CreateFileAsync, etc... don't even know why that's even relevant to the discussion, but oke.
lukeblevins
commented
4 years ago @Qws That's a very legitimate concern. Luckily, low-trust UWP apps require a capability to access these locations at all. Users can already disable access to the Documents, Downloads, etc in Windows Settings under privacy.
Remember, we all want these improved APIs to provide a quality experience for users, so let's remain civil when stating our perspectives.
jtorjo
commented
4 years ago What do I need to know about Windows.Storage? You should be more detailed instead of encouraging people to be quiet with just baseless assumptions.
Well, a lot - if you're to discuss stuff here - please take a look at the title of this issue, for starters.
I'm certainly not discouraging people from venting about UWP - I've done it quite a bit myself. But this is about File-access APIs, and there are quite a lot of people that want improvements on their speed, myself included.
shresthasource
commented
4 years ago People don't want apps to dump files on random system locations behind the scene. Totally agreed. I do not want apps to create files in random places, and I myself do not like seeing apps creating files everywhere.
In my response specifically, I am considering a file manager scenario. If your app is a file manager, you cannot say do not create a file here and there or do not show this and that file (well, okay, system directories should be read only type, that's fine).
shresthasource
commented
4 years ago If I delete an app, everything related to the app and has no purpose for the user should be deleted, sadly, lots of time that just doesn't get happened with legacy Win32 software and its chaotic nature.
Would you want to have all your MS word documents deleted because you uninstalled MS Office Package? all PDFs because you uninstalled PDF reader? Images if you uninstalled a specific picture viewer?
So, while your concern in legit, there are scenarios where app should be allowed to read/write/store files is different locations - specifically with user's permission/knowledge of some sort.
shresthasource
commented
4 years ago And yes, I have used Windows.Storage apis like FileIO, StorageFolder, CreateFileAsync, etc... don't even know why that's even relevant to the discussion, but oke.
The whole discussion is titled "Proposal: Improved file-access APIs for UWP and AppContainer." And StorageSpace API is, at very least, one of the main API related to file access. So, not sure why you would think the discussion about StorageFolder.CreateFileAsync, etc. is irrelevant.
brabebhin
commented
4 years ago @Qws,
The main problem here is the read-only APIs. Most applications need to be able to read the file system in order to do useful things. File manager, photo managers and editors, video and media players, any productively app for that matter needs file system access to work properly. Just image visual studio using Windows.Storage api to do file access. It would essentially kill windows development, that's how bad UWP file access performance is.
You can't compare iOS and Android to windows. Windows produces things that android and ios consume. If your computer use case is facebook, you shouldn't be here.
MarkIngramUK
commented
4 years ago You can't compare iOS and Android to windows. Windows produces things that android and ios consume. If your computer use case is facebook, you shouldn't be here.
I'm not sure why iOS is being thrown in here, but they have a secure sandbox, and sensible APIs (any file the user chooses can be opened by any valid API, e.g. fopen). And it's fast.
michael-hawker
commented
4 years ago I think there's been two points that have been getting mixed here.
One is the fact that the current UWP APIs for file access or slow compared to Win32 equivalents. This just makes them unpleasant to use as a developer and effects end-user perception as well.
Second, there are legitimate cases where you don't want to be in a sandbox and be able to read a general file that's referenced in data you have from a file the user has already permitted you access to (like a link to an image in some other folder that's referenced in the current file).
I know I find the sandbox a great environment due to its protection on user data; however, it's performance needs to match expectations, and there do needs to be a gradient of permissions available for different types of scenarios.
ZodmanPerth
commented
4 years ago I agree.
In my case at least, I do not mind the way UWP file access is restricted (though helping developers adopt the mindset could use better support from Microsoft).
My issues with the UWP APIs are:
I'll repeat my expectations (from my original post) to reiterate how ProjectReunion can help address this aspect of the problem:
What I expect from from a v1 Reunion API for filesystem access is:
- The ability to get the id+name+type of files/folders within a specified folder the UWP app has rights to, given filter options. I don't care if it's an enumerator, collection, or stream of simple objects, I just want it to be very fast. It should be asynchronous.
- A callback/event to notify me when the contents of a folder has changed, given filter options. The args should include the id+name+type of files/folders that have been added/renamed/removed, and grouped as such. It could also provide some id to pass to an async function to get results and allow the callback/event to complete.
- A function (or functions) to call with the id of a file/folder to obtain more details. This could be file properties, image data etc.
We need simple, fast access for simple use cases. We know this is possible using ...FromApp functions, but there are holes needing to be addressed. ProjectReunion has the opportunity to manage all this and expose a fast-access facade/API that will solve for these use cases.
ZodmanPerth
commented
4 years ago I've posted a 2 minute video to YouTube showing the speed at which the UWP API performs for large folders. https://youtu.be/40Nmy3Oon78
There are 3 stages in the video:
The app returns results in chunks of 50 to keep the UI usable. If you mark the time between selecting the folder in the app (or modifying the folder in Windows Explorer) and the time when the count finishes, almost the entire time between marks is the speed of the StorageItemQueryResult.GetItemsAsync().
Note that the poor performance demonstrated is entirely from the UWP API itself. Adding items to the UI is very fast and takes an insignificant amount of time.
Observe how the time for a full scan goes up by a linear factor as the number of items increases.
Also observe that to keep the UI in sync with the actual folder, you have to complete a full re-scan of the folder because the API doesn't report changes. Coupled with the performance of the scan, this compounds issues with performance for the app.
I hope this shows the core issue with performance of the UWP API, why it isn't good enough, and why I hope ProjectReunion can resolve the issue gracefully for us.
brabebhin
commented
4 years ago @ZodmanPerth That link doesn't work. Says video is private.
ZodmanPerth
commented
4 years ago I double-checked; it's definitely marked as public. Maybe its a timing thing?
brabebhin
commented
4 years ago Your video shows essentially the best case scenario. If you want to get 10000 individual files by path, things get far worse, especially when certain files are missing.
ZodmanPerth
commented
4 years ago Yes that was the point. This is as fast as things can get, and it's still very slow.
ptorr-msft
commented
4 years ago Do you believe Windows needs to add net-new APIs for the features you're describing, or do existing APIs (Win32 or .NET or C++/STL etc.) cover your use-cases, assuming you could use them from a UWP process?
eklipse2k8
commented
4 years ago @ptorr-msft If i could create a sqlite3 database where ever the user wants, for example something that works with Win32 CreateFile2 API (or c++ fstream, or c fopen) without the need to use StorageFile APIs is all I would need.
Right now the limitation with CreateFile2 only able to create files inside the appdata folder that then gets deleted on app uninstall makes it pretty frustrating to use.
brabebhin
commented
4 years ago Hello @ptorr-msft
The way I see it, UWP should be able to use any win32/.net/C++ API in the folders & files it already has access to, and this should also work with the broadFileSystemAccess permission.
in a v2 you could have different permissions for read-only and read-write access broad filesystem access. The read-only should be enough for most scenarios.
benstevens48
commented
4 years ago @ptorr-msft - I don't think we need any new low-level APIs for the time being. Just being able to use the existing APIs from a UWP process wouldn't really be sufficient however - we need Project Reunion to write some wrappers for us and help integrate them with the other UWP APIs, otherwise the effort required is too much for the developer. (The .Net APIs are generally quite user-friendly, but we would still need to integrate with other UWP APIs that use IStorageItem like clipboard etc, and also .Net is missing the equivalent of RetrievePropertiesAsync I think).
Also, in terms of the permissions, as I described in my earlier long post, I think we need some adjustments or more of a permissions gradient with additional capabilities than the current situation.
Proposal: Improved file-access APIs for UWP and AppContainer
Summary
Provide an updated version of
StorageFilethat fixes performance and usability issues in the currentWindows.Storage.Storage*set of types. Support apps that think in terms of file paths & handles instead of StorageFile objects.Rationale
The current
Windows.Storage.Storage*types are designed to give programmatic access to regular files, shell namespace items, and other Windows entities backed by files. A single abstraction over all those spaces is interesting but comes with high costs - namely performance and impedence-mismatch with existing application code. Project Reunion should clearly explain why StorageFile is important and - where possible - provide lower-level access to the user's files.BroadFileAccesscapability is "scary" to customers but required even for limited UWP scenarios....FromAppAPI variants, but higher-level code (JS, C#) interop for them is very limited.Scope
...FromAppplatform APIsIStream/ISequentialStreamAPIs for these filesW.S.Storage*and types in this functionalityOpen Questions
Should UWPs be given access to the
IShell*family of APIs?As these APIs are not truly universal on all editions of Windows, if an app were to program against them directly the app would not work on editions like HoloLens or Xbox.
Should Reunion warn the user about apps using this capability?
Some platforms periodically remind users that "App X is using feature Y, are you OK with that?" How should Reunion make the user aware that a "low rights" application is using capabilities?