Requesting a folder Api for files

[fwanicka]

I think it is critical we have a cross-app way to organize files that will scale properly in the long run. The previously proposed annotation (https://github.com/appdotnet/object-metadata/issues/7) is a good workaround given the current Api, but I don't believe it will scale well at all. The problem is that to know all the possible folders, you have to download every single file entity in advance. This completely negates one of the the big advantages of a folder structure, which is starting at the root and loading file entities on demand as you drill down. I believe we will eventually have users with 100,000+ files. At that point, and well before probably, any file manager type app is going to be virtually unusable.

The other issue with the annotation approach is that it requires every developer to adopt the annotation spec. If the dev chooses another format, or does not implement it correctly, it will be a mess, leading to a bad UX. If App.net is to support unbundling, then there needs to be a spec that works across all apps to move your data around.

So, I am proposing a Folder Api. Each user account would start with a default root folder. Folders other than the root could be added, edited (name, parent folder, etc.) and deleted. Each folder would have a parent_folder_id. Endpoints could return all folders, or folders underneath a given folder Id. An additional property (folder_id) would need to be added to the File entity. When a new file is created, the creating app can optionally specify a folder_id. If none is specified, it will be set to the root folder_id on the server side.

With this, you could have an Instagram type app, and have the files be manageable. A new folder could be added for each day or month based on a user preference, and over time, it would stay nicely organized.

[kosso]

+1 - This is the one reason why I have yet to implement the File API in my systems yet. I have yet to figure out an efficient way of doing this - that won't need to change in the future.

Also, I am poised to fire over a lot of file assets from bli.ms users and I'd rather not clog up everyone's 'root folder' file list with hundreds of files - which I have a feeling will be a rather unpleasant experience to sort through for the user, when it comes to managing their files.

I use a "YYYY/MM/DD/TT:TT - Title" folder structure in another system I build similar to this which uses Dropbox. It makes life a lot easier. Especially as a single post can have 5 or 6 assets attached to it (to provide full viewing support)

Another crucial thing I would like to see is a top level 'app folder' - which contains only the files uploaded by a particular app - though I know I can filter the list by the 'type' property, I feel this is a workaround until a more appropriate method appears.

We need things like this, or 10Gb of file storage is going to get very very hard to manage, very very quickly.

[fwanicka]

The problem with each app filtering the available files by its own FileType is that it directly contradicts the goal of unbundling. The idea is to have all your data accessible by any app at any time potentially. As an App.net user, if I upload a photo from my phone, and put it in my Photos folder, I expect to be able to open my desktop client, and browse to my Photos folder and see that photo and attach it to a post.

If each app comes up with it's own annotation to describe a folder structure and they end up being different, the UX is going to be terrible. We would all have to hard-code each others formats in our code.

[kosso]

There are many uses cases I can think of where this makes sense.

It also highlights a potential permissions layer for file access - eg: Full Acess / App-level access.

I know the App.net File API isn't designed to be like Dropbox, but there are good reasons to have that sort of access control.

I spoke with Dalton yesterday about the idea of a (very) private messaging app I am building, where I will be encrypting the text part of the messages, but I want to ensure that ONLY the dedicated app can access the files that go with it. Without that, it's a no go. But WITH it creates masses of valuable opportunities.

One reason I don't use the private messaging here (and especially file attachments) much on App.net is that any developer/user of any app can access the files created by users of my (private) app. Therefore it is not private any more and not something I can ever attest to. Which is a shame, as what I'm suggesting could solve that very very easily.

[berg]

We have no plans to build an indexed folder-like hierarchy on top of the existing files API. The whole point is that it's not designed to map to a filesystem; it's designed as blob storage for apps. So we suggest the use of file types for organization. In addition, if/when we implement more granular access control, it will almost certainly be based around file types, rather than anything else. Obviously we have no problem getting a folder annotation into the API for recommended use, but most apps likely won't have to take advantage of this (and they shouldn't.)

@kosso you're obviously welcome to encrypt any messages. I suggest that you encrypt file content as well, if total isolation is your goal. Otherwise, it goes against the grain of the API and core principles of the service -- it's not designed to let you lock someone into using only your app to access certain files in their own file storage. As I said, we have discussed internally the possibility of additional access control, but haven't come up with a solution without nightmarish UX (where we can prevent app lock-in and also create something that users will be able to read and understand before accepting permissions.)

[kosso]

got it

On 11 March 2013 17:23, Bryan Berg notifications@github.com wrote:

We have no plans to build an indexed folder-like hierarchy on top of the existing files API. The whole point is that it's not designed to map to a filesystem; it's designed as blob storage for apps. So we suggest the use of file types for organization. In addition, if/when we implement more granular access control, it will almost certainly be based around file types, rather than anything else. Obviously we have no problem getting a folder annotation into the API for recommended use, but most apps likely won't have to take advantage of this (and they shouldn't.)

@kosso https://github.com/kosso you're obviously welcome to encrypt any messages. I suggest that you encrypt file content as well, if total isolation is your goal. Otherwise, it goes against the grain of the API and core principles of the service -- it's not designed to let you lock someone into using only your app to access certain files in their own file storage. As I said, we have discussed internally the possibility of additional access control, but haven't come up with a solution without nightmarish UX (where we can prevent app lock-in and also create something that users will be able to read and understand before accepting permissions.)

— Reply to this email directly or view it on GitHubhttps://github.com/appdotnet/api-spec/issues/310#issuecomment-14727047 .

[fwanicka]

Thanks for the quick reply, even if it's not what I was hoping to hear. Regarding your comment in the Patter room: "the suggested way to deal with this is keeping a local cache of files. we'll possibly add a sync API for you to get incremental updates on files, etc", a sync API would definitely help to mitigate the performance hit of downloading all the file entities every time the app loads.

[berg]

@fwanicka yeah, having some sort of API for getting deltas of metadata makes sense to me. We're a small team, and fairly limited in the number of cycles we have. I think that a delta API is more useful to a wide variety of use cases rather than files/folders being indexed across everything, so it'll likely get done much sooner than building files/folders into the core file API.

[kosso]

delta +1

On 11 March 2013 18:58, Bryan Berg notifications@github.com wrote:

@fwanicka https://github.com/fwanicka yeah, having some sort of API for getting deltas of metadata makes sense to me. We're a small team, and fairly limited in the number of cycles we have. I think that a delta API is more useful to a wide variety of use cases rather than files/folders being indexed across everything, so it'll likely get done much sooner than building files/folders into the core file API.

— Reply to this email directly or view it on GitHubhttps://github.com/appdotnet/api-spec/issues/310#issuecomment-14733416 .

[fwanicka]

Great. That would solve the biggest issue i have, which is dozens or possibly hundreds of API calls every time the app spools up. I also agree it applies to broader set of use cases. That would be a good compromise.

Now, we just need to get the devs to agree to annotations or channels for a pseudo folder structure or tagging system.