File API

[McCrea]

APIPermissions, I don't like it. These days it does more than just permissions and that isn't really right.

I propose it be split into three APIs:

• User API
• Files API
• Permissions API

They might use some common PAL components for actual storage, but the methods for reading/writing should probably be separated out properly. This would also be a good opportunity to enhance how we use the NDB datastore in the GAE PAL; I propose the following:

• Use keys-only queries - I've already changed PALPermissions to use these where possible. With some re-design it may be possible to replace more queries with keys-only operations. They're free (or cheaper) and faster as they only involve an index lookup, so it's worth using them where possible.
• Use ancestors for files - we currently store a parent key for files, we should do this properly and then make use of the ancestor queries where useful - this can allow us to guarantee no stale results in certain circumstances
• Investigate transactions - At the API layer, some operations may involve multiple read/write ops, these should be grouped into transactions if possible.

I think it would be prudent to start with the files API in order to facilitate the development of efficient hierarchical gate display. I believe there's already a branch created by @Ahemagain for the users API, so that just leaves the permissions API. Once the APIs are done, we'll need to build another utility layer on top of the APIs which link them together for things which require interaction with multiple APIs (such as getting all the files a user has permission to see). Thoughts?

[notchris1]

I agree with all of these, the split seems logical, the users api I creates pertains to the authentication cleanup (users PAL) to improve consistency before major rewrites, however this has not yet started, so I could migrate the purpose of this, i would be Interested in being involved In this section

[notchris1]

Also see #167

[McCrea]

I think the users API should stay separate to authentication, but the API (and underlying storage) need some work in order to facilitate multiple authentication methods. If we're doing an API overhaul, might as well fit that in.

[notchris1]

I agree it should be seperate, but at the moment the authentication is known as users PAL, hence the users API. Users is adopted from googles naming of authentication. I am not suggesting combining them.

[McCrea]

Ah, okay. Perhaps we should call it the 'Authentication PAL' - we needn't be tied to Google's naming conventions. The way I see it, such a PAL would actually contain little-to-no authentication logic - it'd just be responsible for loading other libraries which would actually contain the auth logic and proxying the results to into user data storage calls. That way we could support multiple authentication methods.

[notchris1]

I agree, perhaps spin up a new thread.

[McCrea]

I've been working on this when I've had spare time (read 'infrequently') though haven't pushed anything yet since it's all sitting in a spec-only or partially-implemented state. I'll do my best to push my proposed API spec in some form in the near future.

The approach I am taking involves the following:

• A files PAL which deals with single file operations:
  • a single 'put' operation which stores a file information object - the API layer will be responsible for checking for overwriting etc.
  • methods for getting a file or multiple files by key, name, child or parent
  • possibly one or more 'update' methods
• A files API which deals with more complex file operations which require multiple primitive operations - any primitive operations provided by the PAL which are useful to external code are provided as a wrapper
• A permissions PAL/API pairing
  • This package has no awareness of what a file or a user looks like, it just stores an gets keys for them to establish user/file relationships.
• A utility package for file access with permissions awareness
  • This exists in the API layer, it performs operations which require use of both the permissions API and the files API (also possibly the users API)
  • This performs operations such as getting a list of files belonging to a certain user
• The FileTree class
  • This will build on the aforementioned utility package; it will likely be the method of file access used by UI centred functionality
  • This class creates a tree rooted with a given file or at a user's root. The tree will pre-load a given number of levels of descendants (e.g. immediate children only, children of children etc.) Further children are fetched for branches as required using lazy loading.
• A set of platform independent classes representing
  • File information
  • Permissions
  • Users
• A helpers package (this is based on GAE's PAL, and may require additional consideration in future)
  • Iterator methods - as GAE's NDB datastore returns an iterator to implementation-specific objects this will allow wrapping of a GAE-returned iterator such that objects retrieved from it are returned as their platform-independent counterpart
  • object abstraction methods - to convert the platform/implementation specific objects (e.g. file information) and return them as instances of their respective platform-independent classes

So far I've done the following:

• The FileTree class is implemented
• The file information PAL is defined
• The file/permissions utility package is defined and partially implemented
• The permissions PAL is largely defined
• The file information API can largely be derived from the functions offered by the PAL
• The platform independent classes are largely defined

[notchris1]

This sounds perfect, is there any documentation of specifics at this stage?

[McCrea]

Short answer: not yet Slightly longer answer: I've written stubs for a fair bit of the functionality so that I can work on the higher layers without implementing the lower layers first. I'll try to document those stubs and push them as a priority.

[notchris1]

Excellent, it will be nice to generate some progress on ksf. It also is a good jumping off point for larger changes.

On 16 Jul 2014, at 21:10, Jack McCrea notifications@github.com wrote:

Short answer: not yet Slightly longer answer: I've written stubs for a fair bit of the functionality so that I can work on the higher layers without implementing the lower layers first. I'll try to document those stubs and push them as a priority.

— Reply to this email directly or view it on GitHub.

[McCrea]

I agree. I'm working on these changes so that it will be easy to work on a hierarchical gating UI. I've got ideas for a lazy-preloading panel which makes it appear that the client has grabbed information about every file a user can access when in reality it's only grabbed a small number. But that's a story for another thread.