Directory Layer v1.1

[KrzysFR]

Placeholder before the full commentary on this huge pull request!

Change required to the Directory Layer, and the subspace APIs in general, to allow safe caching of directory subspaces, even if the underlying metadata is mutated.

The goals that this PR wants to achieve are the following:

• Protect against mutation of the partitions that could change their prefix
• Provide a way to safely "cache" Directory Subspace for Layer or application code.
• Prevent application code from misusing subspaces (ex: storing and reusing subspace instances across transactions)
• Simplify the API and the way to represent a "location" in the keyspace by introducing ISubspaceLocation<TSubspace> which can be "resolved" into the concrete subspace implementation inside a transaction.
• Update the global API to make working with the new DL API "easier".

The recently introduced \xFF/metadataVersion key helps in building a metadata cache that is "safe".

We could use this key to keep a cache for the prefix of the different directories, but:

• This key can be changed by other layers for unrelated reasons, so we need another key, specific to the DL, that can be used to detect "foreign" changes to changes made by the DL itself.
• Any creation/deletion/move of a directory should touch both keys to invalidate the DL cache of other servers
• Partitions should have their own key, in order to "contain" cache invalidation inside each partition, but the prefix of a partition itself can change in its parent partition!

The changes to the DL API have some repercussions on the global API:

• It is not possible to restrict the key prefix visible to a database to a directory partition, since the prefix can change at anytime, so the GlobalSubspace is gone, and replaced with a Root property that makes it easier to build subspace locations.
• All directory paths are now absolute (because the cache need to make sure that all parent partitions are unchanged), and there is only "one" DirectoryLayer instance that handles all the partitions (instead of each partition having its own DL instance).
• All the retry loop handlers must be updated to make it easier to handle resolving subspace locations into subspaces (which is now async).

This is PR is very large, but unfortunately all these changes have to be done together in order to have a working build.

Most of the changes needed to update existing code:

• use db.Root["foo"]["bar"] to easily create a DirectorySubspaceLocation that points to some sub-folder under the root partition
• the root partition is specified in the connection options passed to Fdb.OpenAsync, like before, exact this is a DirectoryPath instead of a KeySubspace
• remove all cases of caching a FdbDirectorySubspace instance in some init codepath, and instead replace that singleton with the corresponding DirectorySubspaceLocation.
• all transactions that need to use this subspace will need to call this.Location.Resolve(tr) instead. Avoid resolving the same location multiple times in the same transaction, even though the instance is cached per transaction.
• instead of opening/creating/deleted directory subspaces via the old db.Directory property (which is gone), prefer using the methods on FdbDirectorySubspaceLocation, i.e: await location.CreateOrOpenAsync(tr). Using the db.DirectoryLayer is still possible but should only be done with absolute paths, and only in layers that have to handle very specific scenarios. Most app code will be fine using subspace locations.
• avoid calling TryOpenCachedAsync(...) on the DirectoryLayer, and prefer using location.Resolve(...) which does the same thing, and is not restricted to directory subspaces only.
• review all uses of ReadWriteAsync(...) that would only return a Task and make sure to either change them to WriteAsync(...) or make them result a Task<TResult>. This is because the compiler messes up the overload resolution, and will invoke a version that returns a Task<Task> instead, and so the loop handler body will not be properly awaited!!!
• review all instances of ReadAsync(tr => tr.GetAsync(this.Subspace.Encode(...))) because they have to converted to async tr => { var subspace = await this.Location.Resolve(tr); return await tr.GetAsync(subspace.Encode(...)); }. It is very easy to forger to add await in the return await tr... and end up with a Task<Task<Slice>> instead of a Task<Slice> !

The most common issue is that there is a lot of code that would run in a non-async handler to Set or Clear some keys, and would used a globally cached Directory subspace instance. This code cannot work anymore, and has to be converted to async in order to call Resolve(..) on the subspace location.

[KrzysFR]

I know that there are still probably some bugs and changes needed to the DL API, but this PR is so big already and require other changes that are not directly related to the DL. We will have to deal with it with other PRs.