[1d] Support many files in a project, faster pushes and mutable less-limited metadata

[alaric-dotmesh]

As dotscience developers, we'd like Dotmesh to be better at storing commit metadata, so we can continue to use it to easily provide awesome Dotscience features.

In particular, we want:

• Large amounts of metadata per commit
• Good performance when handling commits with lots of metadata.
• Getting away from ZFS' restrictions on metadata property names.

ACs

• [ ] No change to existing Dotmesh APIs.
• [ ] The ability to handle large amounts of commit metadata (let's say 1MiB as a limit for now) without error.
• [ ] Performance of all operations being acceptable when dealing with such large commits.

Implementation notes

• For all new commits, store the metadata as a JSON string:string map in <filesystem root>/dotmesh.metadata/<COMMIT ID>.json
• When reading commits from ZFS, read any properties from ZFS, but override them with any metadata found in the metadata file (if it exists) for that commit in the latest state of that filesystem, so old ZFS metadata still works but can be overridden.

[alice-sowerby]

@alaric-dotmesh @lukemarsden I can't prioritise this unless I know what the awesome features will be and whether we need them.

[Godley]

Going to handle this in stages in order to get a faster release out ready for tomorrow:

• [x] Commit any metadata to a json file which is bigger than 1024 bytes. This should allow us to keep collecting metadata no matter how many files there are, but also only use this in cases where this is necessary as this step does not include being able to read back that data.
• [x] Read back that metadata on filesystem discovery. This requires that each filesystem be mounted, read then unmounted
• [x] rip out the safety net and make ALL metadata go to a file not using zfs props. This should make metadata reads slightly faster