pnarayanan
commented
7 years ago All good points, @siepkes. A key-value API for Ambry is certainly useful and this is something that has come up before.
Self generation of handles keeps things simple internally, particularly around preventing key collisions. That said, although the ids or "handles" are generated at the router today, the datanodes to which they are sent have a key-value API. It should not be too hard to provide an API for letting the client pass in an id/key/filename for the object that is uploaded. The main question to answer is how key collisions should be handled or prevented. Basically, what happens if two requests come in to upload objects with the same key and they go to different nodes and are close enough in time. Since only a few datanodes are contacted in the synchronous path, both requests could succeed and the duplication might only get detected during the async replication, by when it would be too late to "correctly" resolve it. There is also the question of whether the system should continue to be an immutable store - whether the key-value PUT API should support updates of an existing key. Once we go down this path, we have fundamental problems to deal with, such as handling versioning for a key and figuring out which one is the latest and so on, those that other general purpose key-value stores deal with.
Perhaps the simplest way to achieve this is to continue to be immutable and put the burden of uniqueness on the client itself. You bring up an interesting point about creating a namespace around the service owner itself. If the service owner field is embedded into the id (which can be easily done by the router), then the service owner will only have to ensure uniqueness of keys that it generates.
siepkes
vgkholla
kkimdev
yhilem
w39hh
zzmao
I was looking in to the possibility of using Ambry as a generic cloud storage solution, basically an S3 replacement. Being a dev myself I'm not afraid to get my hands dirty to implement storage implementations with Ambry's API for various applications like Phabricator, Discourse, XWiki, Open-Xchange, etc.
However I bumped into the issue that all applications I looked into assume that they can control the naming schema of the blob they are storing. Now obviously that is not something that is un-fixable; All of these solutions are opensource and could be made compatible with storage backends which don't let them choose their own "filenames" (blob names would probably be more appropriate). However that would mean getting API changes in to all kinds of upstream projects, which is usually a lot more work then just providing a new storage implementation class (all the projects I encountered already have an abstract concept of storage).
So I was wondering how realistic it would be to implement something in Ambry that would for example create a namespace per blob owner for unique blob names.
An alternative would be to create a totally separate database or gateway in front of Ambry in which a sort of virtual namespace is created. However that would create an additional database to backup, maintain, etc. It would be neat if that data could somehow be stored in Ambry itself to ease the maintenance burden of Amrby.