Why a 1TB limit?

[markw-]

Hi,

This might not be the right place to ask but there isn't any other way I know of to ask - why is the value size limited to 1MiB? That with the key size limit gives a max storage size of 1TiB and the developer drives (and presumably the drives that will be sold first) are 4TB. Not to mention the fact that most objects in the real world will be way above 1MiB.

If there is a usable forum where I can ask I would love to know where that is.

Sorry for the "off topic" "issue".

Cheers Mark

[jphughes]

Hello Mark:

This is not off topic at all. Thanks for bringing this up.

What "Key Size Limit" are you talking about? The key size is 4KBytes. That is more than enough key space for nearly infinite amount of storage. Can you explain?

Regarding the 1MB limit on the values, this was put into the system to make handling large object easier. That is, if you have a large Object (like a 2.6GB movie) called "Alien". You could create keys like "Alien,00001" for the first megabyte and "Alien,02600" for the last megabyte. For instance, this would allow the application to stripe the data. It allows the drive not to allocate a huge chunk of data and eliminated a large number of corner cases.

In the systems we have looked at the "chunking" at 1MB is not a challenge, and in most cases eliminates a lot of complexity in the system.

[markw-]

Hi Jim,

Sorry - I have been a blockhead. I worked out 256 per byte and multiplied that by 4096 instead of 2^32768. In my defence, I recently aquired a daughter and I'm not sleeping much :-)

I wanted to work out how much data a Kinetic namespace could hold after I read about the distributed namespace. To get high storage speed alot of drives are needed and if each has their own namespace it could get complicated to manage object naming, so a distributed namespace is a really good idea.

Now I understand the 1 MiB chunking and it makes alot of sense, spreading data across alot of drives using the distributed namespace. But it will make alot more objects, I guess this is where things like Scality come in to allow finding objects quickly. This reminds me I still need to look into the metadata options in Kinetic as it doesn't look like it's a traditional metadata model used in current object storage products and object access protocols like CDMI and S3 (user metadata in addition to system metadata). That kind of metadata is probably intended to be in the management system like Scality or Swift too?

One of the things I'm thinking about is how Kinetic will compare with existing storage systems that can do block-level-incremental versioning, compression and system wide block/object deduplication. The last years have seen every storage vendor implement BLI versioning and compression/deduplication to drive down cost per GB. Customers will have reservations about switching to a system needing double the raw disk (rule of thumb of 2:1 dedupe/compression ratio). I'm ignoring the BLI versioning - Kinetic seems more suited to WORM type data (rather than data with a non-zero change rate) so data like entertainment, medical records, seismic, etc.

One thing that seems to be coming out from the Github forums and doco is that the key namespace is being thought of as including information about the data. That looks a bit dangerous to me as it will result in an explosion of customised key naming schemes specific to each application. Given the ease of use of the Kinetic API many people can write their own application code to access Kinetic with their own naming scheme which has to either co-exist with all the other naming schemes in a distributed namespace. The alternative is to have specifc drives allocated to an application in a distributed namespace or have a distributed namespace dedicated to an application. Has this been considered and is there a concept in mind for how to support different key naming schemes?

These are very interesting concepts, I need to think about if/how Kinetc changes the way data will be processed (like with non-volatile main memory).

Cheers Mark