We need the key naming fn to randomize the key prefixes to ensure proper s3 sharding. This could be done by generating a timestamp and reversing the end of the timestamp to ensure randomness.
The input plugin would then need to be modified to make this more pleasing to use, and ensure order is maintained.
This would break prefixing, as it depends on keys sharing a prefix. This could be worked around by listing all keys and filtering from onyx, rather than having amazon's servers pre filter. If we go this route we should have a metric for the time it takes to filter the keys.
We need the key naming fn to randomize the key prefixes to ensure proper s3 sharding. This could be done by generating a timestamp and reversing the end of the timestamp to ensure randomness.
The input plugin would then need to be modified to make this more pleasing to use, and ensure order is maintained.
This would break prefixing, as it depends on keys sharing a prefix. This could be worked around by listing all keys and filtering from onyx, rather than having amazon's servers pre filter. If we go this route we should have a metric for the time it takes to filter the keys.