Add Support for Configurable Future Epoch Stamp for Generating Decreasing KSUIDs

kamaz commented 7 months ago

Background

Currently, the ksuid library generates KSUIDs that are naturally ordered by generation time. While this ordering is suitable for many use cases, there are scenarios where it would be advantageous to generate KSUIDs in decreasing order, with the latest keys appearing first. This is particularly valuable in situations where managing a large volume of data is involved, and there's a significant performance penalty for using databases that do not support descending indexes or where the order by operation is resource-intensive. By enabling the generation of decreasing KSUIDs, users can optimise database operations and enhance overall system performance, especially in scenarios with substantial data processing requirements.

Proposal

I propose extending the ksuid library to add support for configuring a future epoch stamp, which would result in generating decreasing KSUIDs based on generation time. By allowing users to specify a future epoch stamp or just being hardcoded to e.g. 2160, the library would subtract this value from the current Unix timestamp during KSUID generation, effectively producing KSUIDs that decrease over time.

Benefits

Allows users to generate KSUIDs that are naturally ordered in decreasing order based on generation time.
Enhances the flexibility and usability of the ksuid library for a wider range of use cases. Additional Considerations:

I would be delighted to contribute the functionality for generating KSUIDs in decreasing order to the library. Currently, to achieve this, we would need to maintain a fork of the library. Adding this feature directly to the main library would hopefully help a few folks out there.

Would you be interested in incorporating this feature into the library?

achille-roussel commented 7 months ago

Hello @kamaz, thanks for opening this issue!

I'm not involved in the development of KSUID anymore, but I still have a fair amount of context that I maybe is relevant to this conversation.

I believe the problem you describe can be solved without changing the epoch and introducing new APIs. One way to solve this problem could be to implement a custom KSUID generator in the application and use ksuid.FromParts. With this function, you can decide the timestamp and random part. You can generate a random base and decrement it by a constant factor for each KSUID you generate, effectively implementing the scheme you described.

Something that I believe is important to consider is that KSUID has implementation in multiple languages. One of the strengths of this project has been the simplicity of implementation. KSUIDs are useful, and libraries implementing them are many because the specification is concise. When we add new generation schemes in the KSUID specification -- and more generally, when we increase complexity -- we will inevitably start seeing more and more divergences across languages, making KSUID less portable while increasing the development cost of library maintainers.

For these reasons, I would recommend the next step to showcase the suggested change as an external package that can exist as part of the KSUID ecosystem; then, if there is a desire from multiple languages and projects to see this added to the specification, it would be a strong signal that it's worth the added complexity cost.

I hope this perspective will be useful!

kamaz commented 6 months ago

@achille-roussel thanks for sharing your view and I believe that make sense as you don't want to break specification which is in placed and shared between different libraries.

segmentio / ksuid