Feature request for bulk generation of IDs

[codemedic]

Use Case

In cases where a process is constantly processing items, for speed of processing and for database efficiency, it is common practice to batch together records into a bulk insert or group them under one transaction. Usually this improves INSERT query performance due to optimal table locking and fewer index rebuilds; of course it varies from database to database.

In these cases, an ID will have to generate for each record in a batch. In a classic scenario these IDs would be generated by the database itself using sequence or auto-increments. In a distributed environment snowflake IDs would do better.

Proposal

Since snowflake IDs, with the default config, are designed to generate only up to 4096 IDs per millisecond, we need to look at options to increase that while staying within the rules. This can be achieved if the Node can be allowed to overflow slightly allowing to generate IDs "in the future". Any chance of a run-away overflow can be checked using a max-overflow limit.

[bwmarrin]

I'm mixed on this - I'm not sure snowflake is really the right tool if you intend to generate IDs in this fashion. There's plenty of other ID types that can be used that would probably make more sense in a usecase where you wish to pre-generate more than 4096 at a time. Though you could still generate 10's of thousands within a few ms with the library as it is currently.

I'm open to seeing a very clean and simple solution though and other user feedback on rather this is something needed/wanted by the wider user base.

[DavidArchibald]

For my use case I want a relatively small batch but wanted to avoid the mutex lock. I see that this issue was closed as completed but I don't see any way to create a batch in the current version.