Reproducible databases (option for no server timestamp in hash)

[MikeBeller]

Please add the ability to run immudb, or an immudb database, in "reproducible" mode -- where the server commit timestamp is not included in the hash.

Use case: I am feeding my database from Kafka for example. With databases other than Immudb, if I need to reload, or if I want to run parallel database replicas, I can just load them independently from the Kafka queue and the databases will be identical from all important perspectives. Not the case with Immudb, as the series of hashes will not be reproducible. The timestamps will be different.

If Immudb had a "don't put your timestamps in this database" option, then this would solve my problem. If that idea doesn't work because Immudb needs timestamps for some reason, then if I could supply the timestamp through the API, that would probably be ok too. ( I would just make it part of the message persisted in the Kafka queue and would feed it into the API when I reload.)

[jeroiraz]

Thanks for creating this enhancement request @MikeBeller

[jeroiraz]

Hi @MikeBeller, we've a PR associated to this feature request. Would you like building immudb with the associated branch?

User created databases either using immuadmin or a sdk would be reproducible by setting exclude-commit-time flag or ExcludeCommitTime attribute.

[MikeBeller]

Great! I was able to build the reproducible_stores branch, start immudb, create two databases, and put the same k/v pairs in them in the same order -- got the same hashes. Thanks for this! Looking forward to seeing it merged.

[victornor]

Hi @MikeBeller, we've a PR associated to this feature request. Would you like building immudb with the associated branch?

User created databases either using immuadmin or a sdk would be reproducible by setting exclude-commit-time flag or ExcludeCommitTime attribute.

Where is this CommitTime accessible? I'm asking here because i see no mention about history timestamps or commit times anywhere else, so I'm wondering if this is something we can access?

I want to compare linux access times to key/value history change timestamps, so that I'm able to determine if a value change was made during an ssh connection to the server.

Is this possible?

[jeroiraz]

Hi @MikeBeller, we've a PR associated to this feature request. Would you like building immudb with the associated branch? User created databases either using immuadmin or a sdk would be reproducible by setting exclude-commit-time flag or ExcludeCommitTime attribute.

Where is this CommitTime accessible? I'm asking here because i see no mention about history timestamps or commit times anywhere else, so I'm wondering if this is something we can access?

I want to compare linux access times to key/value history change timestamps, so that I'm able to determine if a value change was made during an ssh connection to the server.

Is this possible?

aham, interesting

Currently by excluding the commit time, the transaction time is not written. This change was introduced before replication was built-in in immudb, so probably the need of this is reduced to certain use cases. I mean excluding the commit time.

Key-Values itself does not get a time associated as all the entries within the same transaction are atomically stored as part of the same transaction.

While validating the time was not altered by doing time value comparison could be done (using transaction timestamp), it may not be as strong as hash calculation done with the cryptographic proofs. Which is done using the verified methods using a sdk.

I may be missing some things, please feel free to further expand your ideas, either here or in Discord channel or by starting a Discussion topic in github

[MikeBeller]

Hi Jeronimo. Thanks so much for getting back to me on this one. Unfortunately meanwhile I moved on to other projects and am not in a position to focus on immudb right now. If it turns out it is relevant in an upcoming project I will definitely try it out again. The product is very intriguing and I hope some day to get back to it. Mike

On Mon, Jun 26, 2023 at 12:51 PM Jeronimo @.***> wrote:

Hi @MikeBeller https://github.com/MikeBeller, we've a PR associated to this feature request. Would you like building immudb with the associated branch? User created databases either using immuadmin or a sdk would be reproducible by setting exclude-commit-time flag or ExcludeCommitTime attribute.

Where is this CommitTime accessible? I'm asking here because i see no mention about history timestamps or commit times anywhere else, so I'm wondering if this is something we can access?

I want to compare linux access times to key/value history change timestamps, so that I'm able to determine if a value change was made during an ssh connection to the server.

Is this possible?

aham, interesting

Currently by excluding the commit time, the transaction time is not written. This change was introduced before replication was built-in in immudb, so probably the need of this is reduced to certain use cases. I mean excluding the commit time.

Key-Values itself does not get a time associated as all the entries within the same transaction are atomically stored as part of the same transaction.

While validating the time was not altered by doing time value comparison could be done (using transaction timestamp), it may not be as strong as hash calculation done with the cryptographic proofs. Which is done using the verified methods using a sdk.

I may be missing some things, please feel free to further expand your ideas, either here or in Discord channel or by starting a Discussion topic in github

— Reply to this email directly, view it on GitHub https://github.com/codenotary/immudb/issues/889#issuecomment-1607855542, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM7HKXP77FXDB34IJLK57DXNG4YDANCNFSM5B7OF76A . You are receiving this because you were mentioned.Message ID: @.***>