search

go-analyze / flat-file-map

Golang library for storing key / value data in a flat file
Mozilla Public License 2.0
1 stars 0 forks source link

Single record json structures do not encode the struct id in the flat file #4

Open jentfoo opened 8 months ago

jentfoo commented 8 months ago

Not encoding the struct id does not cause an issue with de-serialization, but it can result in future writes to not be able to optimize within the file. For example if one record is added before each Commit(), it's likely that there would be no field de-duplication even if all the structs were the same.

jentfoo commented 8 months ago

I have considered this further, I have not decided how I want to address this. There are three options:

  1. Encoding the struct id is trivial, but will result in increased file sizes which would only be beneficial given very specific circumstances
  2. We could keep the behavior as it is currently. There is no behavior issue in retrieving values, this just prevents the possible multiple record size optimization. So this option is just facing a different form of size increase, but is arguably fairly contrived.
  3. We can make this a configurable behavior

Right now I lean towards option 2. This does not seem like a behavior that should need configuration. And the Commit / Load situation is probably contrived enough that it's not worth the proactive file size lost to encode the struct id.

We will keep considering this, and welcome feedback if anyone has any.