I am currently storing tick data in binary encoding format of my own (primarily using varint). I am looking into FBE as an alternative to store and fast replaying of data in my application. After playing a bit with FBE, i found that the storage requirement of FBE encoded data is must larger than my own version in order of 5x to 7x on an average. This is not useful for me as small storage is also one of the main requirements of the tick data. I am exploring my options which takes less storage space and is faster to decode and replay.
FBE is optimized for fast decoding but it is still storing native data as it is (double, float, decimal, long etc) and hence the message size is still relatively large.
By introducing Varint in encoding and decoding of the data, the message size can be reduced substantially by giving away few cycles of computation. I do feel that there could be a gain in performance due to much reduced message size as well.
I understand that in FBE, the message size is fixed and writes the length in advance and varint will defeat this purpose, but it will be much easier to just change the length after encoding the message. (Varint wil make the message size variable instead of fixed as it is now).
I am wondering if there is any plan to introduce Varint encoding and decoding to FBE, at least as an option so some users may use it and others not. I would like to do a performance analysis when I get time, wanted some input from others if anyone has explored this option or do see any benefits of this? Thank you.
I am currently storing tick data in binary encoding format of my own (primarily using varint). I am looking into FBE as an alternative to store and fast replaying of data in my application. After playing a bit with FBE, i found that the storage requirement of FBE encoded data is must larger than my own version in order of 5x to 7x on an average. This is not useful for me as small storage is also one of the main requirements of the tick data. I am exploring my options which takes less storage space and is faster to decode and replay.
FBE is optimized for fast decoding but it is still storing native data as it is (double, float, decimal, long etc) and hence the message size is still relatively large.
By introducing Varint in encoding and decoding of the data, the message size can be reduced substantially by giving away few cycles of computation. I do feel that there could be a gain in performance due to much reduced message size as well.
I understand that in FBE, the message size is fixed and writes the length in advance and varint will defeat this purpose, but it will be much easier to just change the length after encoding the message. (Varint wil make the message size variable instead of fixed as it is now).
I am wondering if there is any plan to introduce Varint encoding and decoding to FBE, at least as an option so some users may use it and others not. I would like to do a performance analysis when I get time, wanted some input from others if anyone has explored this option or do see any benefits of this? Thank you.