adam-dot-cohen
commented
7 months ago ``Few things...
-
Your candle is a struct - copying it to to
Memory<T>(let alone ontoMemoryStream) when you can use the non-asyncSerializemethod to get a stack basedSpan<byte>will be significantly faster vs allocating a pointer on the heap. -
You're copying around the serialized bytes to the heap and subsequently copy 3 more times unnecessarily. You pass
Memory<byte>toMemoryStreamwhich data from one array to another array for no reason, and then you're copying the entire stream again withToArray- iterating over the stream's internal array byte-by-byte making a copy of yet another array that gets passed to 'File.WriteAllBytes. **If you just use a FileStream (e.g.FileStream fs = File.OpenWrite()) andSpanbytes = HyperSerializer.Serialize(item) - and the callfs.Write(bytes)` it eliminates all the copying and leverages the Formula 1 speed of Span- plus you eliminate the the overhead of pointers, copying and iterating 3 times unnecessarily, etc.**
The most important point is that you don't really need to serialize each array item one-by-one. Just pass the array to the Serialize function - like this example.
In related news, I copied your code from above into Linqpad (see below), ran some benchmarks (see below) and provided both for reference. If you use Linqpad, you can just add the NuGet packages, copy the code into a "Program" script, and finally highlight the functions you want to benchmark and hit "ctrl+shift+b." I added a class version of a candle for further comparison vs TeaCup (only supports structs) and HyperSerializer still 33% faster (see bottom benchmark line. This is an important point because classes tend to be more suitable for stream processing and ML trading algos).
All benchmarks are 10M randomly generated candle objects (structs and/or classes).
_I'm going write a separate post to address some of your other questions which are equally if not more important asap. Here's the Linqpad script...
Linqpad "Program" script type...add HyperSerializer and Teacup references. Teacup's bits are still .Net Framework Linqpad (and .Net in general) doesn't play well with...
async Task Main()
{
}
#region Highlight and CTRL + SHIFT + B
void Tea_ForEach_Struct_10M()
{
var path = @"e:\temp\teaFile6.tea";
using (var tf = TeaFile<CandlestickLongStruct>.Create(path))
{
foreach (var item in Bars)
{
tf.Write(item);
}
}
File.Delete(path);
}
void HyperSerializer_ForEach_Struct_FileStream_10M()
{
var path = @"e:\temp\hyper.bin";
using (var fs = File.OpenWrite(path))
{
foreach (var item in Bars)
{
Span<byte> bytes = Hyper.HyperSerializer.Serialize(item);
fs.Write(bytes);
}
}
}
void HyperSerializer_NoLoop_FileStream_10M()
{
var path = @"e:\temp\hyper.bin";
using (var fs = File.OpenWrite(path))
{
Span<byte> bytes = Hyper.HyperSerializer.Serialize(Bars);
fs.Write(bytes);
}
File.Delete(path);
}
void HyperSerializer_ForEach_Class_FileStream_10M()
{
var path = @"e:\temp\hyper.bin";
using (var fs = File.OpenWrite(path))
{
foreach (var item in BarsClass)
{
var bytes = Hyper.HyperSerializer.Serialize(item);
fs.Write(bytes);
}
}
File.Delete(path);
}
#endregion
#region Benchark Data
Random rand = new Random();
CandlestickLongStruct[] _bars;
public CandlestickLongStruct[] Bars => _bars ??=
Enumerable.Range(0, 10_000_000).Select(x =>
new CandlestickLongStruct
{
High = rand.Next(),
Open = rand.Next(),
Close = rand.Next(),
Low = rand.Next()
})
.ToArray();
CandlestickLongClass[] _barsClass;
public CandlestickLongClass[] BarsClass => _barsClass ??=
Enumerable.Range(0, 10_000_000).Select(x =>
new CandlestickLongClass
{
High = rand.Next(),
Open = rand.Next(),
Close = rand.Next(),
Low = rand.Next()
})
.ToArray();
public struct CandlestickLongStruct
{
public int High { get; set; }
public int Low { get; set; }
public int Close { get; set; }
public int Open { get; set; }
}
public class CandlestickLongClass
{
public int High { get; set; }
public int Low { get; set; }
public int Close { get; set; }
public int Open { get; set; }
}
#endregion
pavlexander
I would like to use this library to save/append the candlestick trading data (OHLCV) into files.
After looking at advertised performance I have tried using this library, but it's just not as fast as simply saving the data to a binary file.. I would like to know if I am misusing the library or it's simply not meant for the given use-case? The test data set consists of 3_020_871 records.
the test code looks like this
for performance comparison, I am also using the
teafileslibrary (which basically is a wrapper for brute-force binary serialization):the results are:
it does seems like the
HyperSerializerproduces almost exact size output file, but the performance is much worse.The aim of this post of course, not to compare this lib to others. I genuinely want to replace the teafiles library and looking for a better solution. I would appreciate to hear a feedback on the performance issue..
for the sake of completeness, here's the serialized data type
.net 7,HyperSerializer 1.4.0,Rubble.TeaFiles.Net 2.0.0additionally I would like to know how would you solve the problem of appending the data to an existing file and also getting/upating a total number of records in file.