jerolba
commented
2 months ago I have implemented Carpet to make it as fast as possible, but it has a problem: it uses reflection to access Java record components.
When using reflection to deal with primitive values, you always get the boxed instance. I couldn't find a way to avoid this without generating bytecode. I found that using method handles is the fastest way to perform reflection, but in any case, you still end up with a wrapped instance.
As a result, in Carpet, there is no performance difference between using primitive or boxed values, because at some point, Carpet uses a boxed instance (which is immediately discarded and collected by the GC in the young generation).
In my tests, Carpet's serialization performance is better than that of Parquet Avro and Parquet Protobuf:
- Parquet Avro: 15,381 ms
- Parquet Protocol Buffers: 16,174 ms
- Carpet: 12,769 ms
If you find a faster way to serialize Parquet primitive data in Java, we can compare it with Carpet and assess the margin for improvement if I manage to work with primitives directly.
javafanboy
In many cases it would be valuable to specify also numeric attributes (= Parquet columns) as Nullable in an efficient way.
Since Carpet uses a "record" to ingest data one way to specify that a numeric value should be nullable would be an annotation that also could specify one value that should be used to represent "NULL". This will not work in all situations but in quite a lot of cases one msy for instance know that only non-negative values will be used and then -1 could be specified to represent "NULL". When the marker is found "carpet" would enter NULL into the Parquet file.
Another way (already used by Carpet)is using the wrapper classes Integer, Short etc. but these are expensive to create (allocated on heap for starters) and are not prefered for performance sensitive use-cases and a marker annotation would therefore be preferred...