search

XpressAI / SparkCyclone

Plugin to accelerate Spark SQL with the NEC Vector Engine.
https://sparkcyclone.io
Apache License 2.0
16 stars 4 forks source link

Re-enable append_bitsets #578

Closed treo closed 2 years ago

treo commented 2 years ago

The problem in NS-49 was that the packed transfer spec failed because during transfer only the bits for existing elements are considered when merging data into the validity buffer. padding bits are kept at 0.

However, the default validity buffer created when converting an Array of scalars into a BytePointerColVector would be set to "all valid" even for the padding bits.

This created a discrepancy after a round trip VH->VE->VH.

In this PR I've adapted the conversion code to adopt the "padding bits are 0" default.

Something appears to be still wrong :/ the TPCH tests fail with just slightly wrong results.

sonarcloud[bot] commented 2 years ago

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication