StarRocks / starrocks-connector-for-apache-spark

Apache License 2.0
36 stars 53 forks source link

map starrocks bitmap type to array<int64> in sparksql #107

Closed zizon closed 6 months ago

zizon commented 7 months ago

What type of PR is this:

Which issues of this PR fixes :

Fixes # Support bitmap type of starrocks in spark context.

Problem Summary(Required) :

This minor patch introduce starrocks bitmap type into spark as native array type, while limited but sill supports read from/write into in both direction.

Before this patch, it is possible and in some case more reliable for user to choose its mapping and walk around to interact with starrocks, like doing the trick manually like this implementation.

Things to note that, as the BE do not natively support non-primitive arrow type such as Array in interaction, this patch use string as a intermedia before handing to user, which could produce huge memory impact if the resulting bitmap have a high rank.

Checklist:

CLAassistant commented 7 months ago

CLA assistant check
All committers have signed the CLA.