Closed chenyongyin closed 1 year ago
@chenyongyin 你好,我也碰到类似问题,请问下怎么解决
I think before 0.41 fix this. You could manually reverse the highToBitmap using code like the following.
reverseHighInt(Roaring64NavigableMap map) {
Long cardinality = map.getLongCardinality();
if (cardinality>32){
Field highToBitmapField = Roaring64NavigableMap.class.getDeclaredField("highToBitmap");
AccessibleObject.setAccessible(new AccessibleObject[]{highToBitmapField}, true);
NavigableMap<Integer, BitmapDataProvider> highToBitmap =
(NavigableMap<Integer, BitmapDataProvider>) highToBitmapField.get(map);
Set<Integer> oldKeys = new HashSet<>(highToBitmap.keySet());
NavigableMap<Integer, BitmapDataProvider> keyThatNeedReverse = new TreeMap<>();
NavigableSet<Integer> keyThatNeedRemove = new TreeSet<>();
oldKeys.forEach(key -> {
BitmapDataProvider provider = highToBitmap.get(key);
keyThatNeedRemove.add(key);
int high = Integer.reverseBytes(key);
keyThatNeedReverse.put(high,provider);
});
for(Integer key: keyThatNeedRemove){
highToBitmap.remove(key);
}
for (Integer reversedKey : keyThatNeedReverse.keySet()) {
highToBitmap.put(reversedKey, keyThatNeedReverse.get(reversedKey));
}
}
}
This has been fixed in v0.4.1, but you'll have to set Roaring64NavigableMap.SERIALIZATION_MODE
to Roaring64NavigableMap.SERIALIZATION_MODE_PORTABLE
at least once before serialization happens. It's probably better to drop RoaringBitmap dependency and move tailored code into the Java client for ease of use and better performance.
before this i have tried to set Roaring64NavigableMap.SERIALIZATION_MODE = 1,but it didn’t work。I guess it's because clickhouse serializes the result differently, because the nbHighs value in the deserializePortable method is -7240854706849841152,The correct value should be 33
陈永银 @.***
------------------ 原始邮件 ------------------ 发件人: "ClickHouse/clickhouse-java" @.>; 发送时间: 2023年3月16日(星期四) 晚上7:46 @.>; @.**@.>; 主题: Re: [ClickHouse/clickhouse-java] The Roaring64NavigableMap result is incorrect when the groupBitmapState query data exceeds 32 (Issue #1157)
This has been fixed in v0.4.1, but you'll have to set Roaring64NavigableMap.SERIALIZATION_MODE to Roaring64NavigableMap.SERIALIZATION_MODE_PORTABLE at least once before serialization happens. It's probably better to drop RoaringBitmap dependency and move tailored code into the Java client for ease of use and better performance.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
Please feel free to re-open the issue if v0.4.1 does not work for you. If you're looking for an working example, you may check the test at here.
sorry,I don't have set Roaring64NavigableMap.SERIALIZATION_MODE = 1 in v0.4.1,I'll try again later。
陈永银 @.***
------------------ 原始邮件 ------------------ 发件人: "ClickHouse/clickhouse-java" @.>; 发送时间: 2023年3月16日(星期四) 晚上7:46 @.>; @.**@.>; 主题: Re: [ClickHouse/clickhouse-java] The Roaring64NavigableMap result is incorrect when the groupBitmapState query data exceeds 32 (Issue #1157)
This has been fixed in v0.4.1, but you'll have to set Roaring64NavigableMap.SERIALIZATION_MODE to Roaring64NavigableMap.SERIALIZATION_MODE_PORTABLE at least once before serialization happens. It's probably better to drop RoaringBitmap dependency and move tailored code into the Java client for ease of use and better performance.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
clickhouse version:22.5.1.2079 clickhouse-jdbc version:0.3.2-patch11 description:The Roaring64NavigableMap result is incorrect when the groupBitmapState query data exceeds 32 test code: ` @Test public void queryAssociatedMemberData() throws SQLException, IOException { ClickHouseConnection conn = dataSource.getConnection(); ClickHouseStatement statement = conn.createStatement(); ClickHouseRowBinaryInputStream clickHouseRowBinaryInputStream = statement.executeQueryClickhouseRowBinaryStream("SELECT groupBitmapState (submit_id) FROM ( select submit_id FROM test_table order by submit_id desc LIMIT 32 )"); ClickHouseBitmap bit = clickHouseRowBinaryInputStream.readBitmap(ClickHouseDataType.UInt64); Roaring64NavigableMap obj = (Roaring64NavigableMap) bit.unwrap(); System.out.println(Arrays.toString(obj.toArray())); }
`