iceberg - dictionaryPagesSize and bloomFilter cannot both be set

[raphaelauv]

hi , I can't run

ALTER TABLE iceberg.default.AA EXECUTE optimize

on a table that have bloom filters

CREATE TABLE iceberg.default.AA 
( 
 created_at timestamp(6) NOT NULL,
 toto varchar NOT NULL
 ) WITH ( 
 format = 'PARQUET',
 format_version = 2,
 location = 's3://XXXX/default/AA',
 parquet_bloom_filter_columns = ARRAY['toto'],
 partitioning = ARRAY['hour(created_at)'] 
 )

it log

dictionaryPagesSize and bloomFilter cannot both be set

[raunaqmorarka]

@jkylling PTAL

[jkylling]

It even fails on

        String tableName = "parquet_with_bloom_filters_" + randomNameSuffix();
        CatalogSchemaTableName catalogSchemaTableName = new CatalogSchemaTableName("iceberg", new SchemaTableName("tpch", tableName));
        assertUpdate(format("CREATE TABLE %s WITH (format = 'PARQUET', parquet_bloom_filter_columns = ARRAY['t']) AS SELECT * FROM (VALUES 1, 1) s(t)", catalogSchemaTableName), 2);

If a page of a column chunk has dictionary encoding we currently do not write its values into the Bloom filter, but we will still try to write a Bloom filter for that column. Luckily this check caught it.

It probably makes sense to remove the check and do one of:

• Not write the Bloom filter if some pages of a column chunk are dictionary encoded?
• Always populate the Bloom filter in memory, but not write it if all the pages of a column chunk are dictionary encoded?
• Refactor the fallback logic such that we write all dictionary values to the Bloom filter on fall back, and not just the values from the page which overflowed the dictionary. We would also update the check to see if a Bloom filter is set, or if the Bloom filter has never been written to.