Converting table to columnar storage fails with out of memory error

[EraYaN]

So the detailed error is as follows:

ERROR:  out of memory
DETAIL:  Cannot enlarge string buffer containing 1073253624 bytes by 1269320 more bytes.
CONTEXT:  SQL statement "INSERT INTO public.raw_p2020_03_2430651750 (id,time,raw) OVERRIDING SYSTEM VALUE SELECT id,time,raw FROM public.raw_p2020_03"
SQL statement "SELECT pg_catalog.alter_table_set_access_method(r.partition, new_access_method)"
PL/pgSQL function alter_old_partitions_set_access_method(regclass,timestamp with time zone,name) line 27 at PERFORM

The problem partition has about 14000 large text blobs that in total are about 8.5GB (range 1.6 MB to 150kB in individual size) and this causes citus_columnar to fail the conversion. But we really like the compression citus can provide in columnar mode for this text (lost of very easy to compress text in those values)

So we set the stripe size to it's lowest possible setting (1000) but this is still not enough. Is there any way to fix this? Can citus automatically make smaller stripes when about to hit this limit?

We'd also get this error if we ran a COPY command to ingest into a table that was already columnar so it seems to be inserting/encoding related.

[onurctirtir]

Conceptually related to https://github.com/citusdata/citus/issues/6420.