Open debugger22m opened 4 years ago
I'm not sure I'm fully following what you are saying. Is it that, for certain chunks, you see the uncompressed size less than the compressed size? (I can't parse the above given "chunks sizes vs. heap bytes).
Also, when you say "more than 5 tables in your segmentby", do you mean 5 columns?
If you specify a large number of columns, or a column with a large number of distinct items, in your segmentby, it can be that basically there is no "batching" that can occur when converting rows to columns. Basically, you only have a single (or small number) of items that match that unique segmentby (particularly within each chunk), such that the "compressed array" only has a very small number of items and thus doesn't compress well at all.
@mfreed
Sorry, I should have provided more details. This is for the entire hypertable
SELECT hypertable_name, uncompressed_heap_bytes, compressed_heap_bytes FROM timescaledb_information.compressed_hypertable_stats;
Yes, I meant segmentby has 5 and more columns. I will try this again by limiting the columns.
Thank you!.
Hi @120bits is this still affecting you?
having more or less the same problem. because of the bigger heap I can't compress certain table as the compressed chunk hits the error: SELECT compress_chunk(i) from show_chunks('mytable', older_than => INTERVAL '2 days') i [54000]: ERROR: row is too big: size 8648, maximum size 8160
There may be too many columns and it hits the block size limit after inefficient compression. Can you give some more information about the schema?
sure, here we go:
CREATE TABLE spg_mon_db2.mon_get_database (
id int8 NOT NULL,
snapshot_time timestamptz NOT NULL,
hostname varchar(255) NOT NULL,
port int4 NOT NULL,
display_name varchar(25) NULL,
inst_name varchar(128) NOT NULL,
db_name varchar(18) NOT NULL,
"member" int2 NOT NULL,
db_status varchar(16) NULL,
db_activation_state varchar(10) NULL,
db_conn_time timestamptz NULL,
catalog_partition int2 NULL,
last_backup timestamptz NULL,
connections_top int8 NULL,
total_cons int8 NULL,
total_sec_cons int8 NULL,
appls_cur_cons int8 NULL,
appls_in_db2 int8 NULL,
num_assoc_agents int8 NULL,
agents_top int8 NULL,
num_coord_agents int8 NULL,
coord_agents_top int8 NULL,
num_locks_held int8 NULL,
num_locks_waiting int8 NULL,
lock_list_in_use int8 NULL,
active_sorts int8 NULL,
active_hash_joins int8 NULL,
active_olap_funcs int8 NULL,
db_path varchar(1024) NULL,
act_aborted_total int8 NULL,
act_completed_total int8 NULL,
act_rejected_total int8 NULL,
agent_wait_time int8 NULL,
agent_waits_total int8 NULL,
pool_data_l_reads int8 NULL,
pool_index_l_reads int8 NULL,
pool_temp_data_l_reads int8 NULL,
pool_temp_index_l_reads int8 NULL,
pool_temp_xda_l_reads int8 NULL,
pool_xda_l_reads int8 NULL,
pool_data_p_reads int8 NULL,
pool_index_p_reads int8 NULL,
pool_temp_data_p_reads int8 NULL,
pool_temp_index_p_reads int8 NULL,
pool_temp_xda_p_reads int8 NULL,
pool_xda_p_reads int8 NULL,
pool_data_writes int8 NULL,
pool_index_writes int8 NULL,
pool_xda_writes int8 NULL,
pool_read_time int8 NULL,
pool_write_time int8 NULL,
client_idle_wait_time int8 NULL,
deadlocks int8 NULL,
direct_reads int8 NULL,
direct_read_time int8 NULL,
direct_writes int8 NULL,
direct_write_time int8 NULL,
direct_read_reqs int8 NULL,
direct_write_reqs int8 NULL,
fcm_recv_volume int8 NULL,
fcm_recvs_total int8 NULL,
fcm_send_volume int8 NULL,
fcm_sends_total int8 NULL,
fcm_recv_wait_time int8 NULL,
fcm_send_wait_time int8 NULL,
ipc_recv_volume int8 NULL,
ipc_recv_wait_time int8 NULL,
ipc_recvs_total int8 NULL,
ipc_send_volume int8 NULL,
ipc_send_wait_time int8 NULL,
ipc_sends_total int8 NULL,
lock_escals int8 NULL,
lock_timeouts int8 NULL,
lock_wait_time int8 NULL,
lock_waits int8 NULL,
log_buffer_wait_time int8 NULL,
num_log_buffer_full int8 NULL,
log_disk_wait_time int8 NULL,
log_disk_waits_total int8 NULL,
rqsts_completed_total int8 NULL,
rows_modified int8 NULL,
rows_read int8 NULL,
rows_returned int8 NULL,
tcpip_recv_volume int8 NULL,
tcpip_send_volume int8 NULL,
tcpip_recv_wait_time int8 NULL,
tcpip_recvs_total int8 NULL,
tcpip_send_wait_time int8 NULL,
tcpip_sends_total int8 NULL,
total_app_rqst_time int8 NULL,
total_rqst_time int8 NULL,
wlm_queue_time_total int8 NULL,
wlm_queue_assignments_total int8 NULL,
total_rqst_mapped_in int8 NULL,
total_rqst_mapped_out int8 NULL,
total_cpu_time int8 NULL,
total_wait_time int8 NULL,
app_rqsts_completed_total int8 NULL,
total_section_sort_time int8 NULL,
total_section_sort_proc_time int8 NULL,
total_section_sorts int8 NULL,
total_sorts int8 NULL,
post_threshold_sorts int8 NULL,
post_shrthreshold_sorts int8 NULL,
sort_overflows int8 NULL,
total_compile_time int8 NULL,
total_compile_proc_time int8 NULL,
total_compilations int8 NULL,
total_implicit_compile_time int8 NULL,
total_implicit_compile_proc_time int8 NULL,
total_implicit_compilations int8 NULL,
total_section_time int8 NULL,
total_section_proc_time int8 NULL,
total_app_section_executions int8 NULL,
total_act_time int8 NULL,
total_act_wait_time int8 NULL,
act_rqsts_total int8 NULL,
total_routine_time int8 NULL,
total_routine_invocations int8 NULL,
total_commit_time int8 NULL,
total_commit_proc_time int8 NULL,
total_app_commits int8 NULL,
int_commits int8 NULL,
total_rollback_time int8 NULL,
total_rollback_proc_time int8 NULL,
total_app_rollbacks int8 NULL,
int_rollbacks int8 NULL,
total_runstats_time int8 NULL,
total_runstats_proc_time int8 NULL,
total_runstats int8 NULL,
total_reorg_time int8 NULL,
total_reorg_proc_time int8 NULL,
total_reorgs int8 NULL,
total_load_time int8 NULL,
total_load_proc_time int8 NULL,
total_loads int8 NULL,
cat_cache_inserts int8 NULL,
cat_cache_lookups int8 NULL,
pkg_cache_inserts int8 NULL,
pkg_cache_lookups int8 NULL,
thresh_violations int8 NULL,
num_lw_thresh_exceeded int8 NULL,
lock_waits_global int8 NULL,
lock_wait_time_global int8 NULL,
lock_timeouts_global int8 NULL,
lock_escals_maxlocks int8 NULL,
lock_escals_locklist int8 NULL,
lock_escals_global int8 NULL,
data_sharing_remote_lockwait_count int8 NULL,
data_sharing_remote_lockwait_time int8 NULL,
reclaim_wait_time int8 NULL,
spacemappage_reclaim_wait_time int8 NULL,
cf_waits int8 NULL,
cf_wait_time int8 NULL,
pool_data_gbp_l_reads int8 NULL,
pool_data_gbp_p_reads int8 NULL,
pool_data_lbp_pages_found int8 NULL,
pool_data_gbp_invalid_pages int8 NULL,
pool_index_gbp_l_reads int8 NULL,
pool_index_gbp_p_reads int8 NULL,
pool_index_lbp_pages_found int8 NULL,
pool_index_gbp_invalid_pages int8 NULL,
pool_xda_gbp_l_reads int8 NULL,
pool_xda_gbp_p_reads int8 NULL,
pool_xda_lbp_pages_found int8 NULL,
pool_xda_gbp_invalid_pages int8 NULL,
audit_events_total int8 NULL,
audit_file_writes_total int8 NULL,
audit_file_write_wait_time int8 NULL,
audit_subsystem_waits_total int8 NULL,
audit_subsystem_wait_time int8 NULL,
diaglog_writes_total int8 NULL,
diaglog_write_wait_time int8 NULL,
fcm_message_recvs_total int8 NULL,
fcm_message_recv_volume int8 NULL,
fcm_message_recv_wait_time int8 NULL,
fcm_message_sends_total int8 NULL,
fcm_message_send_volume int8 NULL,
fcm_message_send_wait_time int8 NULL,
fcm_tq_recvs_total int8 NULL,
fcm_tq_recv_volume int8 NULL,
fcm_tq_recv_wait_time int8 NULL,
fcm_tq_sends_total int8 NULL,
fcm_tq_send_volume int8 NULL,
fcm_tq_send_wait_time int8 NULL,
total_routine_user_code_proc_time int8 NULL,
total_routine_user_code_time int8 NULL,
tq_tot_send_spills int8 NULL,
evmon_wait_time int8 NULL,
evmon_waits_total int8 NULL,
total_extended_latch_wait_time int8 NULL,
total_extended_latch_waits int8 NULL,
total_stats_fabrication_time int8 NULL,
total_stats_fabrication_proc_time int8 NULL,
total_stats_fabrications int8 NULL,
total_sync_runstats_time int8 NULL,
total_sync_runstats_proc_time int8 NULL,
total_sync_runstats int8 NULL,
total_disp_run_queue_time int8 NULL,
total_peds int8 NULL,
disabled_peds int8 NULL,
post_threshold_peds int8 NULL,
total_peas int8 NULL,
post_threshold_peas int8 NULL,
tq_sort_heap_requests int8 NULL,
tq_sort_heap_rejections int8 NULL,
pool_queued_async_data_reqs int8 NULL,
pool_queued_async_index_reqs int8 NULL,
pool_queued_async_xda_reqs int8 NULL,
pool_queued_async_temp_data_reqs int8 NULL,
pool_queued_async_temp_index_reqs int8 NULL,
pool_queued_async_temp_xda_reqs int8 NULL,
pool_queued_async_other_reqs int8 NULL,
pool_queued_async_data_pages int8 NULL,
pool_queued_async_index_pages int8 NULL,
pool_queued_async_xda_pages int8 NULL,
pool_queued_async_temp_data_pages int8 NULL,
pool_queued_async_temp_index_pages int8 NULL,
pool_queued_async_temp_xda_pages int8 NULL,
pool_failed_async_data_reqs int8 NULL,
pool_failed_async_index_reqs int8 NULL,
pool_failed_async_xda_reqs int8 NULL,
pool_failed_async_temp_data_reqs int8 NULL,
pool_failed_async_temp_index_reqs int8 NULL,
pool_failed_async_temp_xda_reqs int8 NULL,
pool_failed_async_other_reqs int8 NULL,
prefetch_wait_time int8 NULL,
prefetch_waits int8 NULL,
app_act_completed_total int8 NULL,
app_act_aborted_total int8 NULL,
app_act_rejected_total int8 NULL,
total_connect_request_time int8 NULL,
total_connect_request_proc_time int8 NULL,
total_connect_requests int8 NULL,
total_connect_authentication_time int8 NULL,
total_connect_authentication_proc_time int8 NULL,
total_connect_authentications int8 NULL,
pool_data_gbp_indep_pages_found_in_lbp int8 NULL,
pool_index_gbp_indep_pages_found_in_lbp int8 NULL,
pool_xda_gbp_indep_pages_found_in_lbp int8 NULL,
comm_exit_wait_time int8 NULL,
comm_exit_waits int8 NULL,
pool_async_data_reads int8 NULL,
pool_async_data_read_reqs int8 NULL,
pool_async_data_writes int8 NULL,
pool_async_index_reads int8 NULL,
pool_async_index_read_reqs int8 NULL,
pool_async_index_writes int8 NULL,
pool_async_xda_reads int8 NULL,
pool_async_xda_read_reqs int8 NULL,
pool_async_xda_writes int8 NULL,
pool_no_victim_buffer int8 NULL,
pool_lsn_gap_clns int8 NULL,
pool_drty_pg_steal_clns int8 NULL,
pool_drty_pg_thrsh_clns int8 NULL,
vectored_ios int8 NULL,
pages_from_vectored_ios int8 NULL,
block_ios int8 NULL,
pages_from_block_ios int8 NULL,
unread_prefetch_pages int8 NULL,
files_closed int8 NULL,
pool_async_data_gbp_l_reads int8 NULL,
pool_async_data_gbp_p_reads int8 NULL,
pool_async_data_lbp_pages_found int8 NULL,
pool_async_data_gbp_invalid_pages int8 NULL,
pool_async_index_gbp_l_reads int8 NULL,
pool_async_index_gbp_p_reads int8 NULL,
pool_async_index_lbp_pages_found int8 NULL,
pool_async_index_gbp_invalid_pages int8 NULL,
pool_async_xda_gbp_l_reads int8 NULL,
pool_async_xda_gbp_p_reads int8 NULL,
pool_async_xda_lbp_pages_found int8 NULL,
pool_async_xda_gbp_invalid_pages int8 NULL,
pool_async_read_time int8 NULL,
pool_async_write_time int8 NULL,
skipped_prefetch_data_p_reads int8 NULL,
skipped_prefetch_index_p_reads int8 NULL,
skipped_prefetch_xda_p_reads int8 NULL,
skipped_prefetch_temp_data_p_reads int8 NULL,
skipped_prefetch_temp_index_p_reads int8 NULL,
skipped_prefetch_temp_xda_p_reads int8 NULL,
skipped_prefetch_uow_data_p_reads int8 NULL,
skipped_prefetch_uow_index_p_reads int8 NULL,
skipped_prefetch_uow_xda_p_reads int8 NULL,
skipped_prefetch_uow_temp_data_p_reads int8 NULL,
skipped_prefetch_uow_temp_index_p_reads int8 NULL,
skipped_prefetch_uow_temp_xda_p_reads int8 NULL,
pool_async_data_gbp_indep_pages_found_in_lbp int8 NULL,
pool_async_index_gbp_indep_pages_found_in_lbp int8 NULL,
pool_async_xda_gbp_indep_pages_found_in_lbp int8 NULL,
caching_tier varchar(8) NULL,
caching_tier_io_errors int8 NULL,
pool_data_caching_tier_l_reads int8 NULL,
pool_index_caching_tier_l_reads int8 NULL,
pool_xda_caching_tier_l_reads int8 NULL,
pool_col_caching_tier_l_reads int8 NULL,
pool_data_caching_tier_page_writes int8 NULL,
pool_index_caching_tier_page_writes int8 NULL,
pool_xda_caching_tier_page_writes int8 NULL,
pool_col_caching_tier_page_writes int8 NULL,
pool_data_caching_tier_page_updates int8 NULL,
pool_index_caching_tier_page_updates int8 NULL,
pool_xda_caching_tier_page_updates int8 NULL,
pool_col_caching_tier_page_updates int8 NULL,
pool_caching_tier_page_read_time int8 NULL,
pool_caching_tier_page_write_time int8 NULL,
pool_data_caching_tier_pages_found int8 NULL,
pool_index_caching_tier_pages_found int8 NULL,
pool_xda_caching_tier_pages_found int8 NULL,
pool_col_caching_tier_pages_found int8 NULL,
pool_data_caching_tier_gbp_invalid_pages int8 NULL,
pool_index_caching_tier_gbp_invalid_pages int8 NULL,
pool_xda_caching_tier_gbp_invalid_pages int8 NULL,
pool_col_caching_tier_gbp_invalid_pages int8 NULL,
pool_data_caching_tier_gbp_indep_pages_found int8 NULL,
pool_index_caching_tier_gbp_indep_pages_found int8 NULL,
pool_xda_caching_tier_gbp_indep_pages_found int8 NULL,
pool_col_caching_tier_gbp_indep_pages_found int8 NULL,
pool_async_data_caching_tier_reads int8 NULL,
pool_async_index_caching_tier_reads int8 NULL,
pool_async_xda_caching_tier_reads int8 NULL,
pool_async_col_caching_tier_reads int8 NULL,
pool_async_data_caching_tier_page_writes int8 NULL,
pool_async_index_caching_tier_page_writes int8 NULL,
pool_async_xda_caching_tier_page_writes int8 NULL,
pool_async_col_caching_tier_page_writes int8 NULL,
pool_async_data_caching_tier_page_updates int8 NULL,
pool_async_index_caching_tier_page_updates int8 NULL,
pool_async_xda_caching_tier_page_updates int8 NULL,
pool_async_col_caching_tier_page_updates int8 NULL,
pool_async_data_caching_tier_pages_found int8 NULL,
pool_async_index_caching_tier_pages_found int8 NULL,
pool_async_xda_caching_tier_pages_found int8 NULL,
pool_async_col_caching_tier_pages_found int8 NULL,
pool_async_data_caching_tier_gbp_invalid_pages int8 NULL,
pool_async_index_caching_tier_gbp_invalid_pages int8 NULL,
pool_async_xda_caching_tier_gbp_invalid_pages int8 NULL,
pool_async_col_caching_tier_gbp_invalid_pages int8 NULL,
pool_async_data_caching_tier_gbp_indep_pages_found int8 NULL,
pool_async_index_caching_tier_gbp_indep_pages_found int8 NULL,
pool_async_xda_caching_tier_gbp_indep_pages_found int8 NULL,
pool_async_col_caching_tier_gbp_indep_pages_found int8 NULL,
rows_deleted int8 NULL,
rows_inserted int8 NULL,
rows_updated int8 NULL,
total_hash_joins int8 NULL,
total_hash_loops int8 NULL,
hash_join_overflows int8 NULL,
hash_join_small_overflows int8 NULL,
post_shrthreshold_hash_joins int8 NULL,
total_olap_funcs int8 NULL,
olap_func_overflows int8 NULL,
dynamic_sql_stmts int8 NULL,
static_sql_stmts int8 NULL,
failed_sql_stmts int8 NULL,
select_sql_stmts int8 NULL,
uid_sql_stmts int8 NULL,
ddl_sql_stmts int8 NULL,
merge_sql_stmts int8 NULL,
xquery_stmts int8 NULL,
implicit_rebinds int8 NULL,
binds_precompiles int8 NULL,
int_rows_deleted int8 NULL,
int_rows_inserted int8 NULL,
int_rows_updated int8 NULL,
call_sql_stmts int8 NULL,
pool_col_l_reads int8 NULL,
pool_temp_col_l_reads int8 NULL,
pool_col_p_reads int8 NULL,
pool_temp_col_p_reads int8 NULL,
pool_col_lbp_pages_found int8 NULL,
pool_col_writes int8 NULL,
pool_async_col_reads int8 NULL,
pool_async_col_read_reqs int8 NULL,
pool_async_col_writes int8 NULL,
pool_async_col_lbp_pages_found int8 NULL,
pool_col_gbp_l_reads int8 NULL,
pool_col_gbp_p_reads int8 NULL,
pool_col_gbp_invalid_pages int8 NULL,
pool_col_gbp_indep_pages_found_in_lbp int8 NULL,
pool_async_col_gbp_l_reads int8 NULL,
pool_async_col_gbp_p_reads int8 NULL,
pool_async_col_gbp_invalid_pages int8 NULL,
pool_async_col_gbp_indep_pages_found_in_lbp int8 NULL,
pool_queued_async_col_reqs int8 NULL,
pool_queued_async_temp_col_reqs int8 NULL,
pool_queued_async_col_pages int8 NULL,
pool_queued_async_temp_col_pages int8 NULL,
pool_failed_async_col_reqs int8 NULL,
pool_failed_async_temp_col_reqs int8 NULL,
skipped_prefetch_col_p_reads int8 NULL,
skipped_prefetch_temp_col_p_reads int8 NULL,
skipped_prefetch_uow_col_p_reads int8 NULL,
skipped_prefetch_uow_temp_col_p_reads int8 NULL,
total_col_time int8 NULL,
total_col_proc_time int8 NULL,
total_col_executions int8 NULL,
num_pooled_agents int8 NULL,
post_threshold_hash_joins int8 NULL,
pkg_cache_num_overflows int8 NULL,
cat_cache_overflows int8 NULL,
total_async_runstats int8 NULL,
stats_cache_size int8 NULL,
total_hash_grpbys int8 NULL,
hash_grpby_overflows int8 NULL,
post_threshold_hash_grpbys int8 NULL,
active_hash_grpbys int8 NULL,
sort_heap_allocated int8 NULL,
sort_shrheap_allocated int8 NULL,
sort_shrheap_top int8 NULL,
post_threshold_olap_funcs int8 NULL,
ext_table_recv_wait_time int8 NULL,
ext_table_recvs_total int8 NULL,
ext_table_recv_volume int8 NULL,
ext_table_read_volume int8 NULL,
ext_table_send_wait_time int8 NULL,
ext_table_sends_total int8 NULL,
ext_table_send_volume int8 NULL,
ext_table_write_volume int8 NULL,
post_threshold_col_vector_consumers int8 NULL,
total_col_vector_consumers int8 NULL,
active_hash_grpbys_top int8 NULL,
active_hash_joins_top int8 NULL,
active_olap_funcs_top int8 NULL,
active_peas int8 NULL,
active_peas_top int8 NULL,
active_peds int8 NULL,
active_peds_top int8 NULL,
active_sort_consumers int8 NULL,
active_sort_consumers_top int8 NULL,
active_sorts_top int8 NULL,
active_col_vector_consumers int8 NULL,
active_col_vector_consumers_top int8 NULL,
sort_consumer_heap_top int8 NULL,
sort_consumer_shrheap_top int8 NULL,
sort_heap_top int8 NULL,
total_backup_time int8 NULL,
total_backup_proc_time int8 NULL,
total_backups int8 NULL,
total_index_build_time int8 NULL,
total_index_build_proc_time int8 NULL,
total_indexes_built int8 NULL,
ida_send_wait_time int8 NULL,
ida_sends_total int8 NULL,
ida_send_volume int8 NULL,
ida_recv_wait_time int8 NULL,
ida_recvs_total int8 NULL,
ida_recv_volume int8 NULL,
fcm_tq_recv_waits_total int8 NULL,
fcm_message_recv_waits_total int8 NULL,
fcm_tq_send_waits_total int8 NULL,
fcm_message_send_waits_total int8 NULL,
fcm_send_waits_total int8 NULL,
fcm_recv_waits_total int8 NULL,
col_vector_consumer_overflows int8 NULL,
total_col_synopsis_time int8 NULL,
total_col_synopsis_proc_time int8 NULL,
total_col_synopsis_executions int8 NULL,
col_synopsis_rows_inserted int8 NULL,
appl_section_inserts int8 NULL,
appl_section_lookups int8 NULL,
shr_workspace_active int8 NULL,
shr_workspace_inactive int8 NULL,
lob_prefetch_wait_time int8 NULL,
lob_prefetch_reqs int8 NULL,
fed_rows_deleted int8 NULL,
fed_rows_inserted int8 NULL,
fed_rows_updated int8 NULL,
fed_rows_read int8 NULL,
fed_wait_time int8 NULL,
fed_waits_total int8 NULL,
adm_overflows int8 NULL,
adm_bypass_act_total int8 NULL,
authn_cache_lookups int8 NULL,
authn_cache_inserts int8 NULL,
authn_cache_hits int8 NULL,
authn_cache_group_inserts int8 NULL,
authn_cache_expired_evictions int8 NULL,
authn_cache_valid_evictions int8 NULL,
authn_cache_memory_usage int8 NULL,
CONSTRAINT mon_get_database_pkey PRIMARY KEY (snapshot_time, hostname, inst_name, db_name, member)
);
A possible fix here would be to check the row size when turning on the compression
flag rather than when trying to compress the chunk. This would give an error when compression is enabled for the table rather than when an attempt is made to compress the table, which is probably easier to follow, understand, and handle.
Hi.
So, I'm really impressed by the compression ratios. However, some of the compressed chunks size are larger than uncompressed heap bytes. What could be the explanation behind it. I do have more than 5 tables in my segmentby.
EDIT: 5 columns in segmentby, NOT tables.
Thank you! Prat