timescale / tsbs

Time Series Benchmark Suite, a tool for comparing and evaluating databases for time series data
MIT License
1.24k stars 293 forks source link

TimescaleDB'compression policy is not effective after inserting data in iot test #227

Closed tomchon closed 1 year ago

tomchon commented 1 year ago

[question]

compression policy cann't work in iot test when data has been inserted

[step]

insert data:

cat /data2//load_data_iot_host//data_timescaledb_iot_scale4000_2016-01-01T00:00:00Z_2016-01-05T00:00:00Z_interval10s_123.dat.gz| gunzip | tsbs_load_timescaledb  --workers=12  --batch-size=10000 --db-name=benchmarkiot  --host=ip-172-31-23-148  --pass=password --chunk-time=8h --hash-workers=false 

add compression policy:

ALTER TABLE diagnostics SET (timescaledb.compress, timescaledb.compress_orderby = 'time DESC,fuel_state',  timescaledb.compress_segmentby = 'tags_id');  
ALTER TABLE readings SET (timescaledb.compress, timescaledb.compress_orderby = 'time DESC,latitude',  timescaledb.compress_segmentby = 'tags_id'); 
SELECT add_compression_policy('diagnostics', INTERVAL '12 hours'); 
SELECT add_compression_policy('readings', INTERVAL '12 hours');

result

benchmarkiot=#  SELECT chunk_name, range_start, range_end, is_compressed FROM timescaledb_information.chunks;
    chunk_name     |      range_start       |       range_end        | is_compressed 
-------------------+------------------------+------------------------+---------------
 _hyper_1_1_chunk  | 2016-01-01 00:00:00+00 | 2016-01-01 08:00:00+00 | f
 _hyper_2_2_chunk  | 2016-01-01 00:00:00+00 | 2016-01-01 08:00:00+00 | f
 _hyper_2_3_chunk  | 2016-01-01 08:00:00+00 | 2016-01-01 16:00:00+00 | f
 _hyper_1_4_chunk  | 2016-01-01 08:00:00+00 | 2016-01-01 16:00:00+00 | f
 _hyper_2_5_chunk  | 2016-01-01 16:00:00+00 | 2016-01-02 00:00:00+00 | f
 _hyper_1_6_chunk  | 2016-01-01 16:00:00+00 | 2016-01-02 00:00:00+00 | f
 _hyper_1_7_chunk  | 2016-01-02 00:00:00+00 | 2016-01-02 08:00:00+00 | f
 _hyper_2_8_chunk  | 2016-01-02 00:00:00+00 | 2016-01-02 08:00:00+00 | f
 _hyper_1_9_chunk  | 2016-01-02 08:00:00+00 | 2016-01-02 16:00:00+00 | f
 _hyper_2_10_chunk | 2016-01-02 08:00:00+00 | 2016-01-02 16:00:00+00 | f
 _hyper_1_11_chunk | 2016-01-02 16:00:00+00 | 2016-01-03 00:00:00+00 | f
 _hyper_2_12_chunk | 2016-01-02 16:00:00+00 | 2016-01-03 00:00:00+00 | f
 _hyper_2_13_chunk | 2016-01-03 00:00:00+00 | 2016-01-03 08:00:00+00 | f
 _hyper_1_14_chunk | 2016-01-03 00:00:00+00 | 2016-01-03 08:00:00+00 | f
 _hyper_2_15_chunk | 2016-01-03 08:00:00+00 | 2016-01-03 16:00:00+00 | f
 _hyper_1_16_chunk | 2016-01-03 08:00:00+00 | 2016-01-03 16:00:00+00 | f
 _hyper_2_17_chunk | 2016-01-03 16:00:00+00 | 2016-01-04 00:00:00+00 | f
 _hyper_1_18_chunk | 2016-01-03 16:00:00+00 | 2016-01-04 00:00:00+00 | f
 _hyper_2_19_chunk | 2016-01-04 00:00:00+00 | 2016-01-04 08:00:00+00 | f
 _hyper_1_20_chunk | 2016-01-04 00:00:00+00 | 2016-01-04 08:00:00+00 | f
 _hyper_2_21_chunk | 2016-01-04 08:00:00+00 | 2016-01-04 16:00:00+00 | f
 _hyper_1_22_chunk | 2016-01-04 08:00:00+00 | 2016-01-04 16:00:00+00 | f
 _hyper_1_23_chunk | 2016-01-04 16:00:00+00 | 2016-01-05 00:00:00+00 | f
 _hyper_2_24_chunk | 2016-01-04 16:00:00+00 | 2016-01-05 00:00:00+00 | f
(24 rows)

benchmarkiot-#   WHERE proc_name='policy_compression';
 job_id |     application_name      | schedule_interval | max_runtime | max_retries | retry_period |      proc_schema      |     proc_name      |  owner   | scheduled | fixed_schedule |                       config                       | next_start | initial_start | hypertable_schema | hypertable_name |     check_schema      |        check_name        
--------+---------------------------+-------------------+-------------+-------------+--------------+-----------------------+--------------------+----------+-----------+----------------+----------------------------------------------------+------------+---------------+-------------------+-----------------+-----------------------+--------------------------
   1000 | Compression Policy [1000] | 04:00:00          | 00:00:00    |          -1 | 01:00:00     | _timescaledb_internal | policy_compression | postgres | t         | f              | {"hypertable_id": 1, "compress_after": "12:00:00"} |            |               | public            | diagnostics     | _timescaledb_internal | policy_compression_check
   1001 | Compression Policy [1001] | 04:00:00          | 00:00:00    |          -1 | 01:00:00     | _timescaledb_internal | policy_compression | postgres | t         | f              | {"hypertable_id": 2, "compress_after": "12:00:00"} |            |               | public            | readings        | _timescaledb_internal | policy_compression_check
(2 rows)

expect

'is_compressed' status is t ,but it can't work normally so how to resolve this problem, timescaledb version :

imescaledb | 2.10.1  
postgresql | 14.7  
tomchon commented 1 year ago

After a while, it inexplicably worked again