Tables partitioned incorrectly

[nevian427]

Hi,

Table rotations done by heplify-server sets incorrect time(local timezone?), e.g. p20180404_pnr0 1522789200 (Tue, 03 Apr 2018 21:00:00 GMT) must be 1522886400 (Thu, 05 Apr 2018 00:00:00 GMT)

And on day 2018-04-04 it can't write as driver error (1526): Table has no partition for value 1522886218

[negbie]

When I check partition 20180427_pnr107 (LESS THAN (1524844800)) with:

SELECT table_rows as 'count()' FROM information_schema.partitions WHERE table_schema = schema() and table_name ='sip_capture_call_20180427' and partition_name = '20180427_pnr107';

I see the number is growing so the right partition is getting used.

[games130]

These are the result i am getting from the similiar command:

mysql> select now(); +---------------------+ | now() | +---------------------+ | 2018-04-30 09:35:07 | +---------------------+

mysql> select date, micro_ts from sip_capture_call_20180430 order by date DESC limit 10; +---------------------+------------------+ | date | micro_ts | +---------------------+------------------+ | 2018-04-30 09:35:19 | 1525052118961388 | | 2018-04-30 09:35:19 | 1525052118646384 | | 2018-04-30 09:35:19 | 1525052118616030 | | 2018-04-30 09:35:19 | 1525052118601044 | | 2018-04-30 09:35:19 | 1525052118531641 | | 2018-04-30 09:35:19 | 1525052118936143 | | 2018-04-30 09:35:19 | 1525052118914296 | | 2018-04-30 09:35:19 | 1525052118883468 | | 2018-04-30 09:35:19 | 1525052118855164 | | 2018-04-30 09:35:19 | 1525052118842800 | +---------------------+------------------+

mysql> show create table sip_capture_call_20180430; /*!50100 PARTITION BY RANGE ( UNIX_TIMESTAMP(date)) (PARTITION 20180430_pnr0 VALUES LESS THAN (1525021200) ENGINE = InnoDB, PARTITION 20180430_pnr1 VALUES LESS THAN (1525024800) ENGINE = InnoDB, PARTITION 20180430_pnr2 VALUES LESS THAN (1525028400) ENGINE = InnoDB, PARTITION 20180430_pnr3 VALUES LESS THAN (1525032000) ENGINE = InnoDB, PARTITION 20180430_pnr4 VALUES LESS THAN (1525035600) ENGINE = InnoDB, PARTITION 20180430_pnr5 VALUES LESS THAN (1525039200) ENGINE = InnoDB, PARTITION 20180430_pnr6 VALUES LESS THAN (1525042800) ENGINE = InnoDB, PARTITION 20180430_pnr7 VALUES LESS THAN (1525046400) ENGINE = InnoDB, PARTITION 20180430_pnr8 VALUES LESS THAN (1525050000) ENGINE = InnoDB, PARTITION 20180430_pnr9 VALUES LESS THAN (1525053600) ENGINE = InnoDB, PARTITION 20180430_pnr10 VALUES LESS THAN (1525057200) ENGINE = InnoDB, PARTITION 20180430_pnr11 VALUES LESS THAN (1525060800) ENGINE = InnoDB, PARTITION 20180430_pnr12 VALUES LESS THAN (1525064400) ENGINE = InnoDB, .. PARTITION 20180430_pnr22 VALUES LESS THAN (1525100400) ENGINE = InnoDB, PARTITION 20180430_pnr23 VALUES LESS THAN (1525104000) ENGINE = InnoDB,

mysql> SELECT PARTITION_NAME, table_rows FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_NAME='sip_capture_call_20180430' AND TABLE_SCHEMA='homer_data' ORDER BY PARTITION_DESCRIPTION ASC; +----------------+------------+ | PARTITION_NAME | table_rows | +----------------+------------+ | 20180430_pnr0 | 57110 | | 20180430_pnr1 | 23211 | | 20180430_pnr2 | 15632 | | 20180430_pnr3 | 10760 | | 20180430_pnr4 | 11603 | | 20180430_pnr5 | 18103 | | 20180430_pnr6 | 65045 | | 20180430_pnr7 | 143889 | | 20180430_pnr8 | 308296 | | 20180430_pnr9 | 326126 | | 20180430_pnr10 | 0 | | 20180430_pnr11 | 0 |

[games130]

When i select on WebUI based on this time range:

This is the result i am getting:-

I check the mysql command log, the query executed was:

2018-04-30T01:58:21.776899Z 10756 Query SET time_zone = "+00:00" 2018-04-30T01:58:21.777430Z 10756 Query SELECT id, date, (micro_ts DIV 1000) as milli_ts, micro_ts,method,reply_reason,ruri,ruri_user,ruri_domain,from_user,from_domain,from_tag, to_user,to_domain,to_tag,pid_user,contact_user,auth_user,callid,callid_aleg,via_1,via_1_branch,cseq,diversion,reason,content_type,auth, user_agent,source_ip,source_port,destination_ip,destination_port,contact_ip,contact_port,originator_ip,originator_port,correlation_id,proto,family,rtp_stat,type,node,'call' as trans,'single' as dbnode FROM sip_capture_call_20180430 WHERE (date BETWEEN FROM_UNIXTIME(1525017600) AND FROM_UNIXTIME(1525103999)) AND ( method = 'INVITE') LIMIT 1000

If i query 00:00 to 01:00

I will get nothing:

Mysql query show that it is checking the wrong table:

2018-04-30T01:57:31.047590Z 10745 Query SET time_zone = "+00:00" 2018-04-30T01:57:31.048154Z 10745 Query SELECT id, date, (micro_ts DIV 1000) as milli_ts, micro_ts,method,reply_reason,ruri,ruri_user,ruri_domain,from_user,from_domain,from_tag, to_user,to_domain,to_tag,pid_user,contact_user,auth_user,callid,callid_aleg,via_1,via_1_branch,cseq,diversion,reason,content_type,auth, user_agent,source_ip,source_port,destination_ip,destination_port,contact_ip,contact_port,originator_ip,originator_port,correlation_id,proto,family,rtp_stat,type,node,'call' as trans,'single' as dbnode FROM sip_capture_call_20180429 WHERE (date BETWEEN FROM_UNIXTIME(1525017600) AND FROM_UNIXTIME(1525021200)) AND ( method = 'INVITE') LIMIT 1000

[games130]

I check on the rotation script done by the original. The tables are partition based on UTC timezone. I believe heplify-server rotation should produce similar partition

/*!50100 PARTITION BY RANGE ( UNIX_TIMESTAMP(date)) (PARTITION p2018050100 VALUES LESS THAN (1525136400) ENGINE = InnoDB, PARTITION p2018050101 VALUES LESS THAN (1525140000) ENGINE = InnoDB, PARTITION p2018050102 VALUES LESS THAN (1525143600) ENGINE = InnoDB, PARTITION p2018050103 VALUES LESS THAN (1525147200) ENGINE = InnoDB, PARTITION p2018050104 VALUES LESS THAN (1525150800) ENGINE = InnoDB, PARTITION p2018050105 VALUES LESS THAN (1525154400) ENGINE = InnoDB, PARTITION p2018050106 VALUES LESS THAN (1525158000) ENGINE = InnoDB, PARTITION p2018050107 VALUES LESS THAN (1525161600) ENGINE = InnoDB, PARTITION p2018050108 VALUES LESS THAN (1525165200) ENGINE = InnoDB, PARTITION p2018050109 VALUES LESS THAN (1525168800) ENGINE = InnoDB, PARTITION p2018050110 VALUES LESS THAN (1525172400) ENGINE = InnoDB, PARTITION p2018050111 VALUES LESS THAN (1525176000) ENGINE = InnoDB, PARTITION p2018050112 VALUES LESS THAN (1525179600) ENGINE = InnoDB, PARTITION p2018050113 VALUES LESS THAN (1525183200) ENGINE = InnoDB, PARTITION p2018050114 VALUES LESS THAN (1525186800) ENGINE = InnoDB, PARTITION p2018050115 VALUES LESS THAN (1525190400) ENGINE = InnoDB, PARTITION p2018050116 VALUES LESS THAN (1525194000) ENGINE = InnoDB, PARTITION p2018050117 VALUES LESS THAN (1525197600) ENGINE = InnoDB, PARTITION p2018050118 VALUES LESS THAN (1525201200) ENGINE = InnoDB, PARTITION p2018050119 VALUES LESS THAN (1525204800) ENGINE = InnoDB, PARTITION p2018050120 VALUES LESS THAN (1525208400) ENGINE = InnoDB, PARTITION p2018050121 VALUES LESS THAN (1525212000) ENGINE = InnoDB, PARTITION p2018050122 VALUES LESS THAN (1525215600) ENGINE = InnoDB, PARTITION p2018050123 VALUES LESS THAN (1525219200) ENGINE = InnoDB,

[negbie]

Ok lets do this. If I will find some time today I will do the changes.

[negbie]

@games130 could you please test the latest commit. This should partition the tables based on utc time.

[games130]

@negbie i am now running the latest commit, table creation so far is ok. will update again tomorrow to see the result when it is executed by the cronjob.

[games130]

@negbie mysql table rotation looks good. thank you 👍

[negbie]

@games130 Thank you for the update. @games130 @nevian427 I will close it now. Feel free to reopen if you find something else.