search

anhnongdan / cBIMAX

Official source code for cBIMAX - the analytics system for CDN
0 stars 0 forks source link

[Deploy] v2.14 for K+ missed archive #13

Closed anhnongdan closed 7 years ago

anhnongdan commented 7 years ago

Archive seems not to be running.

anhnongdan commented 7 years ago

In flow.sh: docker ps|grep pw$id | head -1 | awk '{print $1}'

=> docker ps:

root@VT-HNI-BIMAX-01:~$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
9e750e3ea8c5        bimax_pw1           "/sbin/my_init"     3 days ago          Up 3 days                               bimax_pw5_1
b981cc5435b6        bimax_db1           "/sbin/my_init"     3 days ago          Up 2 days                               bimax_db5_1
a42e9098f804        bimax_redis1        "/sbin/my_init"     3 days ago          Up 3 days                               bimax_redis5_1
4a8fbc40930b        bimax_redis1        "/sbin/my_init"     4 weeks ago         Up 4 weeks                              bimax_redis1_1
14ff95e047a1        bimax_pw1           "/sbin/my_init"     4 weeks ago         Up 4 weeks                              bimax_pw1_1
b8fe02d33f4d        bimax_db1           "/sbin/my_init"     4 weeks ago         Up 4 weeks                              bimax_db1_1
8b1e526550e6        bimax_db1           "/sbin/my_init"     5 weeks ago         Up 5 weeks                              bimax_db3_1
392696545a33        bimax_redis1        "/sbin/my_init"     5 weeks ago         Up 5 weeks                              bimax_redis3_1
e4a4f5c6ea1f        bimax_pw1           "/sbin/my_init"     5 weeks ago         Up 5 weeks                              bimax_pw3_1

=> Docker ps show pw1 instances above pw3 so flow.sh get the correct id for PW1, but when PW5 is started, flow.sh get pw5 id instead of pw1.

=> fix: docker ps|grep pw${id}_ | head -1 | awk '{print $1}' Reload supervisord after change flow.sh

anhnongdan commented 7 years ago

After fix flow.sh, archive is running but data is not correctly importing:

INFO [2017-06-30 03:30:10] SCHEDULED TASKS
INFO [2017-06-30 03:30:10] Starting Scheduled tasks...
INFO [2017-06-30 03:30:10] Scheduler: executing task Piwik\Plugins\CoreAdminHome\Tasks.purgeOutdatedArchives...
INFO [2017-06-30 03:30:10] Purging archives in 451 archive tables.
INFO [2017-06-30 03:30:10] Deleted 829 rows in archive tables (numeric + blob) for 2017-06-15.
MariaDB [pw1]> select max(server_time) from piwik_log_link_visit_action;
+---------------------+
| max(server_time)    |
+---------------------+
| 2017-06-26 03:09:07 |
+---------------------+
1 row in set (0.00 sec)

MariaDB [pw1]> select max(server_time) from piwik_log_link_visit_action_tracker;
+---------------------+
| max(server_time)    |
+---------------------+
| 2017-06-30 03:54:09 |
+---------------------+
1 row in set (0.15 sec)

MariaDB [pw1]> select min(server_time) from piwik_log_link_visit_action;
+---------------------+
| min(server_time)    |
+---------------------+
| 2017-06-23 11:50:00 |
+---------------------+
1 row in set (0.11 sec)

MariaDB [pw1]> select min(server_time) from piwik_log_link_visit_action_tracker;
+---------------------+
| min(server_time)    |
+---------------------+
| 2017-06-26 03:10:00 |
+---------------------+
1 row in set (0.15 sec)
anhnongdan commented 7 years ago

=> need to replace the db$id to db${id}_ as well.

=> Need to observe how Archive will recover itself.

Need to take into account that for 2.17.1: archive temp is numeric_temp_archive while from 3.0.1 numeric_archive_temp

anhnongdan commented 7 years ago

12

anhnongdan commented 7 years ago

=> reduce job wipe all old tables, this is not good for recovery scenarios => rotate when archive and only reduce the rotated 'old tables' => TODO: need to stop reduce (rotate) and recover first:

2017-06-23-13-40 vs  2017-06-29-21-00
create table piwik_log_link_visit_action_2017_06_23_13_40 like piwik_log_link_visit_action;
insert into piwik_log_link_visit_action_2017_06_23_13_40 select * from piwik_log_link_visit_action where server_time >= '2017-06-23 13:40:00' and server_time <= '2017-06-23 13:49:59';
delete piwik_log_link_visit_action from piwik_log_link_visit_action inner join piwik_log_link_visit_action_2017_06_23_13_40 on piwik_log_link_visit_action.idlink_va = piwik_log_link_visit_action_2017_06_23_13_40.idlink_va;
2017-06-23 13 50
2017-06-23-13-50 vs  2017-06-29-21-01
create table piwik_log_link_visit_action_2017_06_23_13_50 like piwik_log_link_visit_action;
insert into piwik_log_link_visit_action_2017_06_23_13_50 select * from piwik_log_link_visit_action where server_time >= '2017-06-23 13:50:00' and server_time <= '2017-06-23 13:59:59';
delete piwik_log_link_visit_action from piwik_log_link_visit_action inner join piwik_log_link_visit_action_2017_06_23_13_50 on piwik_log_link_visit_action.idlink_va = piwik_log_link_visit_action_2017_06_23_13_50.idlink_va;
2017-06-23 14 0
2017-06-23-14-0 vs  2017-06-29-21-01
create table piwik_log_link_visit_action_2017_06_23_14_0 like piwik_log_link_visit_action;
insert into piwik_log_link_visit_action_2017_06_23_14_0 select * from piwik_log_link_visit_action where server_time >= '2017-06-23 14:0:00' and server_time <= '2017-06-23 14:9:59';
delete piwik_log_link_visit_action from piwik_log_link_visit_action inner join piwik_log_link_visit_action_2017_06_23_14_0 on piwik_log_link_visit_action.idlink_va = piwik_log_link_visit_action_2017_06_23_14_0.idlink_va;
2017-06-23 14 10
2017-06-23-14-10 vs  2017-06-29-21-01
create table piwik_log_link_visit_action_2017_06_23_14_10 like piwik_log_link_visit_action;
insert into piwik_log_link_visit_action_2017_06_23_14_10 select * from piwik_log_link_visit_action where server_time >= '2017-06-23 14:10:00' and server_time <= '2017-06-23 14:19:59';
| piwik_log_link_visit_action_2017_06_23_16_0  |
| piwik_log_link_visit_action_2017_06_23_16_10 |
| piwik_log_link_visit_action_2017_06_23_16_20 |
| piwik_log_link_visit_action_2017_06_23_16_30 |
| piwik_log_link_visit_action_2017_06_23_16_40 |
| piwik_log_link_visit_action_2017_06_23_16_50 |
| piwik_log_link_visit_action_2017_06_23_17_0  |
| piwik_log_link_visit_action_2017_06_23_17_10 |
| piwik_log_link_visit_action_2017_06_23_17_20 |
| piwik_log_link_visit_action_tracker
anhnongdan commented 7 years ago

Notice:

anhnongdan commented 7 years ago

Monitor rotation:

MariaDB [pw1]> select min(server_time) from piwik_log_link_visit_action_tracker;
+---------------------+
| min(server_time)    |
+---------------------+
| 2017-06-30 07:10:00 |   //from last rotate
+---------------------+
1 row in set (0.00 sec)

MariaDB [pw1]> select min(server_time) from piwik_log_link_visit_action;
+---------------------+
| min(server_time)    |
+---------------------+
| 2017-06-27 14:10:00 |    //from last (success) archive
+---------------------+
1 row in set (0.03 sec)

MariaDB [pw1]> select max(server_time) from piwik_log_link_visit_action;
+---------------------+
| max(server_time)    |
+---------------------+
| 2017-06-30 07:09:09 |    // from last rotate
+---------------------+
1 row in set (0.03 sec)

MariaDB [pw1]> select max(server_time) from piwik_log_link_visit_action_tracker;
+---------------------+
| max(server_time)    |
+---------------------+
| 2017-06-30 07:17:09 | .  //realtime
+---------------------+
1 row in set (0.00 sec)
anhnongdan commented 7 years ago

fixed on bimax_seed: https://bitbucket.org/anhnongdan/bimax_seed/commits/cb7f38a18f911ae6c81823ef3b5fb70d329014e1