search

anhnongdan / bimax-counters

Bimax's counter, splitter and extractor. These are completely independent modules and can be deployed on different hosts. Used for both cbimax and sbimax.
0 stars 0 forks source link

Verify import log lines with Raw files #18

Open anhnongdan opened 7 years ago

anhnongdan commented 7 years ago

Counting and compare give result of cBimax missed about 85% of logs from raw files.

anhnongdan commented 7 years ago

From issue 32 of cBimax:

Verify hit: on PW2 VNPT -> K+ Total sum of all log_link_visit_action_2017_08_27: 1,215,565 on MultiSite: 1,215,565 action.

Do some math on log files: 338M 928,680 vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_19_31 1,367,542 vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_19_16

=> 340MB ~ 1M log line 500MB ~ 1,367,542 line

=> 08/30: from 09_33 to 19_34: 68G => this should be 190.4M hits => MultiSite shows from 0h: 21,8M

On K+: 19M ~ 53K => multiSite: 13K

anhnongdan commented 7 years ago

filter log shows all files are input to queue:

[root@VNPT-HCM-BIMAX-01 ~]# tailf /var/log/filter.log  | grep vtvgolive-appobj.b5695cde.cdnviet.com
Wed Aug 30 22:11:01 ICT 2017:/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_11
Wed Aug 30 22:12:01 ICT 2017:/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_12
Wed Aug 30 22:13:01 ICT 2017:/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_13
Wed Aug 30 22:14:01 ICT 2017:/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_14
Wed Aug 30 22:15:02 ICT 2017:/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_15
Wed Aug 30 22:16:04 ICT 2017:/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_16
Wed Aug 30 22:17:04 ICT 2017:/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_17
Wed Aug 30 22:18:05 ICT 2017:/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_18
Wed Aug 30 22:19:01 ICT 2017:/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_19
Wed Aug 30 22:20:01 ICT 2017:/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_20
anhnongdan commented 7 years ago

monitor Redis on extractor server also give expected result:

[root@VNPT-HCM-BIMAX-03 ~]# echo "monitor" | redis-cli -h 172.20.4.63 -p 6379  | grep vtvgolive-appobj.b5695cde.cdnviet.com
1504107061.456395 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4003" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_31"
1504107061.482951 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4005" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_31"
1504107061.511399 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4001" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_31"
1504107061.533323 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4002" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_31"
1504107061.547131 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4004" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_31"
1504107061.585295 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4006" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_31"
1504107121.361636 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4003" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_32"
1504107121.370045 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4005" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_32"
1504107121.403153 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4001" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_32"
1504107121.414637 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4002" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_32"
1504107121.432472 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4004" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_32"
1504107121.486040 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4006" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_32"
1504107180.995289 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4003" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_33"
1504107180.997417 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4005" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_33"
1504107181.005030 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4001" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_33"
1504107181.032056 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4002" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_33"
1504107181.043064 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4004" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_33"
1504107181.045897 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4006" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_33"
1504107241.227402 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4003" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_34"
1504107241.252028 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4005" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_34"
1504107241.280061 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4001" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_34"
1504107241.295234 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4002" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_34"
1504107241.327406 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4004" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_34"
1504107241.342434 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4006" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_34"
1504107301.458508 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4003" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_35"
1504107301.476426 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4005" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_35"
1504107301.508772 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4001" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_35"
1504107301.534107 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4002" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_35"
1504107301.544754 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4004" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_35"
1504107301.575440 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4006" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_35"
1504107365.053902 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4003" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_36"
1504107365.057699 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4005" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_36"
1504107365.083119 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4001" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_36"
1504107365.097321 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4002" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_36"
1504107365.112663 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4004" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_36"
1504107365.140171 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4006" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_36"
1504107422.294288 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4003" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_37"
1504107422.316101 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4005" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_37"
1504107422.344241 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4001" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_37"
1504107422.367190 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4002" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_37"
1504107422.400245 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4004" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_37"
1504107422.420009 [0 unix:/tmp/redis.sock] "rpush" "queue_extract_4006" "/data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/30/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_30_22_37"
anhnongdan commented 7 years ago

Watching extractor.

PM2 logs sometime show urllib2 messages:

 1|extract_ | 2017-08-31 10:20:03,806: [DEBUG] hit.extension: ts
1|extract_ | 2017-08-31 10:20:03,806: [DEBUG] hit.extension: ts
1|extract_ | 2017-08-31 10:20:03,807: [DEBUG] hit.extension: ts
1|extract_ | 2017-08-31 10:20:03,807: [DEBUG] hit.extension: ts
1|extract_ | 2017-08-31 10:20:03,807: [DEBUG] hit.extension: m3u8
1|extract_ | 2017-08-31 10:20:03,807: [DEBUG] hit.extension: m3u8
1|extract_ | 2017-08-31 10:20:03,790: [DEBUG] <urllib2.Request instance at 0x7f58c0070d88>
1|extract_ | 2017-08-31 10:20:03,812: [DEBUG] <urllib2.Request instance at 0x7f58c409bfc8>
1|extract_ | 2017-08-31 10:20:03,815: [DEBUG] <urllib2.Request instance at 0x7f58c409bab8>
anhnongdan commented 7 years ago

Action: Stop extractor and import logs by hand. Need to bench-marking how long it take to import the whole 1 minute file.

=> Step1: verify importing with 1 thread.

202054 /data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/31/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_31_11_04
202054 requests imported successfully
    0 requests were downloads
    0 requests ignored:
        0 HTTP errors
        0 HTTP redirects
        ... all 0s

But it tooks almost 4 minutes to import a log file for traffic in 1 min.

Step2: modify extract.sh to import a file with multi-thread.

anhnongdan commented 7 years ago

Extractor splitting log file to run parallel.

+ split -l 11047 /data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/31/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_31_11_25
+ find /tmp/tmp.fdT4wcKNPu -type f
+ read ff
+ echo /usr/bin/python /data/app/bimax-counters/extract/queue_piwik_2.1 /tmp/tmp.fdT4wcKNPu/xaa 1 172.40.4.90:4001
+ id=2
+ read ff
+ echo /usr/bin/python /data/app/bimax-counters/extract/queue_piwik_2.1 /tmp/tmp.fdT4wcKNPu/xab 2 172.40.4.90:4001
+ id=3
+ read ff
+ echo /usr/bin/python /data/app/bimax-counters/extract/queue_piwik_2.1 /tmp/tmp.fdT4wcKNPu/xac 3 172.40.4.90:4001
+ id=4
+ read ff
+ echo /usr/bin/python /data/app/bimax-counters/extract/queue_piwik_2.1 /tmp/tmp.fdT4wcKNPu/xad 4 172.40.4.90:4001
+ id=5
+ read ff
+ echo /usr/bin/python /data/app/bimax-counters/extract/queue_piwik_2.1 /tmp/tmp.fdT4wcKNPu/xae 5 172.40.4.90:4001
+ id=6
+ read ff
+ echo /usr/bin/python /data/app/bimax-counters/extract/queue_piwik_2.1 /tmp/tmp.fdT4wcKNPu/xaf 6 172.40.4.90:4001
+ id=7
+ read ff
+ echo /usr/bin/python /data/app/bimax-counters/extract/queue_piwik_2.1 /tmp/tmp.fdT4wcKNPu/xag 7 172.40.4.90:4001
+ id=8
+ read ff
+ echo /usr/bin/python /data/app/bimax-counters/extract/queue_piwik_2.1 /tmp/tmp.fdT4wcKNPu/xah 8 172.40.4.90:4001
+ id=9
anhnongdan commented 7 years ago

extract.sh 47: ll value looks weird:

 echo "`date`:$port process $f" >> $log
                ll=`wc -l $f | awk -v ll=$requeue_ll  '{printf("%.f", $1/ll);}'`
                if [ $ll -eq 0 ];then
                        continue
                fi

On sh -x:

+ echo 'Thu Aug 31 11:25:42 ICT 2017:4001 process /data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/31/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_31_11_25'
++ wc -l /data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/31/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_31_11_25
++ awk -v ll=16 '{printf("%.f", $1/ll);}'
+ ll=11047

Verify:

[root@VNPT-HCM-BIMAX-03 extract]#  wc -l /data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/31/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_31_11_25
269598 /data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/31/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_31_11_25
[root@VNPT-HCM-BIMAX-03 extract]# echo "269598/16" | bc
16849
[root@VNPT-HCM-BIMAX-03 extract]#  wc -l /data/cdn/vtvgolive-appobj.b5695cde.cdnviet.com/2017/08/31/vtvgolive-appobj.b5695cde.cdnviet.com_2017_08_31_11_25 | awk -v ll=16 '{printf("%.f", $1/ll);}'
16850
anhnongdan commented 7 years ago

extract.log: is this too soon to process??

Thu Aug 31 11:51:01 ICT 2017:4005 process /data/cdn/live.thvli.vn/2017/08/31/live.thvli.vn_2017_08_31_11_51
Thu Aug 31 11:51:03 ICT 2017:4005 process /data/cdn/vod.thvli.vn/2017/08/31/vod.thvli.vn_2017_08_31_11_51
Thu Aug 31 11:51:06 ICT 2017:4005 process /data/cdn/vod-geo.thvli.vn/2017/08/31/vod-geo.thvli.vn_2017_08_31_11_51
Thu Aug 31 11:51:07 ICT 2017:4005 process /data/cdn/thvllive.bb87e240.cdnviet.com/2017/08/31/thvllive.bb87e240.cdnviet.com_2017_08_31_11_51
Thu Aug 31 11:51:07 ICT 2017:4005 process /data/cdn/catchup.thvli.vn/2017/08/31/catchup.thvli.vn_2017_08_31_11_51
Thu Aug 31 11:51:13 ICT 2017:4002 process /data/cdn/ott-fpt.kplus.vn/2017/08/31/ott-fpt.kplus.vn_2017_08_31_11_51

Filter doesn't work as expected, log files are input too soon into queue => review: monitor_filter.sh from line 37

Thu Aug 31 12:08:06 ICT 2017:/data/cdn/stream1obj.f6d7c336.vboost.vn/2017/08/31/stream1obj.f6d7c336.vboost.vn_2017_08_31_12_08
Thu Aug 31 12:08:06 ICT 2017:/data/cdn/vtvgovod-web.04477775.sabai.vn/2017/08/31/vtvgovod-web.04477775.sabai.vn_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/media.yomedia.vn/2017/08/31/media.yomedia.vn_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/www.nguoiduatin.vn/2017/08/31/www.nguoiduatin.vn_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/htvvod.1c656bad.cdnviet.com/2017/08/31/htvvod.1c656bad.cdnviet.com_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/vtvgomp4.04477775.sabai.vn/2017/08/31/vtvgomp4.04477775.sabai.vn_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/static.easyvideo.vn/2017/08/31/static.easyvideo.vn_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/hanoitvlive.8ef55e0c.cdnviet.com/2017/08/31/hanoitvlive.8ef55e0c.cdnviet.com_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/vtvgots-webobj.b5695cde.cdnviet.com/2017/08/31/vtvgots-webobj.b5695cde.cdnviet.com_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/stream.easyvideo.vn/2017/08/31/stream.easyvideo.vn_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/img.easyvideo.vn/2017/08/31/img.easyvideo.vn_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/vtvgovod-wap.04477775.sabai.vn/2017/08/31/vtvgovod-wap.04477775.sabai.vn_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/liverestreamusobj.5b1df984.cdnviet.com/2017/08/31/liverestreamusobj.5b1df984.cdnviet.com_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/sslstatic.yomedia.vn/2017/08/31/sslstatic.yomedia.vn_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/image.danet.vn/2017/08/31/image.danet.vn_2017_08_31_12_08
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/btvobj.1c656bad.cdnviet.com/2017/08/31/btvobj.1c656bad.cdnviet.com_2017_08_31_12_07
Thu Aug 31 12:08:07 ICT 2017:/data/cdn/bhd-streamsobj.fad89c7a.cdnviet.com/2017/08/31/bhd-streamsobj.fad89c7a.cdnviet.com_2017_08_31_12_08
anhnongdan commented 7 years ago

Logical processing of monitor_filter.sh is severely wrong. notifywatch detect new files created across multiple CDNs. Current implemented logic can only be used in case there's only 1 CDN.

For multiple CDN, when newest file of CDN A is created, newest file of CDN B that is created previously is pushed to the queue.

 + echo /data/cdn/net2evnobj.8df128ca.cdnviet.com/2017/08/31/net2evnobj.8df128ca.cdnviet.com_2017_08_31_12_33
+ grep -F '\'
+ '[' 1 -eq 0 ']'
+ '[' /data/cdn/net2evnobj.8df128ca.cdnviet.com/2017/08/31/net2evnobj.8df128ca.cdnviet.com_2017_08_31_12_33 = - ']'
++ awk -F= '/filter_queue_log=/ {print $2}' /mnt/app/bimax-counters/monitor/bi_filter.conf
++ head -1
+ log=/var/log/filter.log
++ awk -F= '/filter_queue_list=/ {print $2}' /mnt/app/bimax-counters/monitor/bi_filter.conf
++ head -1
+ requeue_list='queue_extract_4003 queue_extract_4005 queue_extract_4001 queue_extract_4002 queue_extract_4004 queue_extract_4006'
++ awk -F= '/filter_queue_state=/ {print $2}' /mnt/app/bimax-counters/monitor/bi_filter.conf
++ head -1
+ requeue_state=start
+ '[' start == pause ']'
+ '[' -z /data/cdn/vtvgotvcobj.b5695cde.cdnviet.com/2017/08/31/vtvgotvcobj.b5695cde.cdnviet.com_2017_08_31_12_33 ']'
++ date
+ echo 'Thu Aug 31 12:33:13 ICT 2017:/data/cdn/vtvgotvcobj.b5695cde.cdnviet.com/2017/08/31/vtvgotvcobj.b5695cde.cdnviet.com_2017_08_31_12_33'
+ for qq in '$requeue_list'
+ echo 'rpush queue_extract_4003 /data/cdn/vtvgotvcobj.b5695cde.cdnviet.com/2017/08/31/vtvgotvcobj.b5695cde.cdnviet.com_2017_08_31_12_33'
+ /usr/bin/redis-cli -s /tmp/redis.sock
(integer) 1208096
+ for qq in '$requeue_list'
+ echo 'rpush queue_extract_4005 /data/cdn/vtvgotvcobj.b5695cde.cdnviet.com/2017/08/31/vtvgotvcobj.b5695cde.cdnviet.com_2017_08_31_12_33'
anhnongdan commented 7 years ago

Fix monitor_filter notifywatch -> get file name newly created > read CDN list file to find if a file is created for that CDN >

=> No, better use Redis. push file name into Redis queue with cdn domain as key

Work like this:

++ awk -F / '{print $4}'
+ domain=thvllive.bb87e240.cdnviet.com
++ echo 'rpop thvllive.bb87e240.cdnviet.com'
++ /usr/bin/redis-cli -s /tmp/redis.sock
+ last=/data/cdn/thvllive.bb87e240.cdnviet.com/2017/08/31/thvllive.bb87e240.cdnviet.com_2017_08_31_13_06
+ '[' -z /data/cdn/thvllive.bb87e240.cdnviet.com/2017/08/31/thvllive.bb87e240.cdnviet.com_2017_08_31_13_06 ']'
++ date
+ echo 'Thu Aug 31 13:07:41 ICT 2017:/data/cdn/thvllive.bb87e240.cdnviet.com/2017/08/31/thvllive.bb87e240.cdnviet.com_2017_08_31_13_06 push this to redis processing queue'
Thu Aug 31 13:07:41 ICT 2017:/data/cdn/thvllive.bb87e240.cdnviet.com/2017/08/31/thvllive.bb87e240.cdnviet.com_2017_08_31_13_06 push this to redis processing queue
+ echo 'rpush thvllive.bb87e240.cdnviet.com /data/cdn/thvllive.bb87e240.cdnviet.com/2017/08/31/thvllive.bb87e240.cdnviet.com_2017_08_31_13_07'

d594d7b39463d3c1f0f153995d672544b26bd364

anhnongdan commented 7 years ago

monitor_filter log looks good after applying the fix:

Thu Aug 31 13:20:03 ICT 2017:/data/cdn/image.danet.vn/2017/08/31/image.danet.vn_2017_08_31_13_19
Thu Aug 31 13:20:03 ICT 2017:/data/cdn/vtvgo-imgobj.b5695cde.cdnviet.com/2017/08/31/vtvgo-imgobj.b5695cde.cdnviet.com_2017_08_31_13_19
Thu Aug 31 13:20:03 ICT 2017:/data/cdn/token01live.1c656bad.cdnviet.com/2017/08/31/token01live.1c656bad.cdnviet.com_2017_08_31_13_19
Thu Aug 31 13:20:03 ICT 2017:/data/cdn/live.csmtalk.vcdn.vn/2017/08/31/live.csmtalk.vcdn.vn_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/cdn2.coccoc.com/2017/08/31/cdn2.coccoc.com_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/catchup.thvli.vn/2017/08/31/catchup.thvli.vn_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/vod.thvli.vn/2017/08/31/vod.thvli.vn_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/img.easyvideo.vn/2017/08/31/img.easyvideo.vn_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/contentobj.695128ef.cdnviet.com/2017/08/31/contentobj.695128ef.cdnviet.com_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/media_obj.9730a73b.cdnviet.com/2017/08/31/media_obj.9730a73b.cdnviet.com_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/liverestreamobj.5b1df984.cdnviet.com/2017/08/31/liverestreamobj.5b1df984.cdnviet.com_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/sslstatic.yomedia.vn/2017/08/31/sslstatic.yomedia.vn_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/vtvgovod-web.04477775.sabai.vn/2017/08/31/vtvgovod-web.04477775.sabai.vn_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/vtvgolive-web.04477775.sabai.vn/2017/08/31/vtvgolive-web.04477775.sabai.vn_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/vtvgomp4.04477775.sabai.vn/2017/08/31/vtvgomp4.04477775.sabai.vn_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/htvvod.1c656bad.cdnviet.com/2017/08/31/htvvod.1c656bad.cdnviet.com_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/assets.nguoiduatin.vn/2017/08/31/assets.nguoiduatin.vn_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/vtvgovod-wap.04477775.sabai.vn/2017/08/31/vtvgovod-wap.04477775.sabai.vn_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/hanoitvlive.8ef55e0c.cdnviet.com/2017/08/31/hanoitvlive.8ef55e0c.cdnviet.com_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/vod-geo.thvli.vn/2017/08/31/vod-geo.thvli.vn_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/media.yomedia.vn/2017/08/31/media.yomedia.vn_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/stream16.tv.zdn.vn/2017/08/31/stream16.tv.zdn.vn_2017_08_31_13_19
Thu Aug 31 13:20:04 ICT 2017:/data/cdn/vtvgo-img.04477775.sabai.vn/2017/08/31/vtvgo-img.04477775.sabai.vn_2017_08_31_13_19
Thu Aug 31 13:20:05 ICT 2017:/data/cdn/static.easyvideo.vn/2017/08/31/static.easyvideo.vn_2017_08_31_13_19
Thu Aug 31 13:20:05 ICT 2017:/data/cdn/liverestreamusobj.5b1df984.cdnviet.com/2017/08/31/liverestreamusobj.5b1df984.cdnviet.com_2017_08_31_13_19