Still issues with the database

[MrGlenn-tech]

Hi,

Still seeing that the database dont match what 3cx says on the amount of calls. This is something that i have from the logs. i also attached the file it is mentions

`cdr-240624063837.292153.log
sv_SE
     historyid                  callid  ...      final_dispname missed_queue_calls
0  Call 292153  5acaf74890010000_30134  ...  Niklas Lastname                NaN

[1 rows x 27 columns]
{"historyid":"Call 292153","callid":"5acaf74890010000_30134","duration":"00:02:28","time_start":1719211117000,"time_answered":1719211131000,"time_end":1719211280000,"reason_terminated":"TerminatedByDst","from_no":"0701814467","to_no":"Ext.80039","from_dn":"10020","to_dn":"80039","dial_no":"80039","reason_changed":"ReplacedDst","final_number":"Ext.10007","final_dn":"10007","bill_code":null,"bill_rate":null,"bill_cost":null,"bill_name":null,"chain":"Chain: 070XXXXX;Ext.80039;Ext.10007;","from_type":"Line","to_type":"Queue","final_type":"Extension","from_dispname":70XXXXXX,"to_dispname":"Arvika fastighets  - felanm\u00e4lan & Jour","final_dispname":"Niklas Wilhelmsson","missed_queue_calls":null}

{"cdr_historyid":"Call 292153","abandonned":false,"handling_time_seconds":149,"waiting_time_seconds":14,"call_date":1719187200000,"call_time":"06:38:37","call_week":26,"day_of_week":"m\u00e5ndag","filename":"cdr-240624063837.292153.log"}

Starting new HTTP connection (1): webapi:8000
http://webapi:8000 "GET /api/v1/cdr/historyid/Call%20292153 HTTP/1.1" 404 29
Starting new HTTP connection (1): webapi:8000
http://webapi:8000 "GET /api/v1/cdrdetails/historyid/Call%20292153 HTTP/1.1" 404 29
404
404
Starting new HTTP connection (1): webapi:8000
http://webapi:8000 "POST /api/v1/cdr HTTP/1.1" 422 123
422
b'{"detail":[{"type":"string_type","loc":["body","from_dispname"],"msg":"Input should be a valid string","input":70XXXXX}]}'
cdr detail existant
422
cdr detail existant
Line1: Call 292153,5acaf74890010000_30134,00:02:28,2024/06/24 06:38:37,2024/06/24 06:38:51,2024/06/24 06:41:20,TerminatedByDst,070XXXXX,Ext.80039,10020,80039,80039,ReplacedDst,Ext.10007,10007,,,,,Chain: 0701814467;Ext.80039;Ext.10007;,Line,Queue,Extension,070XXXXXX,Arvika fastighets  - felanmälan & Jour,Niklas Last Name,
Line2:
cdr-240624063837.292153.log`

cdr-240624063837.292153.csv

[dorel14]

Hello

Sorry to hear that , i see what happen , i will have a look at this

[dorel14]

That was due to a value sent by the file , database wait for text and file send numbers

It is fixed in latest Master

Thx for your patience

[MrGlenn-tech]

Aha okey, i git cloned the last realease just now and deleted all my old cdr-files

and still see stange behaviur, i see all the files getting .old but not apperaring in the database and no errors in the logs. please see pictures.

Feels like just some files are getting pushed to the database and some are skipped.

[dorel14]

Hello, in your pictures, i don't see thé docker stack running , i only see the main container look at the SCP folder and list the files it found

Can you stop all thé containers, rename some files by removing .old, and run a docker compose up.

Normal behaviour will be : 1 start postgres 2 wait for it 3 when dB IS ok start webapi 4 start grafana 5 start PG admin 6 start ui 7 start cdrserver container

After this cdrserver will start looking at your scp folder and download files After download, read it , look if data in file is conform to what it waits for and if ok look if call already exist in dB, If not existing, there is a post request to webapi to push data in dB

When ok dB respond by sending back the datas

For each step there are logs in the window

[MrGlenn-tech]

Okey looking at the logs now, and i see

Starting new HTTP connection (1): webapi:8000 http://webapi:8000 "GET /api/v1/cdr/historyid/Call%20293336 HTTP/1.1" 404 29 Starting new HTTP connection (1): webapi:8000 http://webapi:8000 "GET /api/v1/cdrdetails/historyid/Call%20293336 HTTP/1.1" 404 29 404 404 Starting new HTTP connection (1): webapi:8000 http://webapi:8000 "POST /api/v1/cdr HTTP/1.1" 422 123

i can see that some data is getting pushed to db but not all calls.

[dorel14]

Ok , so i need more logs to see what some files are ignorés, i think some data is not what WE wait for Can you give more logs around thé ones you send me before and add thé file which is passed without intégration ?

[dorel14]

Issue was closed by my commit

I have made some changes in latest master to improve file data checking also in docker-compose.yml so that you don't have to destroy container , just pulling master wil be ok to get lastest code

[MrGlenn-tech]

Did a new gitclone,

And got issues with the timezone in the logs about Time zone Europe/Stockhom (forgott to copy the logs sorry) then as a test i switched to TZ=Europe/Paris as a test.

And now it all seems to be working alot better to be honest i can see what i think is the correct values in the db. having some issues confirming this since it is the middle of the day.

I can get back to you tomorrow morning to confirm if it is the correct values since it is so easy for me to count then, with a report from 3cx and a qury to the db to see if they match up.

[dorel14]

Ok , no problem

I will test with you files and you timezone to see what happen

If you can fond some logs, it will be easier to debug

[MrGlenn-tech]

Hi,

There is not so much diffrenc now between the db and 3cx report there are a few calls missing from the db, i, see some 404 in the logs like this bellow

cdr-240628061140.294469.log sv_SE historyid callid ... final_dispname missed_queue_calls 0 Call 294469 a18e785d90010000_32452 ... Niklas Lastname NaN [1 rows x 27 columns] {"historyid":"Call 294469","callid":"a18e785d90010000_32452","duration":"00:00:49","time_start":1719547900000,"time_answered":1719547910000,"time_end":1719547959000,"reason_terminated":"TerminatedByDst","from_no":"0764154709","to_no":"Ext.80014","from_dn":"10020","to_dn":"80014","dial_no":"80014","reason_changed":"ReplacedDst","final_number":"Ext.10007","final_dn":"10007", "bill_code":null,"bill_rate":null,"bill_cost":null,"bill_name":null,"chain":"Chain: 076XXXXXX;Ext.80014;Ext.10007;","from_ty pe":"Line","to_type":"Queue","final_type":"Extension","from_dispname":"076XXXXXX","to_dispname":"Roseng\u00e5rd fastigheter","final_dispname":"Niklas Lastname","missed_queue_calls":null} {"cdr_historyid":"Call 294469","abandonned":false,"handling_time_seconds":49,"waiting_time_seconds":10,"call_date":1719532800000,"call_time":"06:11:40","call_week":26,"day_of_week":"fredag","filename":"cdr-240628061140.294469.log"} Starting new HTTP connection (1): webapi:8000 http://webapi:8000 "GET /api/v1/cdr'/historyid/'Call%20294469 HTTP/1.1" 404 22 Starting new HTTP connection (1): webapi:8000 http://webapi:8000 "GET /api/v1/cdrdetails/historyid/Call%20294469 HTTP/1.1" 404 29 Status get cdr: 404 Status get cdrdetail 404 Starting new HTTP connection (1): webapi:8000 http://webapi:8000 "POST /api/v1/cdr HTTP/1.1" 200 719 Statut get cdr 200 Texte statut get cdr b'{"historyid":"Call 294469","callid":"a18e785d90010000_32452","duration":"00:00:49","time_start":"2024-06-28T04:11:40","time_answered":"2024-06-28T04:11:50","time_end":"2024-06-28T04:12:39","reason_terminated":"TerminatedByDst", "from_no":"0764154709","to_no":"Ext.80014","from_dn":"10020","to_dn":"80014","dial_no":"80014","reason_changed":"ReplacedDst","final_number":"Ext.10007","final_dn":"10007","bill_code":null,"bill_rate":null,"bill_cost":null,"bill_name":null,"chain":"C hain: 0764154709;Ext.80014;Ext.10007;","from_type":"Line","to_type":"Queue","final_type":"Extension","from_dispname":"0764154 709","to_dispname":"Roseng\xc3\xa5rd fastigheter","final_dispname":"Niklas Wilhelmsson","missed_queue_calls":null,"id":668}' Starting new HTTP connection (1): webapi:8000 http://webapi:8000 "POST /api/v1/cdrdetails HTTP/1.1" 200 239 200 b'{"cdr_historyid":"Call 294469","abandonned":false,"handling_time_seconds":49,"waiting_time_seconds":10,"call_date":"2024-06-28","call_time":"06:11:40","call_week":26,"day_of_week":"fredag","filename":"cdr-240628061140.294469.log","id":668}' 200 200 Line1: Call 294469,a18e785d90010000_32452,00:00:49,2024/06/28 06:11:40,2024/06/28 06:11:50,2024/06/28 06:12:39,TerminatedByDst,0764154709,Ext.80014,10020,80014,80014,ReplacedDst,Ext.10007,10007,,,,,Chain: 0764154709;Ext.80014;Ext.10007;,Line,Queue,Ex tension,0764154709,Rosengård fastigheter,Niklas Wilhelmsson, cdr-240628061140.294469.log

[dorel14]

Hello, happy to ear that we are Bear the solution

404 After get request are "normal" because i make this to look if the call exist in dB and if not it return a 404

Can you try to find a call not in dB and send me the in order to debug why ?

[dorel14]

For the timezone , it's just a missing letter Europe/Stockhom => Europe/Stockholm , with all the letters in Stockholm : no problem with your files

[MrGlenn-tech]

Okey so i cahnged the time-zone and it now works as expected, but i think i found the reason why im seeing the reuslts i am, in the cdr-files im seeing the time for a call as 07:41:49 and the time for the call was 07/03/2024 9:41:49 AM So im guessing that the issue is comming from 3cx since it is not 24H Clock.. and therefor the time is not correct and so my results in the DB is of by 2h

[dorel14]

Hi , perhaps it's due to the conversion to UTC Time, i will look at this later

I create a New issue for this and close this one