[Bug] Can't insert to postgresql from janus

[hangocanh2303]

Hi, I used HEPop for receive events from janus webrtc gateway. When janus send event data, hepop received data but it crash with exception Here is the result log:

[04:50:03 PM] ▶️ Initializing PGSql driver [{"host":"localhost","password":"FT5QRbUOflpHMrWqXGUwu39r9cQ","port":5432,"user":"homer_user"}] [04:50:03 PM] ✔ Metrics enabled {"dbName":"homer","expire":300000,"hostname":"localhost:8086","period":30000} [04:50:03 PM] ✔ Metrics enabled {"dbName":"homer","expire":300000,"hostname":"localhost:8086","period":30000} [04:50:03 PM] ▶️ HTTP 10.60.139.130 7777

[04:50:09 PM] ⚪ (xx.xx.xx.xx 23654) [04:50:09 PM] ✔ JSON Data [[{"emitter":""janus"","event":{"data":{"admin_api":false,"event":"connected","ip":"27.67.129.20"},"id":"0x7f7270004530","transport":"janus.transport.websockets"},"timestamp":1581933009867894,"type":128}]][undefined] SET Bucket ID: hep_proto_1000_default [04:50:09 PM] ⚫️️️ (xx.xx.xx.xx 23654) [04:50:09 PM] ⚪ (xx.xx.xx.xx 23656) [04:50:09 PM] ✔ JSON Data [[{"emitter":""janus"","event":{"name":"created","transport":{"id":"0x7f7270004530","transport":"janus.transport.websockets"}},"session_id":8099827284712534,"timestamp":1581933009912767,"type":1}]][undefined] [04:50:09 PM] ⚫️️️ (xx.xx.xx.xx 23656) [04:50:09 PM] ⚪ (xx.xx.xx.xx 23658) [04:50:09 PM] ✔ JSON Data [[{"emitter":""janus"","event":{"name":"attached","plugin":"janus.plugin.echotest"},"handle_id":4672445153156144,"session_id":8099827284712534,"timestamp":1581933009944801,"type":2}]][undefined] [04:50:09 PM] ⚫️️️ (xx.xx.xx.xx 23658) [04:50:09 PM] ✔ PGSQL BULK ID hep_proto_1000_default [04:50:09 PM] ✔ PGSQL BULK Out [{"create_date":"2020-02-17T09:50:09.887Z","data_header":{},"protocol_header":{"address":"xx.xx.xx.xx","family":"IPv4","port":23654},"raw":[{"emitter":""janus"","event":{"data":{"admin_api":false,"event":"connected","ip":"27.67.129.20"},"id":"0x7f7270004530","transport":"janus.transport.websockets"},"timestamp":1581933009867894,"type":128}]},{"create_date":"2020-02-17T09:50:09.916Z","data_header":{},"protocol_header":{"address":"xx.xx.xx.xx","family":"IPv4","port":23656},"raw":[{"emitter":""janus"","event":{"name":"created","transport":{"id":"0x7f7270004530","transport":"janus.transport.websockets"}},"session_id":8099827284712534,"timestamp":1581933009912767,"type":1}]},{"create_date":"2020-02-17T09:50:09.946Z","data_header":{},"protocol_header":{"address":"xx.xx.xx.xx","family":"IPv4","port":23658},"raw":[{"emitter":""janus"","event":{"name":"attached","plugin":"janus.plugin.echotest"},"handle_id":4672445153156144,"session_id":8099827284712534,"timestamp":1581933009944801,"type":2}]}] [04:50:09 PM] Create new tables for id: hep_proto_1000_default [04:50:09 PM] GOT BULK ID:hep_proto_1000_default: [{"protocol_header":{"address":"xx.xx.xx.xx","family":"IPv4","port":23654},"data_header":{},"create_date":"2020-02-17T09:50:09.887Z","raw":[{"emitter":""janus"","type":128,"timestamp":1581933009867894,"event":{"transport":"janus.transport.websockets","id":"0x7f7270004530","data":{"event":"connected","admin_api":false,"ip":"27.67.129.20"}}}]},{"protocol_header":{"address":"xx.xx.xx.xx","family":"IPv4","port":23656},"data_header":{},"create_date":"2020-02-17T09:50:09.916Z","raw":[{"emitter":""janus"","type":1,"timestamp":1581933009912767,"session_id":8099827284712534,"event":{"name":"created","transport":{"transport":"janus.transport.websockets","id":"0x7f7270004530"}}}]},{"protocol_header":{"address":"xx.xx.xx.xx","family":"IPv4","port":23658},"data_header":{},"create_date":"2020-02-17T09:50:09.946Z","raw":[{"emitter":""janus"","type":2,"timestamp":1581933009944801,"session_id":8099827284712534,"handle_id":4672445153156144,"event":{"name":"attached","plugin":"janus.plugin.echotest"}}]}] /root/.nvm/versions/node/v8.9.1/lib/node_modules/hepop/node_modules/pg-promise/lib/formatting.js:164 throw new Error('Property \'' + v.name + '\' doesn\'t exist.'); ^

Error: Property 'sid' doesn't exist. at query.replace.name (/root/.nvm/versions/node/v8.9.1/lib/node_modules/hepop/node_modules/pg-promise/lib/formatting.js:164:19) at String.replace () at Object.object (/root/.nvm/versions/node/v8.9.1/lib/node_modules/hepop/node_modules/pg-promise/lib/formatting.js:144:22) at $formatQuery (/root/.nvm/versions/node/v8.9.1/lib/node_modules/hepop/node_modules/pg-promise/lib/formatting.js:298:29) at format (/root/.nvm/versions/node/v8.9.1/lib/node_modules/hepop/node_modules/pg-promise/lib/formatting.js:837:16) at data.map (/root/.nvm/versions/node/v8.9.1/lib/node_modules/hepop/node_modules/pg-promise/lib/helpers/methods/insert.js:137:26) at Array.map () at Object.insert (/root/.nvm/versions/node/v8.9.1/lib/node_modules/hepop/node_modules/pg-promise/lib/helpers/methods/insert.js:133:29) at Object.insert (/root/.nvm/versions/node/v8.9.1/lib/node_modules/hepop/node_modules/pg-promise/lib/helpers/index.js:61:24) at Object.exports.insert (/root/.nvm/versions/node/v8.9.1/lib/node_modules/hepop/src/pgsql.js:101:34)

Please fix this problem

[lmangani]

Are you sending events in bulk mode from Janus? If so, try switch that off.

[hangocanh2303]

Are you sending events in bulk mode from Janus? If so, try switch that off.

ok, i will try this and reply soon

[hangocanh2303]

When i setting grouping = false in janus setting, HEPop printed error:

[hangocanh2303]

When i created root database in postgresql, HEPop write data to root database myconfig.js file: { "id" : "HEPop101", "socket": "http", "port": 7777, "address": "localhost", "tls": { "key":"/etc/keys/self.key", "cert":"/etc/keys/self.crt" }, "queue": { "timeout": 2000, "maxSize": 1000, "useInterval": true }, "dbName": "homer-data", "tableName": "hep", "db": { "rawSize": 8000, "pgsql":{ "host": "localhost", "port": 5432, "user": "homer_user", "password": "FT5QRbUOflpHMrWqXGUwu39r9cQ" } },

But homer-app show nothing. Why myconfig.js file set homer-data database but HEPop choose root database ?

[hangocanh2303]

ok, edit myconfig.js, add "database" : "homer_data". You need change example config in your project. Now, postgresql insert database success, but i don't see anything in homer-app. Why?

"tableName": "hep", "db": { "rawSize": 8000, "pgsql":{ "host": "localhost", "port": 5432, "database": "homer_data", "user": "homer_user", "password": "FT5QRbUOflpHMrWqXGUwu39r9cQ" } }, I chose Janus default but it's still query {"level":"info","module":"gorm","msg":"SELECT * FROM \"hep_proto_1_default\" WHERE (create_date between $1 AND $2) LIMIT 10","time":"2020-02-18T14:05:27+07:00","type":"sql"} Sometime when i clicked search button, it printed message couldn't retrieve the correct settings for this mapping @adubovikov @lmangani

[adubovikov]

the issue with a message popup has been fixed in the master and we are uploading the new packages right now. Please update and try again: homer-app-1.1.29 (UI 7.7.027)

[hangocanh2303]

Here is result after update to homer-app 1.1.29 and UI 7.7.027, still nothing show in UI

{"level":"info","msg":"Claims","time":"2020-02-18T15:37:44+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582086915 0 0 }}","time":"2020-02-18T15:37:44+07:00"} {"level":"info","msg":"Claims","time":"2020-02-18T15:37:44+07:00"} {"level":"info","module":"gorm","msg":"SELECT FROM \"alias\" WHERE (status = true)","time":"2020-02-18T15:37:44+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T15:37:44+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582086915 0 0 }}","time":"2020-02-18T15:37:44+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582086915 0 0 }}","time":"2020-02-18T15:37:44+07:00"} {"level":"info","module":"gorm","msg":"SELECT FROM \"user_settings\" ","time":"2020-02-18T15:37:44+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T15:37:44+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT count() FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T15:37:44+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T15:37:44+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582086915 0 0 }}","time":"2020-02-18T15:37:44+07:00"} {"level":"info","module":"gorm","msg":"SELECT FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T15:37:44+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT count() FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T15:37:44+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T15:37:44+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582086915 0 0 }}","time":"2020-02-18T15:37:44+07:00"} {"level":"info","module":"gorm","msg":"SELECT FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T15:37:44+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT count() FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T15:37:44+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT * FROM \"hep_proto_1_default\" WHERE (create_date between $1 AND $2) LIMIT 200","time":"2020-02-18T15:37:44+07:00","type":"sql"}

[adubovikov]

confirmed. Fixing...

[adubovikov]

homer-app package 1.1.31 has been uploaded. Please update and retest.

[hangocanh2303]

after i updated to new version 1.1.31, still nothing show, but it's can be update infomation for you

You will see, now, it's query to right table hep_proto_1000_default {"level":"info","module":"gorm","msg":"SELECT FROM \"alias\" WHERE (status = true)","time":"2020-02-18T18:11:16+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:11:16+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT count() FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:11:16+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT FROM \"hep_proto_1000_default\" WHERE (create_date between $1 AND $2) LIMIT 200","time":"2020-02-18T18:11:16+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T18:11:16+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096276 0 0 }}","time":"2020-02-18T18:11:16+07:00"} {"level":"info","msg":" Database Session created ","time":"2020-02-18T18:11:16+07:00"} {"level":"info","msg":"Claims","time":"2020-02-18T18:11:16+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096276 0 0 }}","time":"2020-02-18T18:11:16+07:00"} {"level":"info","module":"gorm","msg":"DELETE FROM \"user_settings\" WHERE (username = $1 AND category = 'dashboard' and param = $2 and partid = $3 )","time":"2020-02-18T18:11:16+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"INSERT INTO \"user_settings\" (\"guid\",\"username\",\"partid\",\"category\",\"create_date\",\"param\",\"data\") VALUES ($1,$2,$3,$4,$5,$6,$7) RETURNING \"user_settings\".\"id\"","time":"2020-02-18T18:11:16+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T18:11:16+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096276 0 0 }}","time":"2020-02-18T18:11:16+07:00"}

///////////////////////////////////////////////////////////////////////////////// {"level":"info","msg":"Claims","time":"2020-02-18T18:26:09+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096468 0 0 }}","time":"2020-02-18T18:26:09+07:00"} {"level":"info","module":"gorm","msg":"SELECT FROM \"alias\" WHERE (status = true)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT count() FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT FROM \"hep_proto_1000_default\" WHERE (create_date between $1 AND $2) LIMIT 200","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T18:26:09+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096468 0 0 }}","time":"2020-02-18T18:26:09+07:00"} {"level":"info","module":"gorm","msg":"SELECT FROM \"alias\" WHERE (status = true)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T18:26:09+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096468 0 0 }}","time":"2020-02-18T18:26:09+07:00"} {"level":"info","module":"gorm","msg":"SELECT FROM \"hep_proto_1000_default\" WHERE (create_date between $1 AND $2) LIMIT 200","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT FROM \"alias\" WHERE (status = true)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T18:26:09+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096468 0 0 }}","time":"2020-02-18T18:26:09+07:00"} {"level":"info","msg":"Claims","time":"2020-02-18T18:26:09+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096468 0 0 }}","time":"2020-02-18T18:26:09+07:00"} {"level":"info","module":"gorm","msg":"SELECT FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT FROM \"alias\" WHERE (status = true)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT FROM \"hep_proto_1000_default\" WHERE (create_date between $1 AND $2) LIMIT 200","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT count() FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT FROM \"hep_proto_1000_default\" WHERE (create_date between $1 AND $2) LIMIT 200","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T18:26:09+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096468 0 0 }}","time":"2020-02-18T18:26:09+07:00"} {"level":"info","module":"gorm","msg":"SELECT FROM \"alias\" WHERE (status = true)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT FROM \"hep_proto_1000_default\" WHERE (create_date between $1 AND $2) LIMIT 200","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T18:26:09+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096468 0 0 }}","time":"2020-02-18T18:26:09+07:00"} {"level":"info","module":"gorm","msg":"SELECT FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T18:26:09+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096468 0 0 }}","time":"2020-02-18T18:26:09+07:00"} {"level":"info","module":"gorm","msg":"SELECT count() FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT FROM \"alias\" WHERE (status = true)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT FROM \"hep_proto_1000_default\" WHERE (create_date between $1 AND $2) LIMIT 200","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T18:26:09+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096468 0 0 }}","time":"2020-02-18T18:26:09+07:00"} {"level":"info","module":"gorm","msg":"SELECT FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT count() FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T18:26:09+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096468 0 0 }}","time":"2020-02-18T18:26:09+07:00"} {"level":"info","module":"gorm","msg":"SELECT FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT count() FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","msg":"Claims","time":"2020-02-18T18:26:09+07:00"} {"level":"info","msg":"\u0026{admin true false { 1582096468 0 0 }}","time":"2020-02-18T18:26:09+07:00"} {"level":"info","module":"gorm","msg":"SELECT FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:26:09+07:00","type":"sql"} {"level":"info","module":"gorm","msg":"SELECT count() FROM \"mapping_schema\" WHERE (partid = $1)","time":"2020-02-18T18:26:09+07:00","type":"sql"}

[adubovikov]

can you please do:

pg_dump homer_data -t hep_proto_1000_default > dump.sql

and share the file with us ? We will re-check

[hangocanh2303]

can you please do:

pg_dump homer_data -t hep_proto_1000_default > dump.sql

and share the file with us ? We will re-check

Here is proto_100 and proto_1000 file dump dump.sql for proto_100, and dump_100 for proto_1000

dump.zip

[adubovikov]

so, once the issues here https://github.com/sipcapture/HEPop/issues/19 will be fixed and we build the new package for homer-app/UI ti will work like here:

[hangocanh2303]

I will try this tommorrow and reply you soon.

[lmangani]

19 has been patched, pending confirmation here - Thanks!

[hangocanh2303]

so, once the issues here #19 will be fixed and we build the new package for homer-app/UI ti will work like here:

yum install homer-app seem like still 1.1.31, old version.

[adubovikov]

https://github.com/sipcapture/homer-app/releases/tag/1.1.32 also the packages are in the repository. Please update

[hangocanh2303]

https://github.com/sipcapture/homer-app/releases/tag/1.1.32 also the packages are in the repository. Please update

unlucky, it still the same result yesterday.

And here is dump database 1000_default. dump_1000.zip You can check it again

[hangocanh2303]

Flollowing description here: so, please do:

homer-app -populate-table-db-config -force-populate Go to F12 and remove everything:

Under Application -> Local Storage. Under Application -> Session Storage.

Clean Javascript cache

and test again.

unlucky, homer-ui still same error yesterday. Here is HEPop log and dump file log.zip

[adubovikov]

1. Please go to Settings -> Mapping make screenshot and send to us.
2. Click on default JANUS copy Fields Mapping and send to us.

[hangocanh2303]

1. Please go to Settings -> Mapping make screenshot and send to us.
2. Click on default JANUS copy Fields Mapping and send to us.

ok, I will show this for you after 2 days. This time,I can't connect from my pc to server. I will show mapping here.

[adubovikov]

Just uploaded a new release 1.1.35 UI 7.7.030 please install

[hangocanh2303]

Just uploaded a new release 1.1.35 UI 7.7.030 please install

Now, It's ok after update 1.1.35

[adubovikov]

super. I will close the ticket

[hangocanh2303]

super. I will close the ticket

Thank you. 👍