Improve the structure of the pkmem stats MI output

[hamid-elaosta]

The pkmem json response for mi_json is poorly structured for parsing.

With a dynamic number of threads trying to JSON marshal these in Golang for example is complex.

# http -j :8888/json/get_statistics params==pkmem:
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Length: 3239
Content-Type: application/json
Date: Mon, 17 Jul 2017 10:40:02 GMT

{
    "pkmem:0-fragments": "1675", 
    "pkmem:0-free_size": "16502424", 
    "pkmem:0-max_used_size": "274792", 
    "pkmem:0-real_used_size": "274792", 
    "pkmem:0-total_size": "16777216", 
    "pkmem:0-used_size": "200880",  
    "pkmem:1-fragments": "1688", 
    ...
    "pkmem:1-used_size": "218704", 
    "pkmem:10-fragments": "1690", 
     ...
    "pkmem:10-used_size": "2516512", 
    "pkmem:11-fragments": "1690", 
    ...
    "pkmem:11-used_size": "2516512", 
    "pkmem:12-fragments": "1691", 
    ...
    "pkmem:12-used_size": "2516624", 
    "pkmem:13-fragments": "1690", 
    ...
    "pkmem:13-used_size": "2516512", 
    "pkmem:14-fragments": "1690", 
     ...
    "pkmem:14-used_size": "2516512", 
    "pkmem:15-fragments": "1655", 
    ...
    "pkmem:15-used_size": "2505392", 
    "pkmem:2-fragments": "1676", 
    ...
    "pkmem:2-used_size": "200944", 
     ...
    "pkmem:9-fragments": "1690", 
    ...
    "pkmem:9-used_size": "2516512"
}

These stats share a structure with the shmem: type, but are grouped into a single object. They are also unordered (incorrectly ordered) in this state, which is less of an issue.

In order to make parsing these easier, given the dynamic number of threads, it would more appropriate to separate these into an array of pkmem:X...;

[
{
    "pkmem:0-fragments": "1675", 
    "pkmem:0-free_size": "16502424", 
    "pkmem:0-max_used_size": "274792", 
    "pkmem:0-real_used_size": "274792", 
    "pkmem:0-total_size": "16777216", 
    "pkmem:0-used_size": "200880", 
},
{
    "pkmem:1-fragments": "1688", 
    "pkmem:1-free_size": "16484288", 
    "pkmem:1-max_used_size": "292928", 
    "pkmem:1-real_used_size": "292928", 
    "pkmem:1-total_size": "16777216", 
    "pkmem:1-used_size": "218704", 
},
...
]

[bogdan-iancu]

@hamid-elaosta , the structuring of the MI reply (via json) is strongly correlated with how the statistics are managed in OpenSIPS - you have only 2 levels : groups of statistics and statistics (like pkgmem is the group and the 1-fragments is the statistic in the group). The MI output automatically follows the internal structuring - so, it is impossible to add a new grouping layer (as you suggested) without changing this in the Statistics Manager too. Maybe something easier will be migrate from "pkmem:1-fragments" to "pkmem1:fragments" - to have the "per process" index inside the group name and not in the statistic name. Will this help you ?

[hamid-elaosta]

@bogdan-iancu Thanks for the explanation. The adjustment of the name would help somewhat, in that it would allow me to query each group, but it still leaves me with the issue of needing to know, programatically how many groups there are to parse, based on how many threads are configured.

At least in Golang, neither version allows the use of the automatic JSON decoding in to the relevant object as the pkmem data is not "semantically correct" in its structure.

[bogdan-iancu]

@hamid-elaosta , I have no idea what are the capabilities of Golang when comes to array manipulation. Still, you can see how many processes are by running the "ps" MI command - this will give you a list of all opensips processes, so you can count and see how many pkmem blocks you should have. Does this help ?

[hamid-elaosta]

@bogdan-iancu The output of ps from mi_json is in the format I'd like to see the pkmem data in. It can be parsed automatically without a custom decoder. Regardless of the capabilities of Golang, it would be nice to see the pkmem data in the same format/structure as the ps output. Is that a possibility?

I guess the real "issue" i'm reporting is that the pkmem data should be an array of pkmem blocks of the same structure as the shmem block, with the number of blocks being dependent on the thread count. It would make the information more usable.

[stale[bot]]

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.