search

sipcapture / homer-app

HOMER 7.x Front-End and API Server
http://sipcapture.io
GNU Affero General Public License v3.0
205 stars 84 forks source link

How does one update the Call search -> Logs -> Loki strings? #397

Closed systemcrash closed 3 years ago

systemcrash commented 4 years ago

LogQL field is pre-filled with: {job="heplify-server"} - how does one change this?

Could not find it ...

lmangani commented 4 years ago

Hi Paul, are you referring to the Session Logs tab query (which is editable) or to the Loki search widget here?

systemcrash commented 4 years ago

Tab.

It seems like this label is hard-coded: this should be made configurable.

Loki 2 just arrived, and the label formatting has been improved, esp for people who use syslog like me:

e.g.

This does not work: {job="heplify-server"} b8be56f-1219057c747-7be12060@sipgt-0cb09c9b but this does {job="heplify-server"}|="b8be56f-1219057c747-7be12060@sipgt-0cb09c9b"

So how can I tweak this in Homer? Possible today?

systemcrash commented 4 years ago

https://grafana.com/blog/2020/10/28/loki-2.0-released-transform-logs-as-youre-querying-them-and-set-up-alerts-within-loki

lmangani commented 4 years ago

Tab query is editable and user is expected to change it to their likes. This said, we can't assume everyone is immediately using loki v2 API (which we support also in other elements) so to be sure we need to move this into an advanced variable to be made configurable.

systemcrash commented 4 years ago

Is there some javascript magic that can be run to wrap the call-id in e.g. |="call-id"

For multiple call-ids, something slightly more complex is required.

lmangani commented 4 years ago

yes we can consider adding it in the coming day, but right now, this is more supposed to be a "ready to use" search window into Loki rather than a preset or query builder per se. That's more of a starting point. ezgif com-optimize (76)

lmangani commented 4 years ago

We need to map this template to an advanced variable to make it flexible enough.

systemcrash commented 4 years ago

Ah, OK. Understood - just that the TS source has:

${labels} - and I was wondering whether the user can mutate this before it goes to the Loki LogQL field

lmangani commented 4 years ago 

{job="heplify-server"}|="${labels}"
`` ``
lmangani commented 4 years ago

Might have to change the current | separators too if you expect multiple ids otherwise, but we don't want to assume the backend version or build anything more than a simple template here, so it should be kept template-friendly to be useful to everyone.

systemcrash commented 4 years ago

It seems to me that the heplify-server part, is an assumption that heplify-server handles ingress logs also.

But we're just bringing logs in via syslog, so nothing is tagged, and searching for the call-id with the label as is produces only:

2020-10-29 21:02:19.019
i: zx6ivNkCBwHEjak4QF@sipgt-49c52a68
2020-10-29 21:02:19.018
i: zx6ivNkCBwHEjak4QF@sipgt-49c52a68
2020-10-29 21:02:19.018
i: zx6ivNkCBwHEjak4QF@sipgt-49c52a68
2020-10-29 21:02:19.015
i: zx6ivNkCBwHEjak4QF@sipgt-49c52a68
2020-10-29 21:02:19.014
i: zx6ivNkCBwHEjak4QF@sipgt-49c52a68
2020-10-29 21:02:19.012
i: zx6ivNkCBwHEjak4QF@sipgt-49c52a68
2020-10-29 21:02:09.899
Call-ID: zx6ivNkCBwHEjak4QF@sipgt-49c52a68
2020-10-29 21:02:09.899
Call-ID: zx6ivNkCBwHEjak4QF@sipgt-49c52a68

and not the rest of the desired message parts..... ;)

systemcrash commented 4 years ago 
{job="heplify-server"}|="${labels}"
`` ``

I point out here that: Things in orange you should already be familiar with. Filtering on log content with |= != |~ and !~, and converting your logs into metrics with rate and count_over_time were available in Loki 1.x. Everything else in white is new in 2.0...

lmangani commented 4 years ago

@AlexeyOplachko could we make this template a Loki object variable via advanced settngs?

Example: image

Potential tasks:

systemcrash commented 3 years ago

To be immediately helpful, I note that a |~ i.e. regex match is more useful than |= literal string match.

For calls with multiple legs, labels comprised of a or b form, e.g. a|b should be (must be):

{job="heplify-server"}|~"a|b"

No extra white-space. As {job="heplify-server"}|~"a | b" is not the same as {job="heplify-server"}|~"a|b"

Altho, if I modify the expression that pre-fills the field to the above 'regex or' format, I get everything back: {job="heplify-server"} |~"UVdJEhSH7IHYgB56Ql@ip|UVdJEhSH7IHYgB56Ql@asdf"

buuuut....

if I remove the trailing space after the braces, I get only the desired results (good thing): {job="heplify-server"}|~"UVdJEhSH7IHYgB56Ql@ip|UVdJEhSH7IHYgB56Ql@asdf"

systemcrash commented 3 years ago

Also, all Log results in this window get hacked up for HTML display:

To: &lt;sip:+567... instead of To: <sip:+567...

The JSON returned shows all angle brackets in the headers are intact, however.

systemcrash commented 3 years ago

The HTML returned in the LogQL results is not even used (bloat):

<div class="item-tex-loki hover loki-highlight lokimessage"><span></span>C<span></span>A<span></span>N<span></span>C<span></span>E<span></span>L<span></span> <span></span>s<span></span>i<span></span>p<span></span>:<span></span>s<span></span>t<span></span>p<span></span>k<span></span>v<span></span>r<span></span>t<span></span>@<span></span>1<span></span>0<span></span>.<span></span>4<span></span>8<span></span>.<span></span>8<span></span>.<span></span>2<span></span>0<span></span>:<span></span>3<span></span>2<span></span>7<span></span>6<span></span>8<span></span>;<span></span>l<span></span>i<span></span>n<span></span>e<span></span>=<span></span>i<span></span>7<span></span>x<span></span>e<span></span>2<span></span>k<span></span>g<span></span>x<span></span> <span></span>S<span></span>I<span></span>P<span></span>/<span></span>2<span></span>.<span></span>0<span></span> <span></span>

Plz fix ☹️

AlexeyOplachko commented 3 years ago

The HTML returned in the LogQL results is not even used (bloat):

<div class="item-tex-loki hover loki-highlight lokimessage"><span></span>C<span></span>A<span></span>N<span></span>C<span></span>E<span></span>L<span></span> <span></span>s<span></span>i<span></span>p<span></span>:<span></span>s<span></span>t<span></span>p<span></span>k<span></span>v<span></span>r<span></span>t<span></span>@<span></span>1<span></span>0<span></span>.<span></span>4<span></span>8<span></span>.<span></span>8<span></span>.<span></span>2<span></span>0<span></span>:<span></span>3<span></span>2<span></span>7<span></span>6<span></span>8<span></span>;<span></span>l<span></span>i<span></span>n<span></span>e<span></span>=<span></span>i<span></span>7<span></span>x<span></span>e<span></span>2<span></span>k<span></span>g<span></span>x<span></span> <span></span>S<span></span>I<span></span>P<span></span>/<span></span>2<span></span>.<span></span>0<span></span> <span></span>

Plz fix ☹️

Can you please clarify where you're getting this? Because this doesn't seem to happen neither on Chrome nor on FF for me. I'm getting just text inside of <div class="item-tex-loki hover loki-highlight lokimessage">. Also based on code it shouldn't happen Edit: doing additional investigation

systemcrash commented 3 years ago

Like this: bloatedhtml

systemcrash commented 3 years ago

Span and formatting errors seem fixed in v1.3.0 🤩 Default {job="heplify-server"}|~"..." is much better and works.

Was this supposed to be in there also?:

@AlexeyOplachko could we make this template a Loki object variable via advanced settngs?

AlexeyOplachko commented 3 years ago

Yes, template should look like following: "template": { "lineFilterOperator": "|~", "logStreamSelector": "{job=\"heplify-server\"}" } In advanced setting category: Search param: lokiserver

systemcrash commented 3 years ago

Yes, template should look like following: "template": { "lineFilterOperator": "|~", "logStreamSelector": "{job=\"heplify-server\"}" } In advanced setting category: Search param: lokiserver

Hmm. No worky. ☹️ 

{
    "host": "http://blah:3100"
    "template": { "lineFilterOperator": "|~", 
    "logStreamSelector": "{job=\"something\"}" }
}

Still displays {job="heplify-server"}

systemcrash commented 3 years ago

I also get a stackdump t is null error when I start to type { to trigger a label fetch. I think this is related to https://github.com/sipcapture/homer-app/issues/305

systemcrash commented 3 years ago

In both of these LogQL cases, if I copy/paste the expression in Grafana explore, it returns the results perfectly. So the expressions are fine.

AlexeyOplachko commented 3 years ago

Yes, template should look like following: "template": { "lineFilterOperator": "|~", "logStreamSelector": "{job=\"heplify-server\"}" } In advanced setting category: Search param: lokiserver

Hmm. No worky. ☹️

{
    "host": "http://blah:3100"
    "template": { "lineFilterOperator": "|~", 
    "logStreamSelector": "{job=\"something\"}" }
}

Still displays {job="heplify-server"}

You missed a comma after host param ;) Your example should be like this


{
    "host": "http://blah:3100",
    "template": { "lineFilterOperator": "|~", 
    "logStreamSelector": "{job=\"something\"}" }
}
systemcrash commented 3 years ago

Yup 🤦 fixed. Thank you.

systemcrash commented 3 years ago

I also get a stackdump t is null error when I start to type { to trigger a label fetch. I think this is related to #305

Still get this though, for fetching labels, after I fixed the JSON typo.

This: 

Object { error: TypeError }
main.26cc673dd343aa6e009b.js:formatted:167975
    getVariabls main.26cc673dd343aa6e009b.js:167975
    s main.26cc673dd343aa6e009b.js:105531
    invoke polyfills.a56ce1af579b5175dbe7.js:1
    onInvoke main.26cc673dd343aa6e009b.js:97557
    invoke polyfills.a56ce1af579b5175dbe7.js:1
    run polyfills.a56ce1af579b5175dbe7.js:1
    P polyfills.a56ce1af579b5175dbe7.js:1
    invokeTask polyfills.a56ce1af579b5175dbe7.js:1
    onInvokeTask main.26cc673dd343aa6e009b.js:97548
    invokeTask polyfills.a56ce1af579b5175dbe7.js:1
    runTask polyfills.a56ce1af579b5175dbe7.js:1
    m polyfills.a56ce1af579b5175dbe7.js:1
    invokeTask polyfills.a56ce1af579b5175dbe7.js:1
    _ polyfills.a56ce1af579b5175dbe7.js:1
    k polyfills.a56ce1af579b5175dbe7.js:1
    (Async: EventListener.handleEvent)
    C polyfills.a56ce1af579b5175dbe7.js:1
    scheduleTask polyfills.a56ce1af579b5175dbe7.js:1
    onScheduleTask polyfills.a56ce1af579b5175dbe7.js:1
    scheduleTask polyfills.a56ce1af579b5175dbe7.js:1
    scheduleTask polyfills.a56ce1af579b5175dbe7.js:1
    scheduleEventTask polyfills.a56ce1af579b5175dbe7.js:1
    N polyfills.a56ce1af579b5175dbe7.js:1
    handle main.26cc673dd343aa6e009b.js:113572
    _trySubscribe main.26cc673dd343aa6e009b.js:29667
    subscribe main.26cc673dd343aa6e009b.js:29662
    call main.26cc673dd343aa6e009b.js:30241
    subscribe main.26cc673dd343aa6e009b.js:29662
    c main.26cc673dd343aa6e009b.js:185399
    _innerSub main.26cc673dd343aa6e009b.js:8714
    _tryNext main.26cc673dd343aa6e009b.js:8708
    _next main.26cc673dd343aa6e009b.js:8697
    next main.26cc673dd343aa6e009b.js:9206
    n main.26cc673dd343aa6e009b.js:105766
    _trySubscribe main.26cc673dd343aa6e009b.js:29667
    subscribe main.26cc673dd343aa6e009b.js:29662
    call main.26cc673dd343aa6e009b.js:8682
    subscribe main.26cc673dd343aa6e009b.js:29662
    call main.26cc673dd343aa6e009b.js:109102
    subscribe main.26cc673dd343aa6e009b.js:29662
    call main.26cc673dd343aa6e009b.js:105028
    subscribe main.26cc673dd343aa6e009b.js:29662
    toPromise main.26cc673dd343aa6e009b.js:29719
    C polyfills.a56ce1af579b5175dbe7.js:1
    toPromise main.26cc673dd343aa6e009b.js:29717
    getVariabls main.26cc673dd343aa6e009b.js:167966
    r main.26cc673dd343aa6e009b.js:105550
    C polyfills.a56ce1af579b5175dbe7.js:1
    r main.26cc673dd343aa6e009b.js:105528
    getVariabls main.26cc673dd343aa6e009b.js:167953
    onKeyUpDiv main.26cc673dd343aa6e009b.js:168040
    template main.26cc673dd343aa6e009b.js:168282
    Is main.26cc673dd343aa6e009b.js:93947
    r main.26cc673dd343aa6e009b.js:93958
    _ main.26cc673dd343aa6e009b.js:102837
    invokeTask polyfills.a56ce1af579b5175dbe7.js:1
    onInvokeTask main.26cc673dd343aa6e009b.js:97548
    invokeTask polyfills.a56ce1af579b5175dbe7.js:1
    runTask polyfills.a56ce1af579b5175dbe7.js:1
    invokeTask polyfills.a56ce1af579b5175dbe7.js:1
    _ polyfills.a56ce1af579b5175dbe7.js:1
    k polyfills.a56ce1af579b5175dbe7.js:1
    (Async: EventListener.handleEvent)
    C polyfills.a56ce1af579b5175dbe7.js:1
    scheduleTask polyfills.a56ce1af579b5175dbe7.js:1
    onScheduleTask polyfills.a56ce1af579b5175dbe7.js:1
    scheduleTask polyfills.a56ce1af579b5175dbe7.js:1
    scheduleTask polyfills.a56ce1af579b5175dbe7.js:1
    scheduleEventTask polyfills.a56ce1af579b5175dbe7.js:1
    N polyfills.a56ce1af579b5175dbe7.js:1
    addEventListener main.26cc673dd343aa6e009b.js:103028
    addEventListener main.26cc673dd343aa6e009b.js:102710
    listen main.26cc673dd343aa6e009b.js:102960
    listen main.26cc673dd343aa6e009b.js:47880
    Ps main.26cc673dd343aa6e009b.js:93924
    Fs main.26cc673dd343aa6e009b.js:93883
    template main.26cc673dd343aa6e009b.js:168281
    br main.26cc673dd343aa6e009b.js:92232
    gr main.26cc673dd343aa6e009b.js:92120
    jr main.26cc673dd343aa6e009b.js:92594
    gr main.26cc673dd343aa6e009b.js:92126
    gr main.26cc673dd343aa6e009b.js:92127
    jr main.26cc673dd343aa6e009b.js:92594
    gr main.26cc673dd343aa6e009b.js:92126
    gr main.26cc673dd343aa6e009b.js:92127
    jr main.26cc673dd343aa6e009b.js:92594
    gr main.26cc673dd343aa6e009b.js:92126
    gr main.26cc673dd343aa6e009b.js:92127
    createEmbeddedView main.26cc673dd343aa6e009b.js:93030
    createEmbeddedView main.26cc673dd343aa6e009b.js:93081
    _updateView main.26cc673dd343aa6e009b.js:107940
    set ngIf main.26cc673dd343aa6e009b.js:107925
    Jr main.26cc673dd343aa6e009b.js:92662