Parsing Logs With JSON and Non-JSON Payloads

[kieronlanning]

Hi,

I've rather a complex log that I can't change the format of that I'm hoping to parse for some analysis.

Each record looks a little like this:

$>{***DATETIME***} {***LEVEL***}: {***CLASS***}: {***LINE***}  {***Method***}
{***PROPERTY-NAME***}: {***GUID/STRING/INT***}
{***COLLECTIONAME***}:
  {***PROPERTY-NAME***}: {***GUID/STRING/INT***}
  {***PROPERTY-NAME***}: {***GUID/STRING/INT***}
  {***PROPERTY-NAME***}: {***GUID/STRING/INT***}
{*** MULTILINE OR SINGLELINE COMPLEX JSON ***}

The start of the record is always marked by a $>, but the combination of properties, collections and JSON payloads are all optional.

It's all working apart from the JSON part, which I'm only really interested in as a string value, rather than doing anything with it at this stage.

My issue seems to be getting it to stop parsing - it will continue parsing into the next record, instead of correctly detecting a new record.

In my tests if I remove the JSON payload from the logs, it will process all combinations of records without an issue.

Here's an example JSON payload (with bearer tokens and GUIDs replaced):

{
    "@odata.type": "#microsoft.graph.commsNotifications",
    "value": [{
        "@odata.type": "#microsoft.graph.commsNotification",
        "changeType": "deleted",
        "resourceUrl": "/communications/calls/guid",
        "resource": "/app/calls/guid",
        "resourceData": {
            "@odata.type": "#microsoft.graph.call",
            "state": "terminated",
            "resultInfo": {
                "@odata.type": "#microsoft.graph.resultInfo",
                "code": 200,
                "subcode": 0,
                "message": "This conversation has ended as only one participant was remaining in the conversation. DiagCode: 0#5010"
            },
            "meetingInfo": {
                "@odata.type": "#microsoft.graph.tokenMeetingInfo",
                "token": "bearer-token"
            },
            "callChainId": "guid",
            "id": "guid"
        }
    }]
}

[johngit22]

Thanks

On Fri, Aug 21, 2020, 9:04 AM Kieron Lanning notifications@github.com wrote:

Hi,

I've rather a complex log that I can't change the format of that I'm hoping to parse for some analysis.

Each record looks a little like this:

$>{DATETIME} {LEVEL}: {CLASS}: {LINE} {Method} {PROPERTY-NAME}: {GUID/STRING/INT} {COLLECTIONAME}: {PROPERTY-NAME}: {GUID/STRING/INT} {PROPERTY-NAME}: {GUID/STRING/INT} {PROPERTY-NAME}: {GUID/STRING/INT} { MULTILINE OR SINGLELINE COMPLEX JSON }

The start of the record is always marked by a $>, but the combination of properties, collections and JSON payloads are all optional.

It's all working apart from the JSON part, which I'm only really interested in as a string value, rather than doing anything with it at this stage.

My issue seems to be getting it to stop parsing - it will continue parsing into the next record, instead of correctly detecting a new record.

In my tests if I remove the JSON payload from the logs, it will process all combinations of records without an issue.

Here's an example JSON payload (with bearer tokens and GUIDs replaced):

{ "@odata.type": "#microsoft.graph.commsNotifications", "value": [{ "@odata.type": "#microsoft.graph.commsNotification", "changeType": "deleted", "resourceUrl": "/communications/calls/guid", "resource": "/app/calls/guid", "resourceData": { "@odata.type": "#microsoft.graph.call", "state": "terminated", "resultInfo": { "@odata.type": "#microsoft.graph.resultInfo", "code": 200, "subcode": 0, "message": "This conversation has ended as only one participant was remaining in the conversation. DiagCode: 0#5010" }, "meetingInfo": { "@odata.type": "#microsoft.graph.tokenMeetingInfo", "token": "bearer-token" }, "callChainId": "guid", "id": "guid" } }] }

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sprache/Sprache/issues/157, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYXHWKJTDYLYSQPPHYF4TTSBZWEPANCNFSM4QHIONCQ .