failed to parse json

[LiorBanai]

Hi, I have the following json

{"@timestamp":"2022-11-08T09:36:37.249Z","log.level":"info","message":"['vo_phi_pkg\\\\runtime_recon.py']","ecs":{"version":"1.6.0"},"log":{"logger":"root","origin":{"file":{"line":90,"name":"main.py"},"function":"prepare_logging"},"original":"['vo_phi_pkg\\\\runtime_recon.py']"},"process":{"name":"MainProcess","pid":35436,"thread":{"id":13180,"name":"MainThread"}}}

I'm trying to parser it using EcsDocument.Deserialize(line); but getting the following error:

using versions:

        <PackageReference Include="Elastic.CommonSchema" Version="8.6.0" />
        <PackageReference Include="Elastic.CommonSchema.Serilog" Version="8.6.0" />

what could be the reason?

[Mpdreamz]

The reason is twofold,

The message is not quite ECS compliant since it expands log.origin.* in to objects where these are actually dotted field names under the log: { } object.

This should not break our deserialisation routines though so will create a PR to address this.

I think log fieldset is special enough for us to try and read this expanded form as well.

[Mpdreamz]

Opened https://github.com/elastic/ecs-dotnet/pull/318 to address this.

[LiorBanai]

@Mpdreamz Thanks!

[regev-zaidenstein]

Hi The failed json was produced by python ecs-logging==1.1.0 package. Would updating to latest package (2.0.2) will have it producing the correct format ?

[Mpdreamz]

cc @basepi @beniwohli I think this might be a 'bug' in ecs-logging-python

See: elastic/beats@e9272ad/libbeat/ecs/log.go#L48

ECS log fieldset documentation: https://www.elastic.co/guide/en/ecs/current/ecs-log.html#field-log-origin-file-line

Since origin is not mapped as an object origin.file.name is a dotted field name under log.

Compare with e.g log.syslog which is an object and thefore facility.code is a dotted field name under log: { syslog: { ... } }

https://www.elastic.co/guide/en/ecs/current/ecs-log.html#field-log-syslog

[basepi]

@Mpdreamz Sorry for the delay in my response here. If I'm understanding correctly, you're saying that the python agent will expand origin.file.name into a nested dictionary? I think that diagnosis is correct. How are you dealing with this in dotnet without having a giant list of exceptions?

[Mpdreamz]

If I'm understanding correctly, you're saying that the python agent will expand origin.file.name into a nested dictionary?

Yeah and I believe technically this is not a valid ECS _source structure. E.g beats and ecs-dotnet will produce

{
   "log": {
      "origin.file.name" : ""
   }
}

How are you dealing with this in dotnet without having a giant list of exceptions?

Since everything is backed by types its impossible to produce it any other way: e.g:

https://github.com/elastic/ecs-dotnet/blob/5fe813d2bad337d7efd084a2790968778fe13333/src/Elastic.CommonSchema/FieldSets.Generated.cs#L2379-L2380

We also support setting key values directly, someone can dynamically call:

var ecsDocument = new EcsDocument();
ecsDocument.AssignField("log.origin.file.name", "Test.cs");

For this we generate prop dispatch setters from the ECS specification: https://github.com/elastic/ecs-dotnet/blob/5fe813d2bad337d7efd084a2790968778fe13333/src/Elastic.CommonSchema/PropDispatch.Generated.cs#L1809

Which take care of ensuring the appropriate typed properties are set.

[basepi]

Thanks for the context. I'll think on this.

[LiorBanai]

Hi, When can a new version be released with the support for the python structures?