Exceptions not logged to Logzio

[Aicun]

Logger.LogError(message) Logs to logzio

Logger.LogError(Exception, message) Calling this method failed to write to logzio

versions: Serilog.Sinks.Logz.Io:2.2.2 Serilog.AspNetCore: 3.2.0

[mantasaudickas]

Can you elaborate more what exactly failed?

[Aicun]

var configuration = GetConfiguration(args); Log.Logger = new LoggerConfiguration() .ReadFrom.Configuration(configuration) .MinimumLevel.Verbose() .CreateLogger();

Log.Logger.Debug("Application is starting.."); //write to Logzio successfully Log.Logger.Error("ERROR message"); //write to Logzio successfully Log.Logger.Error(new Exception("With Exception"), "Test log with exception"); //Failed to write to Logzio

[mantasaudickas]

Do you see any error? Maybe your index at logz.io fails because you already have Exception field with different mapping?

[Aicun]

I didn't get any error. Here is a similar issue with writing log to ElasticSearch: https://github.com/serilog/serilog/issues/384

[mantasaudickas]

To be honest - for me it works fine also in productive system, but will try to investigate a bit more and try to find something.

[Aicun]

In LogzioSink.FormatLogEvent, if I change

var values = new Dictionary<string, object> { {"@timestamp", loggingEvent.Timestamp.ToString("O")}, {"level", level}, {"message", loggingEvent.RenderMessage()}, {"exception", loggingEvent.Exception} }; to var values = new Dictionary<string, object> { {"@timestamp", loggingEvent.Timestamp.ToString("O")}, {"level", level}, {"message", loggingEvent.RenderMessage()}, {"exception", JsonConvert.SerializeObject(loggingEvent.Exception, Newtonsoft.Json.Formatting.None)} }; The exception will be written to Logzio successfully. I think it was not serialized properly.

Following are the same exception serialized by before and after change: Before change: "{\"@timestamp\":\"2020-04-30T17:23:31.9602988-04:00\",\"level\":\"Error\",\"message\":\"Test Test Error\",\"exception\":{\"ClassName\":\"System.Exception\",\"Message\":\"EEEEEEEEEEE\",\"Data\":null,\"InnerException\":{\"ClassName\":\"System.Exception\",\"Message\":\"IIIIIIIIIIIIII\",\"Data\":null,\"InnerException\":null,\"HelpURL\":null,\"StackTraceString\":null,\"RemoteStackTraceString\":null,\"RemoteStackIndex\":0,\"ExceptionMethod\":null,\"HResult\":-2146233088,\"Source\":null,\"WatsonBuckets\":null},\"HelpURL\":null,\"StackTraceString\":null,\"RemoteStackTraceString\":null,\"RemoteStackIndex\":0,\"ExceptionMethod\":null,\"HResult\":-2146233088,\"Source\":null,\"WatsonBuckets\":null}}"

after change: "{\"@timestamp\":\"2020-04-30T17:24:40.6558112-04:00\",\"level\":\"Error\",\"message\":\"Test Test Error\",\"exception\":\"{\\"ClassName\\":\\"System.Exception\\",\\"Message\\":\\"EEEEEEEEEEE\\",\\"Data\\":null,\\"InnerException\\":{\\"ClassName\\":\\"System.Exception\\",\\"Message\\":\\"IIIIIIIIIIIIII\\",\\"Data\\":null,\\"InnerException\\":null,\\"HelpURL\\":null,\\"StackTraceString\\":null,\\"RemoteStackTraceString\\":null,\\"RemoteStackIndex\\":0,\\"ExceptionMethod\\":null,\\"HResult\\":-2146233088,\\"Source\\":null,\\"WatsonBuckets\\":null},\\"HelpURL\\":null,\\"StackTraceString\\":null,\\"RemoteStackTraceString\\":null,\\"RemoteStackIndex\\":0,\\"ExceptionMethod\\":null,\\"HResult\\":-2146233088,\\"Source\\":null,\\"WatsonBuckets\\":null}\"}"

The difference is that in the after change json string ,the whole exception is wrapped within "".

[mantasaudickas]

I don't think its a good idea to wrap exception to string. In that way you lose the whole benefit of structured logging, which means you cannot filter based on exception fields anymore. Are you sure you don't have indexing failures at logz.io side? Just search for: type: logzio-index-failure

[Aicun]

No, I don't have indexing failures in logzio-index-failure, and even if those exceptions were logged to logzio-index-failure, it would be really confusing cause I log the error to type A but were written to type logzio-index-failure. What do you think.

[SeppPenner]

I can see a similar problem. Sometimes, messages are logged to logz.io, sometimes not.

I just send simple heartbeat messages like:

this.logger.Information("Heartbeat at {@date}.", DateTimeOffset.Now);

The configuration looks like this:

var loggerConfiguration = new LoggerConfiguration()
    .MinimumLevel.Override("Microsoft", LogEventLevel.Information)
    .MinimumLevel.Information()
    .Enrich.FromLogContext()
    .Enrich.WithExceptionDetails()
    .Enrich.WithMachineName()
    .WriteTo.Async(a => a.File(Path.Combine("C:\\log", "SomeService.txt"), rollingInterval: RollingInterval.Day, rollOnFileSizeLimit: true))
    .WriteTo.LogzIo(
        "apiKey",
        "my-service",
        new LogzioOptions
        {
            UseHttps = true,
            RestrictedToMinimumLevel = LogEventLevel.Information,
            Period = TimeSpan.FromSeconds(1),
            BatchPostingLimit = 1,
            BoostProperties = false,
            DataCenterSubDomain = "listener-eu"
        });

Log.Logger = loggerConfiguration.CreateLogger();

I have honestly no idea why the logging works sometimes and sometimes not... Any ideas? My heartbeat message is sent every 30 seconds. Before it was every 5 seconds (Nothing changed in case of the reliability).

[frankaxela]

I had the same issue and it was the format of the exception file in Logz.io. the system is waiting for a string and receiving an object, fails the parsing, look for this type in your logs: type: logzio-index-failure

Contacting support from Logz.io should fix this as they can map this field

[SeppPenner]

@frankaxela This might be the case for @Aicun, it's safe not for me as my message looks the same all the time (Right now it's just Heartbeat.). The logzio-index-failure doesn't show the messages either (Just some others from other projects we use which are not related at all).

[mantasaudickas]

Would you be able to enable serilog debugging to check if there are any errors? See https://github.com/serilog/serilog/wiki/Debugging-and-Diagnostics

[frankaxela]

Hi, I can confirm I cannot ship logs to Logz.io configuring the sink in code, previously I had it configured in JSON and worked ok.

More info: No exception is thrown by Serilog nor while posting the log to Logz.io. URLs are the same. I'm debugging your code @mantasaudickas but so far no big difference

[SeppPenner]

In our case, the reason was that we destructured exceptions like this:

logger.Error("Some error occurred: {@ex}", ex);

and this caused a 413 HTTP error: 413 REQUEST ENTITY TOO LARGE.

Somehow, this also broke the sink (And Serilog logging) causing some strange memory leak all the time. We're still investigating further and I will see if I can make some proposals for the sink later.

[SeppPenner]

This https://github.com/mantasaudickas/serilog-sinks-logz-io/pull/16 should help to avoid the issue I described with the 413 error, I guess...

[Synergia503]

I have the same problem as @Aicun had. I got beautiful exception from System.Text.Json:

2021-06-30T10:08:37.7197305Z Event at 2021-06-30T12:08:37.5547493+02:00 with message template error could not be formatted into JSON and will be dropped: System.InvalidOperationException: Method may only be called on a Type for which Type.IsGenericParameter is true. at System.RuntimeType.get_DeclaringMethod() at System.Text.Json.JsonPropertyInfo1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Converters.IEnumerableOfTConverter2.OnWriteResume(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter2.OnTryWrite(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonPropertyInfo1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonPropertyInfo1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonPropertyInfo1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonPropertyInfo1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Converters.IEnumerableOfTConverter2.OnWriteResume(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter2.OnTryWrite(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonPropertyInfo1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Converters.IEnumerableOfTConverter2.OnWriteResume(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter2.OnTryWrite(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonPropertyInfo1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Converters.IEnumerableOfTConverter2.OnWriteResume(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter2.OnTryWrite(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonPropertyInfo1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonPropertyInfo1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonPropertyInfo1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonPropertyInfo1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWriteAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Converters.DictionaryOfTKeyTValueConverter3.OnWriteResume(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Converters.DictionaryDefaultConverter3.OnTryWrite(Utf8JsonWriter writer, TCollection dictionary, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonSerializer.WriteCore[TValue](JsonConverter jsonConverter, Utf8JsonWriter writer, TValue& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonSerializer.WriteCore[TValue](Utf8JsonWriter writer, TValue& value, Type inputType, JsonSerializerOptions options) at System.Text.Json.JsonSerializer.Serialize[TValue](TValue& value, Type inputType, JsonSerializerOptions options) at System.Text.Json.JsonSerializer.Serialize[TValue](TValue value, JsonSerializerOptions options) at Serilog.Sinks.Http.LogzIo.LogzIoTextFormatter.Format(LogEvent logEvent, TextWriter output)

Code snippet try { Logger.Information(""); } catch (Exception exception) { Logger.Error(new Exception("Ex"), "error"); ---> this works Logger.Error(exception, "error"); --- > this does not work }

@mantasaudickas Would you recommend some work-around?

EDIT: I would say this could be somehow related: elastic

[mantasaudickas]

@Synergia503 sorry, at the moment I do not have a workaround for this. It happens also in my projects and so far did not found a solution. Only way for now probably is to implement a possibility to change a serializer to Newtonsoft.Json which does not have this problem. I guess it is more related to this issue: https://github.com/dotnet/runtime/issues/30425 There is also a workaround provided. So will try to check it.

[Synergia503]

@mantasaudickas Thank you for the quick response. For now, as a "workaround" I switched to Newtonsoft.Json which indeed does not have this problem. Maybe I will have the time to check the workaround provided in the 30425, thanks for this link!

[SeppPenner]

@mantasaudickas The System.Text.Json serializer is complete crap if you ask me... At my company, we still use Newtonsoft as it is way more powerful and doesn't have these strange childhood problems...

[mantasaudickas]

@SeppPenner I totally agree with you - however - just wanted to avoid additional dependency..

[mantasaudickas]

I have modified packages to use Newtonsoft.Json. So just update and check again. For further steps, I would recommend to switch to durable http sink. This would ensure that records at least are stored in the file (even if sending fails).

For additional debug ouput you can use such code (can be done once on application startup after logger initialization):

try
{
    if (!Directory.Exists($"c:/temp/logs"))
        Directory.CreateDirectory($"c:/temp/logs");
}
catch (Exception)
{
    // ignore
}

try
{
    var serilog = $"c:/temp/logs/serilog-{DateTime.UtcNow:yyyyMMddHHmm}.log";

    var writer = File.CreateText(serilog);
    writer.AutoFlush = true;

    var textWriter = TextWriter.Synchronized(writer);
    SelfLog.Enable(textWriter);
}
catch (Exception)
{
    // ignore
}

It helps to identify further issues.

[mantasaudickas]

I am closing this issue for now due inactivity. If you still get any issues, feel free to continue discussion.

[angularsen]

I'm seeing a related problem, on 6.0.0 and 7.2.0, where log events are dropped if exceptions are thrown by properties on the included exception object.

Here is a proposed fix that works for us: https://github.com/serilog-contrib/Serilog.Sinks.Logz.Io/pull/35