long log line → INVALID_FIELD_VALUE_LENGTH

[jobayle]

Dears,

I'm trying to investigate an issue related to long log lines (due to a long stack trace caused by a stack overflow).

It seems that very long lines (exceeding the 500k limit) are truncated to ~32k and are properly indexed on logz.io. But... long lines over 50k (maybe over 32k?) and under 500k result in an invalid payload and are not properly indexed on logz.io :

Sending log line (len=50000)
10:52:04,581 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - Successfully sent bulk to logz.io, size: 51187
Sending log line (len=150000)
10:52:14,824 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - Successfully sent bulk to logz.io, size: 153343
Sending log line (len=200000)
10:52:25,126 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - Successfully sent bulk to logz.io, size: 204384
Sending log line (len=300000)
10:52:30,452 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - Successfully sent bulk to logz.io, size: 306492
Sending log line (len=400000)
10:52:35,909 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - Successfully sent bulk to logz.io, size: 408505
Sending log line (len=500000)
10:52:41,094 |-INFO in io.logz.logback.LogzioLogbackAppender[LogzioLogbackAppender] - Successfully sent bulk to logz.io, size: 33506

I've created a minimal project to play with this limit :

LogzioTest.java

import java.util.InputMismatchException;
import java.util.Scanner;

import org.apache.commons.lang3.RandomStringUtils;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class LogzioTest {

  @SuppressWarnings("java:S106")
  public static void main(String[] args) {
    Logger logger = LoggerFactory.getLogger(LogzioTest.class);
    try ( Scanner sc = new Scanner(System.in)) {
      while (sc.hasNext()) {
        try {
          int toGen = sc.nextInt();
          String toSend = RandomStringUtils.randomPrint(toGen);
          System.out.println("Sending log line (len=" + toSend.length() + ")");
          logger.info(toSend);
        } catch (InputMismatchException ex) {
          System.out.println("Please type the number of chars to log...");
          sc.next();
        }
      }
    }
  }
}

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>eu.jobayle</groupId>
  <artifactId>logzio-test</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>17</maven.compiler.source>
    <maven.compiler.target>17</maven.compiler.target>
    <exec.mainClass>LogzioTest</exec.mainClass>
  </properties>

  <dependencies>
    <dependency>
      <groupId>io.logz.logback</groupId>
      <artifactId>logzio-logback-appender</artifactId>
      <version>1.0.28</version>
    </dependency>

    <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-classic</artifactId>
      <version>1.3.5</version>
    </dependency>

    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
      <version>2.0.6</version>
    </dependency>

    <dependency>
      <groupId>org.apache.commons</groupId>
      <artifactId>commons-lang3</artifactId>
      <version>3.12.0</version>
    </dependency>
  </dependencies>
</project>

logback.xml

<!-- Use debug=true here if you want to see output from the appender itself -->
<!-- Use line=true here if you want to see the line of code that generated this log -->
<configuration debug="true">
    <!-- Use shutdownHook so that we can close gracefully and finish the log drain -->
    <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook"/>
    <appender name="LogzioLogbackAppender" class="io.logz.logback.LogzioLogbackAppender">
        <token>TOKEN_HERE</token>
        <logzioType>myAwesomeType</logzioType>
        <logzioUrl>https://listener-eu.logz.io:8071</logzioUrl>
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>INFO</level>
        </filter>
    </appender>
    <root level="debug">
        <!-- IMPORTANT: This line is required -->
        <appender-ref ref="LogzioLogbackAppender"/>
    </root>
</configuration>

Thank you!

[tamir-michaeli]

Fixed with #90, @jobayle Can you confirm that the issue is solved?

[jobayle]

Issue fixed for us, thank you very much 👍