bug: VictoriaLogs - missing records (but present in Loki)

[arseny-zinchenko]

Chart name and version Helm chart: victoria-logs-single. version: v0.6.3 Docker image: victoriametrics/victoria-logs:v0.35.0-victorialogs Grafana VictoriaLogs Data source version: 0.5.0

Describe the bug We have a log records with a big body field. They are written to the Loki, but can't be found in VMLogs.

Loki query: {app="backend-api", namespace="prod-backend-api-ns"} |= "b4c2ecd2-000b-46e5-a448-0983963c0688"

• returns desired records

VMlogs query: app:="backend-api" namespace:="prod-backend-api-ns" "b4c2ecd2-000b-46e5-a448-0983963c0688"

• returns noting

If query by an Amazon X-Ray ID related to the records, then:

Loki query: {app="backend-api", namespace="prod-backend-api-ns"} |= "Root=1-670e6e90-2e398c1005aa39f076ec1428"

• returns 8 results, including the record with the big body field

VMlogs query: app:="backend-api" namespace:="prod-backend-api-ns" "b4c2ecd2-000b-46e5-a448-0983963c0688"

• returns 7 results, and no record with the big body field

Custom values Please provide only custom values (excluding default ones):

  grafana:
    enabled: true
   ...
    env:
      GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: "victorialogs-datasource"
    ...
    plugins:
     ...
      - https://github.com/VictoriaMetrics/victorialogs-datasource/releases/download/v0.5.0/victorialogs-datasource-v0.5.0.zip;victorialogs-datasource

[AndrewChubatiuk]

do you observe the same situation using both grafana and vmui?

[arseny-zinchenko]

@AndrewChubatiuk Ha, no! Can found the record using the app:="backend-api" namespace:="prod-backend-api-ns" "b4c2ecd2-000b-46e5-a448-0983963c0688" query in the VM UI:

PS Sorry for the wrong repo to create the issues.

[AndrewChubatiuk]

transfered issue to victorialogs-datasource as this issue looks like datasource related

[dmitryk-dk]

Hi @arseny-zinchenko ! Can you share the response you get if you call the victorialogs API ? You can hide sensitive data.

[hagen1778]

@arseny-zinchenko kindly pinging you regarding https://github.com/VictoriaMetrics/victorialogs-datasource/issues/91#issuecomment-2414711370 We need to understand if this is datasource related or vlogs related problem.

[dmitryk-dk]

@arseny-zinchenko kindly pinging you regarding #91 (comment) We need to understand if this is datasource related or vlogs related problem.

I got the JSON, will add the test, and check where the problem is. Will write about my investigation soon

[dmitryk-dk]

@hagen1778 , @arseny-zinchenko i figured out the problem it is related to the Scanner limitation in the golang.

const (
    // MaxScanTokenSize is the maximum size used to buffer a token
    // unless the user provides an explicit buffer with [Scanner.Buffer].
    // The actual maximum token size may be smaller as the buffer
    // may need to include, for instance, a newline.
    MaxScanTokenSize = 64 * 1024

    startBufSize = 4096 // Size of initial allocation for buffer.
)

// NewScanner returns a new [Scanner] to read from r.
// The split function defaults to [ScanLines].
func NewScanner(r io.Reader) *Scanner {
    return &Scanner{
        r:            r,
        split:        ScanLines,
        maxTokenSize: MaxScanTokenSize,
    }
}

I will fix this problem soon.

[hagen1778]

https://github.com/VictoriaMetrics/victorialogs-datasource/pull/95 has been merged and will be available in the next release.