Index out of range while trying to get inode

[uri-weisman]

While running trivy version 0.35, I noticed multiple panics due to a runtime error: index out of range, related to this line. This was observed only on a Graviton 3 machine.

Stack trace:

02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] panic: runtime error: index out of range [133514] with length 64
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] goroutine 195 [running]:
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/masahiro331/go-ext4-filesystem/ext4.(*FileSystem).getInode(0x4002205b00, 0x80fbf80b)
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/masahiro331/go-ext4-filesystem@v0.0.0-20221016160854-4b40d7ee6193/ext4/ext4.go:77 +0x404
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/masahiro331/go-ext4-filesystem/ext4.(*FileSystem).listFileInfo(0x4001fc9a29?, 0x13?)
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/masahiro331/go-ext4-filesystem@v0.0.0-20221016160854-4b40d7ee6193/ext4/fs.go:167 +0x11c
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/masahiro331/go-ext4-filesystem/ext4.(*FileSystem).readDirEntry(0x40025bf648?, {0x4001fc9a28, 0x14})
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/masahiro331/go-ext4-filesystem@v0.0.0-20221016160854-4b40d7ee6193/ext4/fs.go:136 +0x1a4
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/masahiro331/go-ext4-filesystem/ext4.(*FileSystem).ReadDir(0x0?, {0x4001fc9a28, 0x14})
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/masahiro331/go-ext4-filesystem@v0.0.0-20221016160854-4b40d7ee6193/ext4/fs.go:93 +0x24
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] io/fs.ReadDir({0xaaaab9354280, 0x4002205b00}, {0x4001fc9a28, 0x14})
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] io/fs/readdir.go:30 +0x2b4
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] io/fs.walkDir({0xaaaab9354280, 0x4002205b00}, {0x4001fc9a28, 0x14}, {0xaaaab93c7ba8, 0x400358d530}, 0x40025bfa90)
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] io/fs/walk.go:73 +0x110
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] io/fs.walkDir({0xaaaab9354280, 0x4002205b00}, {0x4001c97070, 0x4}, {0xaaaab93c7ba8, 0x4006474570}, 0x40025bfa90)
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] io/fs/walk.go:87 +0x258
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] io/fs.walkDir({0xaaaab9354280, 0x4002205b00}, {0xaaaab815635c, 0x1}, {0xaaaab93c6638, 0x4007621050}, 0x40025bfa90)
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] io/fs/walk.go:87 +0x258
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] io/fs.WalkDir({0xaaaab9354280, 0x4002205b00}, {0xaaaab815635c, 0x1}, 0x40025bfa90)
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] io/fs/walk.go:114 +0xc4
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy/pkg/fanal/walker.(*VM).diskWalk(0x4000836960, {0xaaaab815635c, 0x1}, {0xaaaab93d40c0, 0x400255e140})
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy@v0.35.0/pkg/fanal/walker/vm.go:107 +0x1fc
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy/pkg/fanal/walker.(*VM).Walk(0x4000836960, 0x0?, {0xaaaab815635c, 0x1}, 0x0?)
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy@v0.35.0/pkg/fanal/walker/vm.go:78 +0xe8
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy/pkg/fanal/artifact/vm.(*Storage).Analyze(_, {_, _}, _)
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy@v0.35.0/pkg/fanal/artifact/vm/vm.go:48 +0x188
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy/pkg/fanal/artifact/vm.(*EBS).Inspect(_, {_, _})
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy@v0.35.0/pkg/fanal/artifact/vm/ebs.go:61 +0x204
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact({{_, _}, {_, _}}, {_, _}, {{0x4001ffd240, 0x2, 0x2}, {0x4002535a30, ...}, ...})
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy@v0.35.0/pkg/scanner/scan.go:138 +0xa0
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy/pkg/commands/artifact.scan({_, _}, {{{0x0, 0x0}, 0x0, 0x0, 0x1, 0x0, 0x34630b8a000, {0x0, ...}, ...}, ...}, ...)
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy@v0.35.0/pkg/commands/artifact/run.go:578 +0x29c
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact(_, {_, _}, {{{0x0, 0x0}, 0x0, 0x0, 0x1, 0x0, 0x34630b8a000, ...}, ...}, ...)
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy@v0.35.0/pkg/commands/artifact/run.go:247 +0xa0
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).ScanVM(_, {_, _}, {{{0x0, 0x0}, 0x0, 0x0, 0x1, 0x0, 0x34630b8a000, ...}, ...})
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/aquasecurity/trivy@v0.35.0/pkg/commands/artifact/run.go:243 +0xc4
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/elastic/cloudbeat/vulnerability.VulnerabilityScanner.scan({_, _, {_, _}, _, {_, _, _}}, {_, _}, ...)
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/elastic/cloudbeat/vulnerability/scanner.go:124 +0x3f4
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/elastic/cloudbeat/vulnerability.VulnerabilityScanner.ScanSnapshot({0x400258c0e0, 0x4000897ec0, {0xaaaab93daaa0, 0x4001169700}, 0x4000458f00, {0xc10bcef76ff4eb3d, 0xb1957877, 0xaaaabc925880}}, {0xaaaab93c3298, 0x4002cff540}, ...)
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/elastic/cloudbeat/vulnerability/scanner.go:83 +0x11c
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/elastic/cloudbeat/vulnerability.(*VulnerabilityWorker).Run.func4()
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] github.com/elastic/cloudbeat/vulnerability/worker.go:122 +0xa4
02:44:58.335
elastic_agent.cloudbeat
[elastic_agent.cloudbeat][error] created by github.com/elastic/cloudbeat/vulnerability.(*VulnerabilityWorker).Run

[masahiro331]

@uri-weisman Can I get OS information?? or AMI ID.

[uri-weisman]

@masahiro331 We got a better understanding of the issue and it does not seem to be related to an instance type. The above error is thrown when we scan a certain volume, might be an issue with a specific file - will update once we'll understand which file is causing this.

[masahiro331]

@uri-weisman The following differences are currently found.

• Different xfs configurations between amazonlinux 2023 and amazonlinux 2.
• Different xfs configurations between encrypted and unencrypted volumes.
• Differences in data structure between Amazon Linux and other operating systems when using the EBS direct API.

Therefore, I would like to investigate by checking Binary instead of guessing. Is it possible to reproduce the above issue in my environment?

[uri-weisman]

Hey @masahiro331, Unfortunately no, it happens only with a certain volume that does not exist anymore. Promise to update you once we gather new information.

[masahiro331]

Thank you for reply.

For now, I will add workaround for validate the length so we don't panic.