CLAMD - instream(<IP>@51080): Can't allocate memory ERROR

[ravikashyap1]

Describe the bug

Our ClamD service based on clamd docker, running in AWS ECS (backed by Ec2) runs into the below error on certain files during scan:

CLAMD - instream(@51080): Can't allocate memory ERROR LibClamAV Error: cli_malloc(): Attempt to allocate 223513648 bytes. Please report to https://github.com/Cisco-Talos/clamav/issues

• The ECS instance has 16GB of RAM, with 13GB reserved for the scanner container.

• The file being scanned is between 200-800MB in size (PDF)

• MaxFileSize 4096M

• MaxScanSize 4096M

• PCREMaxFileSize 4096M

• This happens only to certain files, and not all files in that size range.

• I can scan much larger files (~1GB) with no errors.

• It is not transient error and is consistently reproducible on the same file

• It is not a noisy neighbor issue - can reproduced on a sandbox quiet ecs cluster

• We currently use: clamav/clamav:0.104.0 tag -- also tried the latest tag - and still see the same issue.

How to reproduce the problem

The file cannot be shared as it is customer data.

Replace this text with the output from the ClamAV command: clamconf -n

I don't have this installed on my local host. How do I get this info to you otherwise?

[micahsnyder]

Hi @ravikashyap1

Regarding clamconf -n, don't worry about it for now.

The error message you're observing is a little misleading.

Some background: When parsing a file that has a field indicating how much memory needs to be allocated to fit the next bit of data from the file into memory, ClamAV cannot trust the file input because it may well say it needs 2 terabytes of memory, for example.

So there's a max-allocation threshold and ClamAV will fail to allocate if the amount the the file claims exceeds that threshold.
Unfortunately, this threshold is non-configurable, and is presently set to 182MB. When scanning very large PDF's, for example, if a section of the file would decompress to more than 182MB, like a really massive compressed image or something -- that would likely trigger this failure.

With the default can limits, you would be unlikely to encounter this error -- but with when scanning files up to 2GB in size (the internal max-filesize is 2GB, despite being set to 4096M in your config), it's far more likely.

Where to go from here... For ClamAV I think we ought to replace our hardcoded CLI_MAX_ALLOCATION limit configurable, and improve the error message to provide some additional context explaining that it's not failing to allocate memory, but that it is refusing to allocate memory because it is exceeding the configurable limit for memory allocation requests from untrusted input (the possibly malicious scan target file).

[micahsnyder]

Related: https://github.com/Cisco-Talos/clamav/issues/522

[ravikashyap1]

Hi @micahsnyder Thank you for the quick response as well as an explanation of the failure. That makes total sense. We figured it was not a sheer memory not available situation but something else that we could not configure/control as user.

Yea I saw the other issue -and since it did not have any traction, I opened a new one with my details.

Would you know when we can expect a fix? (e.g. configurable CLI_MAX_ALLOCATION) ?

[micahsnyder]

Honestly I have no idea. We're in the middle of task planning for the next feature release right now, and while this is a smaller item, it would be a new feature so it should go in a feature release and not a patch release. I'll create an (internal) jira task for it and bring it up in a meeting next week with me team to prioritize it.

For internal reference: Jira is CLAM-1775

[hellfish88]

Hi @micahsnyder Any updates on which release this parameter might be included with? This issue is unfortunately halting a project we're in right now

[jpfourny]

This may be pure speculation, but my observations have led me to believe that many memory allocation errors are specific to certain EC2 AMIs. I have run many experiments that show the same ClamAV Docker image, loaded with the same configuration and virus definitions, behaves differently on some EC2 instances than others. Same machine type, but different AMIs. Specifically, the optimized Amazon Linux flavours seem to be problematic.

[micahsnyder]

We forgot to put a note on this ticket.

We increased the max allocation limit in ClamAV in version 1.0.0 as a response to this ticket. It probably doesn't solve every case of this, but should have reduced frequency of this occurring. We also changed the severity of the message from error to warning and changed the message to better explain what is happening.

PR for this was https://github.com/Cisco-Talos/clamav/pull/723