search

Cisco-Talos / clamav

ClamAV - Documentation is here: https://docs.clamav.net
https://www.clamav.net/
GNU General Public License v2.0
4.41k stars 705 forks source link

EICAR is not detected if the magic byte is altered #1354

Closed Pierre-Gronau-ndaal closed 2 months ago

Pierre-Gronau-ndaal commented 2 months ago

Problem

I encountered an issue where ClamAV does not detect the EICAR test file if i alter the original eicar.txt the magic byte of this file to different values.

Eicar: X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*

Expected Behavior

ClamAV should detect and flag the EICAR files as containing the EICAR test virus.

Observed Behavior

ClamAV does not detect the EICAR test virus in several manupulated files. The scan completes without any alerts or detections.

Environment

ClamAV Version: 1.4

Test System: Linux, macOS

Additional Information

Example

EICAR_magic_byte_testfile_00_00_00_0C_6A_50_20_20.jp2

further test files are available here:

https://gitlab.com/ndaal_open_source/ndaal_public_eicar_test_files/-/tree/main/dataset/EICAR_magic_byte?ref_type=heads

How di i create these files

https://gitlab.com/ndaal_open_source/ndaal_public_eicar_test_files/-/blob/main/create_EICAR_magic_byte_testfiles.sh?ref_type=heads

#!/usr/bin/env bash
# Author: Pierre Gronau <Pierre.Gronau@ndaal.eu>
# Copyright 2024
# License: All content is licensed under the terms of <the MIT License>
# Developed on: Debian 12.x; macOS Sonoma x86 architecture
# Tested on: Debian 12.x; macOS Sonoma x86 architecture
# Exit on error. Append "|| true" if you expect an error.
set -o errexit
# Exit on error inside any functions or subshells.
set -o errtrace
# Do not allow use of undefined vars. Use ${VAR:-} to use an undefined VAR
set -o nounset
# Catch the error in case mysqldump fails (but gzip succeeds) in `mysqldump |gzip`
# https://vaneyckt.io/posts/safer_bash
set -o pipefail
# Turn on traces, useful while debugging but commented out by default
# set -o xtrace

# Set $IFS to only newline and tab.
#
# http://www.dwheeler.com/essays/filenames-in-shell.html
# nosemgrep: ifs-tampering
IFS=$'\n\t'

trap cleanup SIGINT SIGTERM ERR EXIT

cleanup() {
    trap - SIGINT SIGTERM ERR EXIT
    printf "%b\n" "\nCleanup is running"
    # Additional cleanup tasks can be added here if needed
}

# Define the EICAR test string
# shellcheck disable=SC2016
input_string='X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*'
readonly input_string

# Create a text file with the EICAR test string and ensure it ends with a newline
printf '%s\n' "${input_string}" > eicar.txt

if [ ! -f eicar.txt ]; then
    printf "Error: eicar.txt not found. Exiting.\n"
    exit 1
fi

# Define magic bytes and their corresponding MIME types
declare -A magic_bytes=(
    ["23 21"]="sh"
    ["02 00 5A 57 52 54"]="cwk"
    ["00 00 02 00 06 04 06 00"]="wk1"
    ["00 00 1A 00 00 10 04 00"]="wk3"
    ["00 00 1A 00 02 10 04 00"]="wk4"
    ["00 00 1A 00 05 10 04"]="123"
    ["00 00 03 F3"]="amiga"
    ["00 00 49 49 58 50 52"]="qxd"
    ["50 57 53 33"]="psafe3"
    ["D4 C3 B2 A1"]="pcap"
    ["4D 3C B2 A1"]="pcap"
    ["0A 0D 0D 0A"]="pcapng"
    ["ED AB EE DB"]="rpm"
    ["53 51 4C 69 74 65 20 66"]="sqlite"
    ["53 50 30 31"]="bin"
    ["49 57 41 44"]="wad"
    ["00"]="pic"
    ["BE BA FE CA"]="dba"
    ["00 01 42 44"]="dba"
    ["00 01 44 54"]="tda"
    ["54 44 46 24"]="tdf"
    ["54 44 45 46"]="tdef"
    ["00 01 00 00"]="palm"
    ["00 00 01 00"]="ico"
    ["69 63 6E 73"]="icns"
    ["66 74 79 70 33 67"]="3gp"
    ["66 74 79 70 68 65 69 63"]="heic"
    ["1F 9D"]="z"
    ["1F A0"]="z"
    ["2D 68 6C 30 2D"]="lzh"
    ["42 41 43 4B 4D 49 4B 45"]="bac"
    ["49 4E 44 58"]="idx"
    ["62 70 6C 69 73 74"]="plist"
    ["42 5A 68"]="bz2"
    ["47 49 46 38 37 61"]="gif"
    ["47 49 46 38 39 61"]="gif"
    ["49 49 2A 00"]="tif"
    ["4D 4D 00 2A"]="tif"
    ["49 49 2A 00 10 00 00 00"]="cr2"
    ["80 2A 5F D7"]="cin"
    ["52 4E 43 01"]="rnc"
    ["4E 55 52 55 49 4D 47"]="nui"
    ["53 44 50 58"]="dpx"
    ["76 2F 31 01"]="exr"
    ["42 50 47 FB"]="bpg"
    ["FF D8 FF DB"]="jpg"
    ["FF D8 FF E0 00 10 4A 46"]="jpg"
    ["FF D8 FF EE"]="jpg"
    ["FF D8 FF E1"]="jpg"
    ["00 00 00 0C 6A 50 20 20"]="jp2"
    ["FF 4F FF 51"]="jp2"
    ["71 6F 69 66"]="qoi"
    ["46 4F 52 4D"]="ilbm"
    ["4C 5A 49 50"]="lz"
    ["30 37 30 37 30 37"]="cpio"
    ["4D 5A"]="exe"
    ["53 4D 53 4E 46 32 30 30"]="ssp"
    ["5A 4D"]="exe"
    ["50 4B 03 04"]="zip"
    ["52 61 72 21 1A 07 00"]="rar"
    ["52 61 72 21 1A 07 01 00"]="rar"
    ["7F 45 4C 46"]="elf"
    ["89 50 4E 47 0D 0A 1A 0A"]="png"
    ["0E 03 13 01"]="hdf4"
    ["89 48 44 46 0D 0A 1A 0A"]="hdf5"
    ["C9"]="com"
    ["CA FE BA BE"]="class"
    ["EF BB BF"]="txt"
    ["FF FE"]="txt"
    ["FE FF"]="txt"
    ["FF FE 00 00"]="txt"
    ["00 00 FE FF"]="txt"
    ["2B 2F 76 38"]="txt"
    ["0E FE FF"]="txt"
    ["DD 73 66 73"]="txt"
    ["FE ED FA CE"]="macho"
    ["FE ED FA CF"]="macho"
    ["FE ED FE ED"]="jks"
    ["CE FA ED FE"]="macho"
    ["CF FA ED FE"]="macho"
    ["25 21 50 53"]="ps"
    ["25 50 44 46 2D"]="pdf"
    ["30 26 B2 75 8E 66 CF 11"]="asf"
    ["24 53 44 49 30 30 30 31"]="sdi"
    ["4F 67 67 53"]="ogg"
    ["38 42 50 53"]="psd"
    ["52 49 46 46"]="avi"
    ["FF FB"]="mp3"
    ["49 44 33"]="mp3"
    ["42 4D"]="bmp"
    ["43 44 30 30 31"]="iso"
    ["4C 00 00 00 01 14 02 00"]="lnk"
    ["62 6F 6F 6B 00 00 00 00"]="alias"
    ["75 73 74 61 72 00 30 30"]="tar"
    ["4D 53 43 46"]="cab"
    ["4B 44 4D"]="vmdk"
    ["43 72 32 34"]="crx"
    ["41 47 44 33"]="fh8"
    ["05 07 00 00 42 4F 42 4F"]="cwk"
    ["00 00 00 14 66 74 79 70"]="mov"
    ["4D 54 68 64"]="mid"
    ["4D 41 52 31 00"]="mar"
    ["4E 45 53 1A"]="nes"
    ["75 73 74 61 72"]="tar"
)

change_magic_byte() {
    local magic_byte="$1"
    local extension="$2"
    local new_file="EICAR_magic_byte_testfile_${magic_byte// /_}.${extension}"

    # Copy the original file
    if ! cp eicar.txt "${new_file}"; then
        printf "Error: Failed to copy eicar.txt to ${new_file}.\n"
        return 1
    fi

    # Replace the beginning with the magic byte
    if ! printf '%b' "$(printf '\\x%s' "${magic_byte//[[:space:]]/\\x}")" | dd of="${new_file}" bs=1 count="${#magic_byte}" conv=notrunc 2>/dev/null; then
        printf "Error: Failed to write magic bytes to ${new_file}.\n"
        return 1
    fi

    # Ensure the file ends with a newline
    if ! sed -i.bak '$a\' "${new_file}"; then
        printf "Error: Failed to add newline to ${new_file}.\n"
        return 1
    fi
    rm -f "${new_file}.bak"

    # Check if the file was created successfully
    if [ ! -f "${new_file}" ]; then
        printf "Error: ${new_file} was not created.\n"
        return 1
    fi

    printf '%s' "${new_file}"
}

# Process all magic bytes
for magic_byte in "${!magic_bytes[@]}"; do
    extension="${magic_bytes[$magic_byte]}"
    new_file=$(change_magic_byte "${magic_byte}" "${extension}")

    if [ $? -ne 0 ]; then
        printf "Error: Failed to process magic byte ${magic_byte}.\n"
        continue
    fi

    # Get file information
    if [ ! -f "${new_file}" ]; then
        printf "Error: ${new_file} does not exist.\n"
        continue
    fi

    size=$(wc -c < "${new_file}" 2>/dev/null)
    if [ $? -ne 0 ]; then
        printf "Error: Failed to get file size for ${new_file}.\n"
        continue
    fi

    mime_type=$(file -b --mime-type "${new_file}" 2>/dev/null)
    if [ $? -ne 0 ]; then
        printf "Error: Failed to get MIME type for ${new_file}.\n"
        continue
    fi

    # Append information to CSV
    printf '%s,%s,%s,%s\n' "${new_file}" "${size}" "${magic_byte}" "${mime_type}" >> EICAR_magic_byte_mime_type.csv
done

# Replace the first 4 bytes with "X5O!" in each created file
for file in EICAR_magic_byte_testfile_*; do
    printf 'X5O!' | dd of="${file}" bs=1 count=4 conv=notrunc
done

# Display file information
printf 'File information:\n'
cat EICAR_magic_byte_mime_type.csv

# List all created files with detailed information
printf '\nDetailed file information:\n'
ls -l eicar.txt EICAR_magic_byte_testfile_*

# Display the content of each file
printf '\nFile contents:\n'
for file in eicar.txt EICAR_magic_byte_testfile_*; do
    printf '\n%s:\n' "${file}"
    cat "${file}"
done

# Compare files using diff
printf '\nFile comparisons:\n'
for file in EICAR_magic_byte_testfile_*; do
    printf '\nComparing eicar.txt with %s:\n' "${file}"
    diff -u eicar.txt "${file}" || true
done

cleanup

script_name1="$(basename "${0}")"
printf "%b\n" "\nscript_name1: ${script_name1}\n"
script_path1="$(realpath "$(dirname "${0}")")"
printf "%b\n" "\nscript_path1: ${script_path1}\n"
script_path_with_name="${script_path1}/${script_name1}"
printf "%b\n" "\nScript path with name: ${script_path_with_name}\n"
printf "%b\n" "\nScript finished\n"
exit 0
HydraDragonAntivirus commented 2 months ago

https://www.virustotal.com/gui/file/9411926ba4f3329336b10b3022e70fda0022727e1cb0a23be2784a2cb9a5e211 only unknown antivruses detected also idea so amazing because you also showing code to detect that.

micahsnyder commented 2 months ago

@Pierre-Gronau-ndaal This is not how EICAR is intended to be used.

Please see my explanation here: https://github.com/Cisco-Talos/clamav/issues/1277#issuecomment-2139686994

Pierre-Gronau-ndaal commented 2 months ago

I did that with real malware as well ! Other vendors recognize it and sorry ClamAV not. It seems that you have in your code a switch for looking of EICAR hashes and this is maybe not the best way …

You can try it by yourself with the following sample

https://github.com/wicar/malware.wicar.org/blob/master/data/vlc_amv.html

https://www.virustotal.com/gui/file/6814c3d7e1d9741555d4bf3d8274a17d838db7be49c0d7a9d6b74bfc15f3a5dc

do you think that the intruders will follow your rules ?

I feel like the hare and the hedgehog and I don't know if I'm the hare

micahsnyder commented 2 months ago

ClamAV is very pedantic about following the eicar detection rules in particular. Some ~7 years ago we had complaints that ClamAV was detecting eicar too easily.

We have a single hash sig for the unmodified eicar, and then have a bytecode signature in bytecode.cvd that matches the eicar pattern plus on allowed variations where whitespace (the space character, tab, LF, CR, CTRL-Z) is added at the end, up to 128 characters.

No other variation is allowed for whole-file eicar detection, although eicar should be detected if attached as a file to a container file, such as included in a zip, attached to an email, attached to an office document, etc.

Other clamav malware detection signatures vary. We don't use hash-based sigs as much as we used to. Most of them are content-match signatures. Most of them are limited to specific target types (roughly equivalent to a file type), which means that changing the magic bytes may prevent the signatures from matching, although some match on any file type.

If you're interested in learning more about clamav signatures, we have documentation here https://docs.clamav.net/manual/Signatures.html

Pierre-Gronau-ndaal commented 2 months ago

thanks for your words …

I looked on your statement: We have a single hash sig for the unmodified eicar, and then have a bytecode signature in bytecode.cvd that matches the eicar pattern plus on allowed variations where whitespace (the space character, tab, LF, CR, CTRL-Z) is added at the end, up to 128 characters.

none of these added characters are detected:

https://www.virustotal.com/gui/file/d3065bd88b816100918d6eb63f9896e06a87cae75201549d95cd5d2571f325d8?nocache=1

https://www.virustotal.com/gui/file/aaf2d071f5f68e5adcba84937a93ee45a01d8e271a1a1c4b6a90a208107d1d14?nocache=1

https://www.virustotal.com/gui/file-analysis/YjdmZTA3MzE3NWJjNGYzNGZkZDEzOTA5MjY5MTkwOTQ6MTcyNTIwMzE4OA==

https://www.virustotal.com/gui/file-analysis/Y2MzZGVjYTQyNTE2OGE3ZDZkYzAwZTVjMjBmYjBkZGI6MTcyNTIwMzUyMA==

micahsnyder commented 2 months ago

@Pierre-Gronau-ndaal thanks for checking. I have somewhat taken this for granted. I'll investigate your findings.