Illegal instruction(coredump) during clamscan

[nehajain29]

Built clamav 0.104.2 version

Machine details: operating system: AIX/PPC oslevel: 7.2.0.0

what is the issue:

getting Illegal instruction and core during clamscan for binary data. Here 2 threads are trying to clear same data which is causing illegal instruction and core dump.

how is being hit :

We are seeing this issue during clamscan on binary data packages if we are running clamscan on single package then it's working fine but if we are running clamscan on whole directory then getting core.

command: clamscan /opt/freeware/bin/

we have tried with debug command also:

LibClamAV debug: Checking realpath of /opt/freeware/bin/g13_64 LibClamAV debug: Recognized binary data LibClamAV debug: cache_check: 81584d1f9490623ff5467ce37767e23c is negative LibClamAV debug: in cli_check_mydoom_log() LibClamAV debug: Matched signature for file type ZIP-SFX at 209672 LibClamAV debug: matcher_run: performing regex matching on full map: 246144+8211(254355) >= 254355 LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0 LibClamAV debug: CL_TYPE_ZIPSFX signature found at 209672 LibClamAV debug: in cli_unzip_single LibClamAV debug: cli_unzip: local header - ZMDNAME:0:flags::1701060708:1701601889:6e650000:124:0:1 LibClamAV debug: CDBNAME:CL_TYPE_ZIP:1701601889:flags::1701601889:1701060708:0:0:1852112896:0 LibClamAV debug: cli_unzip: local header - stream out of file Illegal instruction(coredump)

clamscan /opt/freeware/bin/g* Loading: 46s, ETA: 0s [========================>] 8.62M/8.62M sigs Compiling: 7s, ETA: 0s [========================>] 41/41 tasks /opt/freeware/bin/g++-8: OK /opt/freeware/bin/g++-8: OK ....... /opt/freeware/bin/gpg-error_32: OK /opt/freeware/bin/gpg-error_64: OK Illegal instruction

Info on stack

core dump: Core was generated by `clamscan_64'. Program terminated with signal SIGILL, Illegal instruction.

0 0x0000000000000000 in ?? ()

[Current thread is 1 (process 1)] (gdb) (gdb) where

0 0x0000000000000000 in ?? ()

1 0x0900000003508c38 in fmap_unneed_off (m=0x1102f4890, at=0, len=30) at /home/buildusr/rpmbuild/BUILD/clamav-0.104.2/64bit/libclamav/fmap.h:264

2 0x090000000350b420 in parse_local_file_header (map=0x90000000350d7a4 <unzip_single_internal+348>, loff=0, zsize=150994944, num_files_unzipped=0x0, file_count=1,

central_header=0xa0040538 <error: Cannot access memory at address 0xa0040538>, ret=0x18a4ffffeb58, ctx=0xfffffffffffe600, tmpd=0x0, detect_encrypted=0, zcb=0xfffffffffffe5fc,
record=0xfffffffffffeb58) at /home/buildusr/rpmbuild/BUILD/clamav-0.104.2/64bit/libclamav/unzip.c:713

3 0x090000000350d7a4 in unzip_single_internal (ctx=0xfffffffffffeb58, local_header_offset=0, zcb=@0x8001000a0031e70: 0x900000003437a80 <cli_magic_scan_desc_type+876>)

at /home/buildusr/rpmbuild/BUILD/clamav-0.104.2/64bit/libclamav/unzip.c:1404

4 0x090000000350d818 in unzip_single_internal (ctx=0x23000000000, local_header_offset=4567847280, zcb=0x300000000)

at /home/buildusr/rpmbuild/BUILD/clamav-0.104.2/64bit/libclamav/unzip.c:1418

5 0x0900000003433164 in scanraw (ctx=0xfffffffffffeb58, type=CL_TYPE_BINARY_DATA, typercg=1 '\001', dettype=0xfffffffffffea0c, refhash=0x1102f2d79 "\201XM\037\224\220b?▒F|▒wg▒<")

at /home/buildusr/rpmbuild/BUILD/clamav-0.104.2/64bit/libclamav/scanners.c:3675

6 0x0900000003436a64 in cli_magic_scan (ctx=0xfffffffffffecd0, type=134221824) at /home/buildusr/rpmbuild/BUILD/clamav-0.104.2/64bit/libclamav/scanners.c:4694

7 0x0900000003438a10 in scan_common (map=0x1102f2cf0, filepath=0x1102f2cb0 "/opt/freeware/bin/g13_64", virname=0xfffffffffffeeb0, scanned=0x110020568 <info+24>, engine=0x11007b670,

scanoptions=0xffffffffffff264, context=0xfffffffffffef90) at /home/buildusr/rpmbuild/BUILD/clamav-0.104.2/64bit/libclamav/scanners.c:5304

8 0x0900000003439290 in cl_scandesc_callback (desc=5, filename=0x1102f2cb0 "/opt/freeware/bin/g13_64", virname=0xfffffffffffeeb0, scanned=0x110020568 <info+24>, engine=0x11007b670,

scanoptions=0xffffffffffff264, context=0xfffffffffffef90) at /home/buildusr/rpmbuild/BUILD/clamav-0.104.2/64bit/libclamav/scanners.c:5484

9 0x000000010000b4d4 in scanfile (filename=0x1102f2cb0 "/opt/freeware/bin/g13_64", engine=0x11007b670, opts=0x1100731f0, options=0xffffffffffff264)

at /home/buildusr/rpmbuild/BUILD/clamav-0.104.2/64bit/clamscan/manager.c:407

10 0x000000010000bd14 in scandirs (dirname=0x1102f2170 "/opt/freeware/bin", engine=0x11007b670, opts=0x1100731f0, options=0xffffffffffff264, depth=2, dev=9223372079804448777)

at /home/buildusr/rpmbuild/BUILD/clamav-0.104.2/64bit/clamscan/manager.c:544

11 0x000000010000f03c in scanmanager (opts=0x1100731f0) at /home/buildusr/rpmbuild/BUILD/clamav-0.104.2/64bit/clamscan/manager.c:1553

12 0x0000000100000a7c in main (argc=0, argv=0x11001e4f8 <ccOe0ebr.rw+664>) at /home/buildusr/rpmbuild/BUILD/clamav-0.104.2/64bit/clamscan/clamscan.c:171

We have tried to trace it back and found below information file location:clamav-0.104.2/64bit/libclamav/fmap.h 262 static inline void fmap_unneed_off(fmap_t *m, size_t at, size_t len) 263 { 264 m->unneed_off(m, at, len); ---------------> here we are getting core 265 }

seems like 2 threads are trying to clear the same pointer data

is there a possible fix

We have a workaround which seems to prevent this issue static inline void fmap_unneed_off(fmap_t *m, size_t at, size_t len) { if (m->handle != NULL) m->unneed_off(m, at, len); }

[ragusaa]

Hi,

Thank you for the bug report. We have released clamav 0.104.3, have you tried that? If that doesn't resolve the issue, is it possible for us to get the file that you are scanning when this crash occurs?

Also, could we have the cmake command you are using to build clam?

Thanks, Andy

[nehajain29]

Hi Andy,

I have already tried with the 0.104.3 version, seeing the same results clamscan dumping core for these packages: zip,g13 cmake options we have used: cmake . \ -D CMAKE_INSTALL_PREFIX=/opt/freeware \ -D CMAKE_INSTALL_LIBDIR=/opt/freeware/lib64 \ -D CMAKE_INSTALL_FULL_SBINDIR=/opt/freeware/sbin \ -D CMAKE_INSTALL_MANDIR=/opt/freeware/man \ -D CMAKE_INSTALL_SYSCONFDIR=/opt/freeware/etc \ -D APP_CONFIG_DIRECTORY=/opt/freeware/etc/clamav \ -D DATABASE_DIRECTORY=/var/lib/clamav \ -D ENABLE_JSON_SHARED=OFF \ -D JSONC_INCLUDE_DIR=/opt/freeware/include/json-c \ -D JSONC_LIBRARY=/opt/freeware/lib/libjson-c.a \ -D ENABLE_TESTS=OFF \ -D ENABLE_MILTER=OFF

Thanks, Neha

[ragusaa]

Hi Neha,

I tried to reproduce this by downloading those packages without any luck. Could you send the files you are having problems with to clamav-bugs@external.cisco.com?

Thanks, Andy

[sr-kuantum]

Hi Andy, Thanks for trying it out at your end. We have also been trying to reproduce this issue on different nodes with the same set of files. But it seems like we are hitting this only on a specific node and not on all the nodes. It seems like some timing issue and there is no specific recreate.

BTW, can you provide your comments on the below fix that we are trying? Would that cause any issue, in case we add to safeguard from the crash? Can you provide some more info around this code area?

libclamav/fmap.h

static inline void fmap_unneed_off(fmap_t *m, size_t at, size_t len)
{
    if (m->handle != NULL)
        m->unneed_off(m, at, len);
}

Thanks, Sriram

[ragusaa]

Hi Sriram,

From the log, it appears that the issue is in the call to 'm->unneed_off', which is a function pointer, so I would probably change it to the following, although what you have looks good, too. if (m && m->unneed_off){ m->unneed_off(m, at, len); } I don't see anywhere in the code where this value should be getting changed, but I'll take another look. If you don't mind, could you also put some logging in and else block for when this condition fails, and send us a log. Maybe that will give us an idea about what is going on.

Thanks, Andy

[nehajain29]

Hi Andy,

Please find the values we have found.

up

1 0x090000000348cba4 in fmap_unneed_off (m=0x1102f13f0, at=0, len=30)

at /home/buildusr/rpmbuild/BUILD/clamav-0.104.3/64bit/libclamav/fmap.h:264

264 m->unneed_off(m, at, len); (gdb) p m $1 = (fmap_t ) 0x1102f13f0 (gdb) p m $2 = {handle = 0x0, pread_cb = 0x0, data = 0x0, mtime = 0, pages = 0, pgsz = 0, paged = 0, aging = 0, dont_cache_flag = 0, handle_is_fd = 0, offset = 0, nested_offset = 0, real_len = 0, len = 0, unmap = 0x0, need = 0x0, need_offstr = 0x0, gets = 0x0, unneed_off = 0x0, have_maphash = false} (gdb)

Let us know if you need more details on the stack. We are working on the log and will give it once we have it.

Thanks, Neha

[ragusaa]

Thanks for the update. It certainly looks like the fmap is being overwritten somewhere between the top of parse_local_file_header and the call for the error condition cli_dbgmsg("cli_unzip: local header - stream out of file\n"); fmap_unneed_off(map, loff, SIZEOF_LOCAL_HEADER); but at this point, I don't see where. Any additional debug logs will be helpful.

Thanks, Andy

[nehajain29]

Hi Andy,

I have tried with clamscan --debug option and given the logs of it. We have tried to add debug statements in fmap.h but build is failing because of that.

Please find the attachment. clamscan_debug_logs.zip

Thanks, Neha

[ragusaa]

Hi Neha,

I am sorry for the late response on this.

I am not seeing anything odd in the log. Did this crash occur with or without your patch? I assume it was without?

Is there anything specific about this one particular node that is different from the other nodes?

[nehajain29]

Hi Andy,

Yes, this core is without any patch. This issue is specific to nodes we are not seeing this issue on all nodes. But i am not sure what is different in this node.

Thanks, Neha

[nehajain29]

Hi Andy,

Saw same issue again in same file but different function.

Type 'help' for help. [using memory image in core] reading symbolic information ...

Illegal instruction (illegal opcode) in . at 0x0 ($t1) .() at 0x0 unzip.fmap_need_ptr_once(m = 0x00000001102c3250, ptr = 0x0a0000004db5b386, len = 11304), line 249 in "fmap.h" parse_local_file_header(map = 0x0900000000bd7740, loff = 0, zsize = 150994944, num_files_unzipped = (nil), file_count = 1, central_header = (invalid char ptr (0x00000000a0040590)), ret = 0x00008a4fffffe748, ctx = 0x0fffffffffffe1f0, tmpd = (nil), detect_encrypted = 0, zcb = 0x0fffffffffffe1ec, record = 0x0fffffffffffe748), line 724 in "unzip.c" unzip_single_internal(ctx = 0x0fffffffffffe748, local_header_offset = 0, zcb = 0x08001000a0031ef0), line 1404 in "unzip.c" cli_unzip_single(ctx = 0x0fffffffffffe748, local_header_offset = 0), line 1422 in "unzip.c" unnamed block in scanraw(ctx = 0x0fffffffffffe748, type = CL_TYPE_BINARY_DATA, typercg = '^A', dettype = 0x0fffffffffffe5fc, refhash = "=\236 ^Q)j\306^D\233<\374\232\213\206\362^U"), line 3679 in "scanners.c" scanraw(ctx = 0x0fffffffffffe748, type = CL_TYPE_BINARY_DATA, typercg = '^A', dettype = 0x0fffffffffffe5fc, refhash = "=\236 ^Q)j\306^D\233<\374\232\213\206\362^U"), line 3679 in "scanners.c" cli_magic_scan(ctx = 0x0fffffffffffe8c0, type = 134221824), line 4694 in "scanners.c" scan_common(map = 0x00000001102c2f50, filepath = "/usr/java5/jre/bin/libiverel23.so", virname = 0x0fffffffffffeaa0, scanned = 0x0000000110020568, engine = 0x00000001100411b0, scanoptions = 0x0ffffffffffff3d4, context = 0x0fffffffffffeb80), line 5311 in "scanners.c" cl_scandesc_callback(desc = 9, filename = "/usr/java5/jre/bin/libiverel23.so", virname = 0x0fffffffffffeaa0, scanned = 0x0000000110020568, engine = 0x00000001100411b0, scanoptions = 0x0ffffffffffff3d4, context = 0x0fffffffffffeb80), line 5487 in "scanners.c" scanfile(filename = "/usr/java5/jre/bin/libiverel23.so", engine = 0x00000001100411b0, opts = 0x00000001100382f0, options = 0x0ffffffffffff3d4), line 407 in "manager.c" scandirs(dirname = "/usr/java5/jre/bin", engine = 0x00000001100411b0, opts = 0x00000001100382f0, options = 0x0ffffffffffff3d4, depth = 6, dev = 9223372079804448772), line 544 in "manager.c" scandirs(dirname = "/usr/java5/jre", engine = 0x00000001100411b0, opts = 0x00000001100382f0, options = 0x0ffffffffffff3d4, depth = 5, dev = 9223372079804448772), line 546 in "manager.c" scandirs(dirname = "/usr/java5", engine = 0x00000001100411b0, opts = 0x00000001100382f0, options = 0x0ffffffffffff3d4, depth = 4, dev = 9223372079804448772), line 546 in "manager.c" scandirs(dirname = "/usr", engine = 0x00000001100411b0, opts = 0x00000001100382f0, options = 0x0ffffffffffff3d4, depth = 3, dev = 9223372079804448772), line 546 in "manager.c" scandirs(dirname = "/", engine = 0x00000001100411b0, opts = 0x00000001100382f0, options = 0x0ffffffffffff3d4, depth = 2, dev = 9223372079804448772), line 546 in "manager.c" scanmanager(opts = 0x00000001100382f0), line 1553 in "manager.c" main(argc = 0, argv = 0x000000011001e4f8), line 171 in "clamscan.c"

(dbx) print m 0x00000001102c3250 (dbx) print *m (handle = (nil), pread_cb = (nil), data = (nil), mtime = 0, pages = 0, pgsz = 0, paged = 0, aging = 0, dont_cache_flag = 0, handle_is_fd = 0, offset = 0, nested_offset = 0, real_len = 0, len = 0, unmap = (nil), need = (nil), need_offstr = (nil), gets = (nil), unneed_off = (nil), have_maphash = [bad size in extractField -- word assumed] 0, 8;maphash = [bad size in extractField -- word assumed] '\0', bitmap = 0x00000001102c3010, name = (nil))

[ragusaa]

Hi Neha,

Thank you for the update. Looks like it was scanning /usr/java5/jre/bin/libiverel23.so. If you scan that file and/or directory, it does not consistently crash? Would you mind scanning that file multiple times (10 or more) to see if it happens intermittently?

Thanks, Andy

[nehajain29]

Hi Andy,

I have tried clamscan on this particular file 15 times but I am not seeing any core. When I trigger clamscan on the whole system then only I am getting core command in which I am getting core : clamscan -rv --exclude-dir=/proc --tempdir=/tmp / the output of clamscan on this particular file.

clamscan -rv /usr/java5/jre/bin/libiverel23.so LibClamAV Warning: ** LibClamAV Warning: The virus database is older than 7 days! LibClamAV Warning: Please update it as soon as possible. LibClamAV Warning: ** Loading: 1m 03s, ETA: 0s [========================>] 8.62M/8.62M sigs Compiling: 7s, ETA: 0s [========================>] 41/41 tasks

Scanning /usr/java5/jre/bin/libiverel23.so LibClamAV Warning: cli_unzip: failed to create temporary file /clamav-f39003b4f0 3276db5e418a9862e3907a.tmp /usr/java5/jre/bin/libiverel23.so: Can't create temporary file ERROR

----------- SCAN SUMMARY ----------- Known viruses: 8618558 Engine version: 0.104.2 Scanned directories: 0 Scanned files: 0 Infected files: 0 Total errors: 1 Data scanned: 0.11 MB Data read: 0.11 MB (ratio 1.00:1) Time: 74.739 sec (1 m 14 s) Start Date: 2022:08:11 09:49:38 End Date: 2022:08:11 09:50:53

I have tried clamscan on /usr/java5 also I am not seeing any core

Thanks, Neha