search

Cisco-Talos / clamav

ClamAV - Documentation is here: https://docs.clamav.net
https://www.clamav.net/
GNU General Public License v2.0
4.34k stars 699 forks source link

Used in windows, the.\clamscan -d test.hdb test.apk command reports ERROR: Malformed database #909

Closed Yesisyes closed 1 year ago

Yesisyes commented 1 year ago

test.txt [Uploading test.zip…]()

Describe the bug

I successfully used the 'sigtool -- md5 test. apk>test. hdb' command on the Windows system, and the 'cat test. hdb' command output was' 790a62a841927f4ec1e0675a590c56d1:198800: test. apk '. However, when I used the'. clamscan - d test. hdb test. apk 'command, I returned' LibClamAV Error: cli '_ loadhash: Problem parsing database at line 1’ ‘LibClamAV Error: Can't load test.hdb: Malformed database’ ‘ERROR: Malformed database [ ] 0/3 sigs’

How to reproduce the problem

PS C:\Program Files\ClamAV> .\clamscan -d test.hdb test.apk

LibClamAV Error: cli_loadhash: Problem parsing database at line 1 LibClamAV Error: Can't load test.hdb: Malformed database ERROR: Malformed database [ ] 0/3 sigs

----------- SCAN SUMMARY ----------- Known viruses: 0 Engine version: 1.0.1 Scanned directories: 0 Scanned files: 0 Infected files: 0 Data scanned: 0.00 MB Data read: 0.00 MB (ratio 0.00:1) Time: 0.009 sec (0 m 0 s) Start Date: 2023:04:27 15:04:36 End Date: 2023:04:27 15:04:36

Attachments

If applicable, add screenshots to help explain your problem.

If the issue is reproducible only when scanning a specific file, attach it to the ticket.

micahsnyder commented 1 year ago

@Yesisyes your description does not make sense.

You wrote that the error is

‘LibClamAV Error: Can't load clean.hdb: Malformed database’

But that your database is named test.hdb.

The filenames are different.

Please attach whatever database files that you're having problems with to this ticket.

Yesisyes commented 1 year ago

Sorry,i made a typo. The actual output is test.hdb.I have improved the problem

Yesisyes commented 1 year ago

@Yesisyes your description does not make sense.

You wrote that the error is

‘LibClamAV Error: Can't load clean.hdb: Malformed database’

But that your database is named test.hdb.

The filenames are different.

Please attach whatever database files that you're having problems with to this ticket.

Welcome someone to replicate the operation in Windows, I hope you get different results, so I can know why mine is error output.I'm using the latest version.

Yesisyes commented 1 year ago

The detailed information is: PS C:\Program Files\ClamAV> .\clamscan --debug --verbose -d .\test.hdb .\test.apk LibClamAV debug: searching for unrar LibClamAV debug: unrar support loaded from libclamunrar_iface.dll LibClamAV debug: Initialized 1.0.1 engine LibClamAV debug: Initializing phishcheck module LibClamAV debug: Phishcheck: Compiling regex: ^ (http|https|ftp:(//)?)?[0-9]{1,3}(.[0-9]{1,3}){3}[/?:]? $ LibClamAV debug: Phishcheck module initialized LibClamAV debug: Bytecode initialized in interpreter mode LibClamAV Error: cli_loadhash: Problem parsing database at line 1 LibClamAV Error: Can't load .\test.hdb: Malformed database ERROR: Malformed database LibClamAV debug: Cleaning up phishcheck LibClamAV debug: Freeing phishcheck struct LibClamAV debug: Phishcheck cleaned up

----------- SCAN SUMMARY ----------- Known viruses: 0 Engine version: 1.0.1 Scanned directories: 0 Scanned files: 0 Infected files: 0 Data scanned: 0.00 MB Data read: 0.00 MB (ratio 0.00:1) Time: 0.009 sec (0 m 0 s) Start Date: 2023:04:27 17:16:34 End Date: 2023:04:27 17:16:34

Sanesecurity commented 1 year ago

Note in the spaces in your hdb example... Try again with no spaces in hdb file...

790a62a841927f4ec1e0675a590c56d1:198800:test.apk

I think I've hit this issue before

Yesisyes commented 1 year ago

Note in the spaces in your hdb example... Try again with no spaces in hdb file...

790a62a841927f4ec1e0675a590c56d1:198800:test.apk

I think I've hit this issue before

There's only one line in my.hdb file, and I'm sure there's no extra space.

demonduck commented 1 year ago

I've tried it on my local clamav 1.0.0 with a couple of versions to try and figure out what is going wrong. Unfortunately all things I tried worked correctly as expected.

$ clamscan -d temp.hdb 790a62a841927f4ec1e0675a590c56d1 
Loading:     0s, ETA:   0s [========================> 
Compiling:   0s, ETA:   0s [========================> 

/tmp/790a62a841927f4ec1e0675a590c56d1: test.apk.UNOFFICIAL FOUND

----------- SCAN SUMMARY -----------
Known viruses: 1
Engine version: 1.0.0
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.62 MB
Data read: 0.19 MB (ratio 3.31:1)
Time: 0.090 sec (0 m 0 s)
Start Date: 2023:04:27 12:33:07
End Date:   2023:04:27 12:33:07
$ xxd temp.hdb 
00000000: 3739 3061 3632 6138 3431 3932 3766 3465  790a62a841927f4e
00000010: 6331 6530 3637 3561 3539 3063 3536 6431  c1e0675a590c56d1
00000020: 3a31 3938 3830 303a 7465 7374 2e61 706b  :198800:test.apk
00000030: 0a                                       .
$ cat temp.hdb
790a62a841927f4ec1e0675a590c56d1:198800:test.apk
$ clamscan --version
ClamAV 1.0.0/26889/Thu Apr 27 07:25:48 2023

I also created a file with a \r\n instead of just a \n and another with no new line or return carriage characters and all versions worked 

$ xxd temp_windows_like.hdb                                    
00000000: 3739 3061 3632 6138 3431 3932 3766 3465  790a62a841927f4e                              
00000010: 6331 6530 3637 3561 3539 3063 3536 6431  c1e0675a590c56d1                              
00000020: 3a31 3938 3830 303a 7465 7374 2e61 706b  :198800:test.apk                              
00000030: 0d0a                                     ..                                            
$ clamscan -d temp_windows_like.hdb 790a62a841927f4ec1e0675a590c56d1                                                                                            
Loading:     0s, ETA:   0s [========================>]        1/1 sigs                           
Compiling:   0s, ETA:   0s [========================>]       10/10 tasks                         

/tmp/790a62a841927f4ec1e0675a590c56d1: test.apk.UNOFFICIAL FOUND           

----------- SCAN SUMMARY -----------                                                             
Known viruses: 1                                                                                 
Engine version: 1.0.0                                                                            
Scanned directories: 0                                                                           
Scanned files: 1                                                                                 
Infected files: 1                                                                                
Data scanned: 0.62 MB                                                                            
Data read: 0.19 MB (ratio 3.31:1)                                                                
Time: 0.067 sec (0 m 0 s)                                                                        
Start Date: 2023:04:27 12:38:45                                                                  
End Date:   2023:04:27 12:38:45                   
$ xxd temp_single_line.hdb 
00000000: 3739 3061 3632 6138 3431 3932 3766 3465  790a62a841927f4e
00000010: 6331 6530 3637 3561 3539 3063 3536 6431  c1e0675a590c56d1
00000020: 3a31 3938 3830 303a 7465 7374 2e61 706b  :198800:test.apk
$ clamscan -d temp_single_line.hdb 790a62a841927f4ec1e0675a590c56d1 
Loading:     0s, ETA:   0s [========================>]        1/1 sigs    
Compiling:   0s, ETA:   0s [========================>]       10/10 tasks 

/tmp/790a62a841927f4ec1e0675a590c56d1: test.apk.UNOFFICIAL FOUND

----------- SCAN SUMMARY -----------
Known viruses: 1
Engine version: 1.0.0
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.62 MB
Data read: 0.19 MB (ratio 3.31:1)
Time: 0.070 sec (0 m 0 s)
Start Date: 2023:04:27 12:39:25
End Date:   2023:04:27 12:39:25
Sanesecurity commented 1 year ago

test.txt

Sanesecurity commented 1 year ago

Download the above, rename to test.hdb does that one work?

Yesisyes commented 1 year ago

Download the above, rename to test.hdb does that one work?

Thank you very much. It is dark now for me in China. I will try it tomorrow morning.

Yesisyes commented 1 year ago

Download the above, rename to test.hdb does that one work?

I tested the files you gave me. Everything's fine. But why not the files I generate using commands? I can't really tell the difference between yours and mine with my naked eye. I will give you running screenshots and sample files, I hope you can test in Windows11 system, thank you very much.

Yesisyes commented 1 year ago

I've tried it on my local clamav 1.0.0 with a couple of versions to try and figure out what is going wrong. Unfortunately all things I tried worked correctly as expected.

$ clamscan -d temp.hdb 790a62a841927f4ec1e0675a590c56d1 
Loading:     0s, ETA:   0s [========================> 
Compiling:   0s, ETA:   0s [========================> 

/tmp/790a62a841927f4ec1e0675a590c56d1: test.apk.UNOFFICIAL FOUND

----------- SCAN SUMMARY -----------
Known viruses: 1
Engine version: 1.0.0
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.62 MB
Data read: 0.19 MB (ratio 3.31:1)
Time: 0.090 sec (0 m 0 s)
Start Date: 2023:04:27 12:33:07
End Date:   2023:04:27 12:33:07
$ xxd temp.hdb 
00000000: 3739 3061 3632 6138 3431 3932 3766 3465  790a62a841927f4e
00000010: 6331 6530 3637 3561 3539 3063 3536 6431  c1e0675a590c56d1
00000020: 3a31 3938 3830 303a 7465 7374 2e61 706b  :198800:test.apk
00000030: 0a                                       .
$ cat temp.hdb
790a62a841927f4ec1e0675a590c56d1:198800:test.apk
$ clamscan --version
ClamAV 1.0.0/26889/Thu Apr 27 07:25:48 2023

I also created a file with a \r\n instead of just a \n and another with no new line or return carriage characters and all versions worked

$ xxd temp_windows_like.hdb                                    
00000000: 3739 3061 3632 6138 3431 3932 3766 3465  790a62a841927f4e                              
00000010: 6331 6530 3637 3561 3539 3063 3536 6431  c1e0675a590c56d1                              
00000020: 3a31 3938 3830 303a 7465 7374 2e61 706b  :198800:test.apk                              
00000030: 0d0a                                     ..                                            
$ clamscan -d temp_windows_like.hdb 790a62a841927f4ec1e0675a590c56d1                                                                                            
Loading:     0s, ETA:   0s [========================>]        1/1 sigs                           
Compiling:   0s, ETA:   0s [========================>]       10/10 tasks                         

/tmp/790a62a841927f4ec1e0675a590c56d1: test.apk.UNOFFICIAL FOUND           

----------- SCAN SUMMARY -----------                                                             
Known viruses: 1                                                                                 
Engine version: 1.0.0                                                                            
Scanned directories: 0                                                                           
Scanned files: 1                                                                                 
Infected files: 1                                                                                
Data scanned: 0.62 MB                                                                            
Data read: 0.19 MB (ratio 3.31:1)                                                                
Time: 0.067 sec (0 m 0 s)                                                                        
Start Date: 2023:04:27 12:38:45                                                                  
End Date:   2023:04:27 12:38:45                   
$ xxd temp_single_line.hdb 
00000000: 3739 3061 3632 6138 3431 3932 3766 3465  790a62a841927f4e
00000010: 6331 6530 3637 3561 3539 3063 3536 6431  c1e0675a590c56d1
00000020: 3a31 3938 3830 303a 7465 7374 2e61 706b  :198800:test.apk
$ clamscan -d temp_single_line.hdb 790a62a841927f4ec1e0675a590c56d1 
Loading:     0s, ETA:   0s [========================>]        1/1 sigs    
Compiling:   0s, ETA:   0s [========================>]       10/10 tasks 

/tmp/790a62a841927f4ec1e0675a590c56d1: test.apk.UNOFFICIAL FOUND

----------- SCAN SUMMARY -----------
Known viruses: 1
Engine version: 1.0.0
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.62 MB
Data read: 0.19 MB (ratio 3.31:1)
Time: 0.070 sec (0 m 0 s)
Start Date: 2023:04:27 12:39:25
End Date:   2023:04:27 12:39:25

I will give you running screenshots and sample files, I hope you can test in Windows11 system, thank you very much.

Yesisyes commented 1 year ago

1682644746290

Yesisyes commented 1 year ago

test.zip

Yesisyes commented 1 year ago

Download the above, rename to test.hdb does that one work?

The file you gave is in the format of "ASCII", while the file generated by my command is in the format of "UTF-16". Is it because clamscan cannot read the file in the format of "UTF-16"? However, in my Windows system, the.hdb file generated by the sigtool --md5 command is in UTF-16 format by default. “ .\sigtool --md5 .\test.apk | Out-File -Encoding ascii test.hdb”.This order will solve my current problem.

Sanesecurity commented 1 year ago

Just tested.

Powershell + sigtool - outputs by default UTF-16 cmd + sigtool - outputs ascii

UTF-16 output won't read correctly as a database, ascii will.

Set Powershell default encoding:

https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_character_encoding?view=powershell-7.3&WT.mc_id=ps-gethelp#changing-the-default-encoding

Yesisyes commented 1 year ago

Just tested.

Powershell + sigtool - outputs by default UTF-16 cmd + sigtool - outputs ascii

UTF-16 output won't read correctly as a database, ascii will.

Set Powershell default encoding:

https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_character_encoding?view=powershell-7.3&WT.mc_id=ps-gethelp#changing-the-default-encoding

This is another solution，i get it. Just for the record,".\sigtool --md5 filepath | Out-File -Append -Encoding ascii test.hdb", use this command to append to the new signature.