search

owncloud / files_antivirus

:space_invader: virus scanner for ownCloud
GNU Affero General Public License v3.0
48 stars 30 forks source link

[QA] cannot upload files larger than ca. 20 MB #443

Open jnweiger opened 3 years ago

jnweiger commented 3 years ago

Seen with files_antivirus 1.0.0-rc2 running on https://oc1070-macafee-20210519.jw-qa.owncloud.works/owncloud connected to Ralf's McAfee ICAP instance. Both ownCloud and McAfee are on separate hetzner machines. Thus file transfer times matter.

Bad workaround:

Acceptable hotfix:

ICAP compliant implementation:

VicDeo commented 3 years ago

@jnweiger here is the C-ICAP response line by line... There is no Content-Length there:

0 = "ICAP/1.0 200 OK\r\n"
1 = "Server: C-ICAP/0.4.4\r\n"
2 = "Connection: close\r\n"
3 = "ISTag: CI0001-VdGmOb1D29kBIlH75+mvLAAA\r\n"
4 = "X-Infection-Found: Type=0; Resolution=2; Threat=Win.Test.EICAR_HDB-1;\r\n"
5 = "X-Violations-Found: 1\r\n"
6 = "\t-\r\n"
7 = "\tWin.Test.EICAR_HDB-1\r\n"
8 = "\t0\r\n"
9 = "\t0\r\n"
10 = "Encapsulated: res-hdr=0, res-body=108\r\n"
11 = "\r\n"
12 = "HTTP/1.0 403 Forbidden\r\n"
13 = "Server: C-ICAP\r\n"
14 = "Connection: close\r\n"
15 = "Content-Type: text/html\r\n"
16 = "Content-Language: en\r\n"
17 = "\r\n"
18 = "1d7\r\n"
19 = "<html>\n"
20 = " <head>\n"
21 = "   <title>VIRUS FOUND</title>\n"
22 = "</head>\n"
23 = "\n"
24 = "<body>\n"
25 = "<h1>VIRUS FOUND</h1>\n"
26 = "\n"
27 = "\n"
28 = "You tried to upload/download a file that contains the virus: \n"
29 = "   <b> Win.Test.EICAR_HDB-1 </b>\n"
30 = "<br>\n"
31 = "The Http location is: \n"
32 = "<b>  127.0.0.1/ </b>\n"
33 = "\n"
34 = "<p>\n"
35 = "  For more information contact your system administrator\n"
36 = "\n"
37 = "<hr>\n"
38 = "<p>\n"
39 = "This message generated by C-ICAP service: <b> avscan?allow204=on&sizelimit=off&mode=simple </b>\n"
40 = "<br>Antivirus engine: <b> clamav-01024/25976 </b>\n"
41 = "\n"
42 = "</p>\n"
43 = "\n"
44 = "</body>\n"
45 = "</html>\n"
46 = "  \r\n"
47 = "0\r\n"
48 = "\r\n"

Is it possible to have a similar response from mcAfee?

VicDeo commented 3 years ago

In fact we don't need the whole response - we can read it line by line until the target header is found or EOF marker. Whatever comes first.

jnweiger commented 3 years ago

What I saw from McAfee had a Content-Length. Seems that ClamAv does not do that.

But in any case, we a) have the length. In your example line 18 is the length (encoded in hex). and b) have an eof marker. (lines 46, 47, 48...)

jnweiger commented 3 years ago

Here is a response I saw from McAfee (\r and \n added to make the line endings clear) The content length (2718 bytes) is specified as a Content-Length: header, and also (in hex) as A9E\r\n at the start of the body.

ICAP/1.0 200 OK\r\n
ISTag: "00007400-18.32.12-00009990"\r\n
Encapsulated: res-hdr=0, res-body=122\r\n
\r\n
HTTP/1.1 403 VirusFound\r\n
Content-Type: text/html\r\n
Cache-Control: no-cache\r\n
Content-Length: 2718\r\n
X-Frame-Options: deny\r\n
\r\n
A9E\r\n
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n
<html>\n
<!-- FileName: index.html\n
     Language: [en]\n
-->\n
<!--Head-->\n
<head>\n
  <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">\n
  <meta http-equiv="X-UA-Compatible" content="IE=7" />\n
  <title>McAfee Web Gateway - Notification</title>\n
  <script src="/mwg-internal/de5fs23hu73ds/files/javascript/sw.js" type="text/javascript" ></script>\n
  <link rel="stylesheet" href="/mwg-internal/de5fs23hu73ds/files/default/stylesheet.css" />\n
</head>\n
<!--/Head-->\n
<!--Body-->\n
<body onload="swOnLoad();">\n
  <table class='bodyTable'>\n
    <tr>\n
      <td class='bodyData' background='/mwg-internal/de5fs23hu73ds/files/default/img/bg_body.gif'>\n
<!--Logo-->\n
<table class='logoTable'>\n
  <tr>\n
    <td class='logoData'>\n
      <a href='http://www.mcafee.com'>\n
        <img src='/mwg-internal/de5fs23hu73ds/files/default/img/logo_mwg.png'></a>\n
    </td>\n
  </tr>\n
</table>\n
<!--/Logo-->\n
<!--Contents-->\n
<!-- FileName: VirusFound.html\n
     Language: [en]\n
-->\n
<!--Title-->\n
<table class='titleTable' background='/mwg-internal/de5fs23hu73ds/files/default/img/bg_navbar.jpg'>\n
  <tr>\n
    <td class='titleData'>\n
      Malware Detected\n
    </td>\n
  </tr>\n
</table>\n
<!--/Title-->\n
\n
<!--Content-->\n
<table class="contentTable">\n
  <tr>\n
    <td class="contentData">\n
      The transferred file contained a virus and was therefore blocked.\n
    </td>\n
  </tr>\n
</table>\n
<!--/Content-->\n
\n
<!--Info-->\n
<table class="infoTable">\n
  <tr>\n
    <td class="infoData">\n
      <b>URL: </b><script type="text/javascript">break_line("http://www.origin-server.example.com/origin-resource/form.pl");</script><br />\n
      <b>Media Type: </b>text/plain<br />\n
      <b>Virus Name: </b>EICAR test file<br />\n
      <b>Full Filename: </b>form.pl<br />\n
    </td>\n
  </tr>\n
</table>\n
<!--/Info-->\n
\n
<!--/Contents-->\n
<!--Policy-->\n
<table class='policyTable'>\n
  <tr>\n
    <td class='policyHeading'>\n
      <hr>\n
      Company Acceptable Use Policy\n
    </td>\n
  </tr>\n
  <tr>\n
    <td class='policyData'>\n
      This is an optional acceptable use disclaimer that appears on every page. You may change the wording or remove this section entirely in index.html.\n
    </td>\n
  </tr>\n
</table>\n
<!--/Policy-->\n
<!--Foot-->\n
<table class='footTable'>\n
  <tr>\n
    <td class='helpDeskData' background='/mwg-internal/de5fs23hu73ds/files/default/img/bg_navbar.jpg'>\n
      For assistance, please contact your system administrator.\n
    </td>\n
  </tr>\n
  <tr>\n
    <td class='footData'>\n
      generated <span id="time">2021-05-21 11:59:45</span> by McAfee Web Gateway\n
      <br />\n
      \n
    </td>\n
  </tr>\n
</table>\n
<!--/Foot-->\n
      </td>\n
    </tr>\n
  </table>\n
</body>\n
<!--/Body-->\n
</html>\n
\r\n
0\r\n
\r\n
VicDeo commented 3 years ago

@jnweiger I checked https://github.com/owncloud/files_antivirus/pull/445 against the content you posted above. Should work as expected now.

VicDeo commented 3 years ago

@jnweiger

here is the response of McAfee on a clean file:

ICAP/1.0 204 No modification needed\r\n
ISTag: "00007405-8.32.140-00009997"\r\n
\r\n
---McAfee hangs here and finally returns boolean false---

Neither Content-Length nor anything meaningful is passed along in this case.

VicDeo commented 3 years ago

~to overcome this I'd suggest to set timeout to 5 seconds in case ICAP status equals to 204. Not that clean but should help.~

Implemented in https://github.com/owncloud/files_antivirus/pull/445 solution: on ICAP status 204 we stop further reading and consider the file to be clean.

jnweiger commented 3 years ago

Confirmed "improved" in files_anntivirus-1.0.0-rc3 (including #445).

The failures show this message: image

Workaround: Patch the timeout to be 5 minutes. -> https://github.com/owncloud/files_antivirus/pull/446 In my setup with the McAfee scanner at vmlab.owncloud.works, the above files then succeed:

Ubuntu 20.04 ships with /etc/php/7.4/apache2/php.ini:default_socket_timeout = 60 @VicDeo should owncloud production systems have a longer timeout?

VicDeo commented 3 years ago

I can say only that timeout should be set at the sane value that considers:

I see no universal value except -1 (infinite). It could be set at the runtime just before sending the data.

jnweiger commented 2 years ago

@GeraldLeikam #445 might improve the situation here. Please retest with 1.1.0-rc.2 to see what the current limits are.