Does cvdupdate work with proxy

[marekgrajcar]

Hello, due to the latest changes I am trying to download signatures using cvdupdate tool. I am performing it from inside of company’s network and I get warning: “Failed to determine available version via DNS TXT query!”. However when I am looking for address names resolution using our internal proxy I am able to get an answer:

# nslookup database.clamav.net <proxy_ip>
Server:         <proxy_ip>
Address:        <proxy_ip>#53

Non-authoritative answer:
database.clamav.net     canonical name = database.clamav.net.cdn.cloudflare.net.
Name:   database.clamav.net.cdn.cloudflare.net
Address: 104.16.219.84
Name:   database.clamav.net.cdn.cloudflare.net
Address: 104.16.218.84
Name:   database.clamav.net.cdn.cloudflare.net
Address: 2606:4700::6810:da54
Name:   database.clamav.net.cdn.cloudflare.net
Address: 2606:4700::6810:db54

Proxy server can forward DNS requests but cannot act as name server. When updating “nameserver” in /root/.cvdupdate/config.json with proxy address I got timeout like below:

# cvd update -V
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking main.cvd for update from https://database.clamav.net/main.cvd
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net
2021-03-16 07:40:28 cvdupdate-0.2.0 INFO Using nameserver specified in the config: 10.111.0.162
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking main.cvd version via DNS TXT advertisement.
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG main.cvd version advertised by DNS: 59
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Downloading CDIFFs first...
2021-03-16 07:40:28 cvdupdate-0.2.0 DEBUG Checking for main-59.cdiff
2021-03-16 07:40:28 urllib3.connectionpool DEBUG Starting new HTTPS connection (1): database.clamav.net:443
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 182, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f7ccf06e6a0>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='database.clamav.net', port=443): Max retries exceeded with url: /main-59.cdiff (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7ccf06e6a0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/cvd", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/__main__.py", line 257, in update_alias
    ctx.forward(db_update)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 628, in forward
    return self.invoke(cmd, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/__main__.py", line 100, in db_update
    errors = m.db_update(db)
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 781, in db_update
    if update(db) == False:
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 767, in update
    return self._download_cvd(db, advertised_version)
  File "/usr/local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 612, in _download_cvd
    'User-Agent': f'ClamAV/{self.dns_version_tokens[0]} (cvdupdate-{self.version})',
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='database.clamav.net', port=443): Max retries exceeded with url: /main-59.cdiff (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7ccf06e6a0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

I am wondering if it’s and option to forward request via proxy correctly, i.e. putting proxy ip in /root/.cvdupdate/config.json as a feature.

[micahsnyder]

It looks like another user was able to get cvdupdate working with a proxy this past weekend: https://github.com/micahsnyder/cvdupdate/issues/7

I'd like to verify their solution and (if it works) add it to the README file. If you get it working, would you consider helping me with the documentation?

[devdroid0]

I am having problems with the proxy settings as well. Our proxy requires authentication and request is not parsing it: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 412, in send conn = self.get_connection(request.url, proxies) File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 305, in get_connection proxy_url = parse_url(proxy) File "/usr/local/lib/python3.6/site-packages/urllib3-1.26.4-py3.6.egg/urllib3/util/url.py", line 392, in parse_url return six.raise_from(LocationParseError(source_url), None) File "", line 3, in raise_from urllib3.exceptions.LocationParseError: Failed to parse: requests.exceptions.InvalidURL: Failed to parse: http://myUsername:myPassword@proxy.domain.com:80/

It seems to be using the http_proxy and https_proxy environment variables. Tried lots of different combinations but to get this far and the proxy to not reject the connection (407) this is a close as I've gotten.

[marekgrajcar]

Hi,

I was able to reproduce solution from https://github.com/micahsnyder/cvdupdate/issues/7. Now I am able to reach for clamav signatures being behind proxy. What I have done is:

1. In /root/.cvdupdate/config.json I have updated nameserver with proxy address: "nameserver": "<proxy_ip>",

2. While running cvd update inside of script I set http_proxy and https_proxy and that usually does a trick: #!/bin/bash http_proxy=http://<proxy_ip>:<proxy_port> export http_proxy https_proxy=http://<proxy_ip>:<proxy_port> export https_proxy /usr/local/bin/cvd update -V

[devdroid0]

Use a script similar to above with our specific proxy information. We have to use authentication with our proxy. All the attempt to not use authentication string in the URL resulted in a 407:

ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required',

Iteration 1: Set the nameserver kev value in the config.json to one of the ip addresses of our proxy server and got this:

2021-03-18 06:31:50 cvdupdate-0.3.0 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net 2021-03-18 06:31:50 cvdupdate-0.3.0 INFO Using nameserver specified in the config: 10.10.7.8 2021-03-18 06:31:55 cvdupdate-0.3.0 DEBUG EXCEPTION OCCURRED: The DNS operation timed out after 5.509402751922607 seconds 2021-03-18 06:31:55 cvdupdate-0.3.0 WARNING Failed to determine available version via DNS TXT query! Traceback (most recent call last): File "/usr/local/bin/cvd", line 11, in load_entry_point('cvdupdate==0.3.0', 'console_scripts', 'cvd')() File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 829, in call return self.main(args, kwargs) File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 610, in invoke return callback(args, kwargs) File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/decorators.py", line 21, in new_func return f(get_current_context(), args, kwargs) File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/main.py", line 259, in update_alias File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 628, in forward return self.invoke(cmd, kwargs) File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 610, in invoke return callback(args, kwargs) File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/main.py", line 101, in db_update File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/cvdupdate.py", line 795, in db_update KeyError: ''

Iteration 2: changed the nameserver key value in the config.json to an ip address that was a nameserver and got a little farther:

2021-03-18 06:33:09 cvdupdate-0.3.0 DEBUG Checking available versions via DNS TXT entry query of current.cvd.clamav.net 2021-03-18 06:33:09 cvdupdate-0.3.0 INFO Using nameserver specified in the config: 10.0.0.100 2021-03-18 06:33:09 cvdupdate-0.3.0 DEBUG Checking main.cvd for update from https://database.clamav.net/main.cvd 2021-03-18 06:33:09 cvdupdate-0.3.0 DEBUG Checking main.cvd version via DNS TXT advertisement. 2021-03-18 06:33:09 cvdupdate-0.3.0 DEBUG main.cvd version advertised by DNS: 59 2021-03-18 06:33:09 cvdupdate-0.3.0 DEBUG Downloading CDIFFs first... 2021-03-18 06:33:09 cvdupdate-0.3.0 DEBUG Checking for main-59.cdiff Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 412, in send conn = self.get_connection(request.url, proxies) File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 305, in get_connection proxy_url = parse_url(proxy) File "/usr/local/lib/python3.6/site-packages/urllib3-1.26.4-py3.6.egg/urllib3/util/url.py", line 392, in parse_url return six.raise_from(LocationParseError(source_url), None) File "", line 3, in raise_from urllib3.exceptions.LocationParseError: Failed to parse: http://myUsername:myPassword@proxy.domain.com:80/

During handling of the above exception, another exception occurred:``` Traceback (most recent call last): File "/usr/local/bin/cvd", line 11, in load_entry_point('cvdupdate==0.3.0', 'console_scripts', 'cvd')() File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 829, in call return self.main(args, kwargs) File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 610, in invoke return callback(args, kwargs) File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/decorators.py", line 21, in new_func return f(get_current_context(), args, kwargs) File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/main.py", line 259, in update_alias File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 628, in forward return self.invoke(cmd, kwargs) File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 610, in invoke return callback(args, kwargs) File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/main.py", line 101, in db_update File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/cvdupdate.py", line 859, in db_update File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/cvdupdate.py", line 845, in update File "/usr/local/lib/python3.6/site-packages/cvdupdate-0.3.0-py3.6.egg/cvdupdate/cvdupdate.py", line 661, in _download_cvd File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/api.py", line 76, in get return request('get', url, params=params, kwargs) File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/api.py", line 61, in request return session.request(method=method, url=url, kwargs) File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 542, in request resp = self.send(prep, send_kwargs) File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/sessions.py", line 655, in send r = adapter.send(request, kwargs) File "/usr/local/lib/python3.6/site-packages/requests-2.25.1-py3.6.egg/requests/adapters.py", line 414, in send raise InvalidURL(e, request=request) requests.exceptions.InvalidURL: Failed to parse: http://myUser:myPassword@proxy.domain.com:80/

[devdroid0]

Finally got it to work. had to update our proxy with the required user-agent to allow it through unauthenticated.

Set the name server to one of our real name servers; Set the httpd_proxy and https_proxy environment variables to our proxy FQDN:port (multiple proxy addresses)

[micahsnyder]

Great! I'll try to summarize all of this in the Readme when time permits. I'll keep this ticket open until I've done so.