--random-agent can't find file

What's the problem (or question)?

When using the --random-agent flag in release version 1.21, the program returns a FileNotFoundError:

[>] Random User-Agent flag set.
Traceback (most recent call last):
  File "/usr/local/bin/VHostScan", line 11, in <module>
    load_entry_point('VHostScan==1.8.3', 'console_scripts', 'VHostScan')()
  File "/usr/local/lib/python3.6/dist-packages/VHostScan-1.8.3-py3.6.egg/VHostScan/VHostScan.py", line 54, in main
    user_agents = load_random_user_agents()
  File "/usr/local/lib/python3.6/dist-packages/VHostScan-1.8.3-py3.6.egg/VHostScan/lib/helpers/file_helper.py", line 65, in load_random_user_agents
    with open('./lib/ua-random-list.txt') as f:
FileNotFoundError: [Errno 2] No such file or directory: './lib/ua-random-list.txt'

Running VHostScan from a parent directory that contains lib/ua-random-list.txt seems to solve the problem.

Do you have an idea for a solution?

Consider having the user manually specify the file containing the list of user agents, or modify the load_random_user_agents() function to specify the base path of where ua-random-list.txt should exist. One example fix:

def load_random_user_agents():
    dir = os.path.dirname(__file__)

    # in my case, this looks for the file in the directory above /usr/local/lib/python3.6/dist-packages/VHostScan-1.8.3-py3.6.egg/VHostScan/lib/helpers
    with open(os.path.join(dir, '../ua-random-list.txt')) as f:
        return f.readlines()

NOTE: When I modified this on my machine, I then received another error indicating an invalid header value:

Traceback (most recent call last):
  File "/usr/local/bin/VHostScan", line 11, in <module>
    load_entry_point('VHostScan==1.8.3', 'console_scripts', 'VHostScan')()
  File "/usr/local/lib/python3.6/dist-packages/VHostScan-1.8.3-py3.6.egg/VHostScan/VHostScan.py", line 101, in main
    scanner.scan()
  File "/usr/local/lib/python3.6/dist-packages/VHostScan-1.8.3-py3.6.egg/VHostScan/lib/core/virtual_host_scanner.py", line 147, in scan
    res = requests.get(dest_url, headers=headers, verify=False)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/api.py", line 71, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/adapters.py", line 403, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/packages/urllib3/connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/dist-packages/requests-2.10.0-py3.6.egg/requests/packages/urllib3/connectionpool.py", line 362, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1280, in _send_request
    self.putheader(hdr, value)
  File "/usr/lib/python3.6/http/client.py", line 1217, in putheader
    raise ValueError('Invalid header value %r' % (values[i],))
ValueError: Invalid header value b'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36\n'

I believe this other issue can be resolved by not reading in newline characters \n when reading the user agent file on line 66 of lib/helpers/file_helper.py. I can create another issue to discuss this if necessary.

How can we reproduce the issue?

Install latest VHostScan release 1.21
Run VHostScan with --random-agent flag against any domain from any directory vs. running it within /VHostScan-1.21/VHostScan/

What are the running context details?

Installation method (e.g. pip, apt-get, git clone or zip/tar.gz):
- Downloaded *.tar.gz release 1.21 from GitHub. Installed with python3 setup.py install.
Client OS (e.g. Microsoft Windows 10):
- Observed on both Kali 2019.2 and Ubuntu 18.04.2 LTS
Program version (see banner):
- 1.21
Target Web Server if known (e.g. Apache):
- Any
Detected WAF/IDS/IPS protection (e.g. ModSecurity or unknown):
- n/a

codingo / VHostScan