search

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.
http://www.spiderfoot.net
MIT License
12.69k stars 2.21k forks source link

Installation Errors and SSL Certificate Verification, TLD Failure in Ubuntu Linux #1870

Open jacobstickney opened 3 months ago

jacobstickney commented 3 months ago

Hello,

I downloaded the Stable version in Ubuntu Linux 24.04, and have Python 3.12.3 installed.

When I run pip3 install -r requirements.txt, it installs all of the requirements until after PyYAML. It then returns this error:

Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/home/jstickney/spiderfoot-4.0/spiderfoot-env/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/jstickney/spiderfoot-4.0/spiderfoot-env/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/jstickney/spiderfoot-4.0/spiderfoot-env/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 968, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 321, in run
          self.find_sources()
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
          mm.run()
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 550, in run
          self.add_defaults()
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 588, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/command/sdist.py", line 102, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 250, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 335, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-auru8hoe/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

After I run python3 ./sf.py -l 127.0.0.1:5001, it is unable to start the server and instead outputs the name of each missing module, such as:

Traceback (most recent call last):
  File "/home/jstickney/spiderfoot-4.0/./sf.py", line 31, in <module>
    from sfwebui import SpiderFootWebUi
  File "/home/jstickney/spiderfoot-4.0/sfwebui.py", line 27, in <module>
    from mako.lookup import TemplateLookup
ModuleNotFoundError: No module named 'mako'

After installing one module via pip3, the name of the next missing module is displayed after attempting to start Spiderfoot. Once all 20+ of the modules are installed, I am able to start the Spiderfoot server and navigate in my web browser to the scan creation page. After configuring and starting the scan, the scan stalls at the "Created" stage, and the following error is returned:

2024-06-17 13:14:56,352 [INFO] sfwebui : Waiting for the scan to initialize...
2024-06-17 13:14:56,515 [INFO] sflib : Downloading configuration data from: https://publicsuffix.org/list/effective_tld_names.dat
2024-06-17 13:14:56,676 [ERROR] sflib : Unable to open option URL, https://publicsuffix.org/list/effective_tld_names.dat: HTTPSConnectionPool(host='publicsuffix.org', port=443): Max retries exceeded with url: /list/effective_tld_names.dat (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')))
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/jstickney/spiderfoot-4.0/sfscan.py", line 30, in startSpiderFootScanner
    return SpiderFootScanner(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jstickney/spiderfoot-4.0/sfscan.py", line 206, in __init__
    self.__sf.cachePut("internet_tlds", self.__config['_internettlds'])
  File "/home/jstickney/spiderfoot-4.0/sflib.py", line 291, in cachePut
    fp.write(data)
TypeError: write() argument must be str, not None

The error message appears to indicate an SSL certificate verification failure when trying to download the latest TLD list from publicsuffix.org. However, an upgrade of certifi, which includes the certificate authority bundle that Requests uses to validate certificates, has already been satisfied.

When I install the Development build, there are no pre-scan installation errors. However, when I start a scan, the scan stalls at "ERROR-FAILED", and the log output errors this message:

2024-06-17 13:54:10,136 [INFO] sfwebui : Waiting for the scan to initialize...
2024-06-17 13:54:10,247 [INFO] sflib : Downloading configuration data from: https://publicsuffix.org/list/effective_tld_names.dat
2024-06-17 13:54:10,448 [ERROR] sflib : Unable to open option URL, https://publicsuffix.org/list/effective_tld_names.dat: HTTPSConnectionPool(host='publicsuffix.org', port=443): Max retries exceeded with url: /list/effective_tld_names.dat (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')))
2024-06-17 13:54:10,448 [INFO] sflib : Scan [90BCFA98] failed: Could not update TLD list
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/jstickney/spiderfoot/sfscan.py", line 28, in startSpiderFootScanner
    return SpiderFootScanner(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jstickney/spiderfoot/sfscan.py", line 206, in __init__
    raise ValueError("Could not update TLD list")
ValueError: Could not update TLD list
LAkahlor commented 2 weeks ago

I'm was having the same issues with PyYaml versions, I actually installed latest 6 "Pip3 install PyYaml" and removed it from the requirements.txt and then re-ran the reinstalled the requirements.txt.

Remove the highlighted line: image

pip3 install pyyaml Just make sure some version of it's loaded. Rerun the requirements.txt `pip3 install -r requirements.txt

I got it working with that.