Windows compatibility for AppLogger & CWEDownloads

oh2fih commented 1 month ago

Import FullSysLogHandler from CveXplore.core.logging.handlers.syslog_handler only if configured. Ref. https://github.com/cve-search/cve-search/issues/143#issuecomment-2275723579
Fix urllib.error.URLError in class CWEDownloads.

oh2fih commented 1 month ago

Testing

Environment

OS: Windows 11 23H2 build 22631.3880
Python 3.12.5 (64-bit)
MongoDB 7.0.12 2008R2Plus SSL (64 bit)
CVE-Search v5.1.0 cloned from GitHub.
CveXplore 0.3.35 from PyPI with this patch applied manually.

Results

Running python.exe .\sbin\db_updater.py -f -c:

✅ The ModuleNotFoundError: No module named 'syslog' error is solved.

There are some unrelated errors regarding printing certain Unicode characters like スリーコム in NVD API responses to the terminal. I don't think we need handling for these errors as they are not terminating the db_updater.py, but only printing stack trace occasionally:

--- Logging error ---
Traceback (most recent call last):
File "%LocalAppData%\Programs\Python\Python312\Lib\logging\__init__.py", line 1163, in emit
  stream.write(msg + self.terminator)
File "%LocalAppData%\Programs\Python\Python312\Lib\encodings\cp1252.py", line 19, in encode
  return codecs.charmap_encode(input,self.errors,encoding_table)[0]
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 538-542: character maps to <undefined>
Call stack:
File "cve-search\sbin\db_updater.py", line 296, in <module>
  main(args)
File "cve-search\sbin\db_updater.py", line 74, in main
  cvex.database.initialize()
File "%LocalAppData%\Programs\Python\Python312\Lib\site-packages\CveXplore\core\database_maintenance\main_updater.py", line 198, in initialize
  cpe_pop.populate()
File "%LocalAppData%\Programs\Python\Python312\Lib\site-packages\CveXplore\core\database_maintenance\sources_process.py", line 270, in populate
  self.process_downloads()
File "%LocalAppData%\Programs\Python\Python312\Lib\site-packages\CveXplore\core\database_maintenance\sources_process.py", line 135, in process_downloads
  for entry in self.api_handler.get_all_data(data_type="cpe"):
File "%LocalAppData%\Programs\Python\Python312\Lib\site-packages\CveXplore\core\nvd_nist\nvd_nist_api.py", line 313, in get_all_data
  data = self.get_count(
File "%LocalAppData%\Programs\Python\Python312\Lib\site-packages\CveXplore\core\nvd_nist\nvd_nist_api.py", line 276, in get_count
  self.logger.debug(f"API response data: {ret_data}")
File "cve-search\sbin\..\lib\LogHandler.py", line 91, in debug
  return super(HelperLogger, self).debug(msg, *args, **kwargs)
Message: "\x1b[35mAPI response data: {'resultsPerPage': 1, 'startIndex': 0, 'totalResults': 1287770, 'format': 'NVD_CPE', 'version': '2.0', 'timestamp': '2024-08-08T14:15:15.550', 'products': [{'cpe': {'deprecated': False, 'cpeName': 'cpe:2.3:a:3com:3cdaemon:-:*:*:*:*:*:*:*', 'cpeNameId': 'BAE41D20-D4AF-4AF0-AA7D-3BD04DA402A7', 'lastModified': '2011-01-12T14:35:43.723', 'created': '2007-08-23T21:05:57.937', 'titles': [{'title': '3Com 3CDaemon', 'lang': 'en'}, {'title': 'スリーコム 3CDaemon', 'lang': 'ja'}]}}]}\x1b[0m"
Arguments: ()

oh2fih commented 1 month ago

Fixed in https://github.com/cve-search/CveXplore/pull/301/commits/43f7636f39c31c4d5b74cc3df81e11e1a8d2a59b, there was another compatibility issue with Windows during processing CWE data.

  File ""%LocalAppData%\Programs\Python\Python312\Lib\site-packages\CveXplore\core\database_maintenance\sources_process.py", line 1003, in file_to_queue
    self.parser.parse(f"file://{filename}")

urllib.error.URLError: <urlopen error [WinError 3] The system cannot find the path specified: ''>

oh2fih commented 1 month ago

Despite there are still Windows incompatibility issues in cve-search...

.\web\index.py fails with ModuleNotFoundError: No module named 'fcntl'
- 🚫 this is unsolvable as Gunicorn does not support Windows; https://github.com/benoitc/gunicorn/issues/524

...and issues with Python 3.12...

.\bin\search.py fails with ModuleNotFoundError: No module named 'pkg_resources'

Traceback (most recent call last):
  File "cve-search\bin\search.py", line 20, in <module>
    import requirements
  File "%LocalAppData%\Programs\Python\Python312\Lib\site-packages\requirements\__init__.py", line 19, in <module>
    from .parser import parse
  File "%LocalAppData%\Programs\Python\Python312\Lib\site-packages\requirements\parser.py", line 23, in <module>
    from .requirement import Requirement
  File "%LocalAppData%\Programs\Python\Python312\Lib\site-packages\requirements\requirement.py", line 24, in <module>
    from pkg_resources import Requirement as Req
ModuleNotFoundError: No module named 'pkg_resources'

🚧 workaround is to pip install setuptools. However, the pkg_resources from setuptools has been deprecated: https://github.com/madpah/requirements-parser/issues/88

...this fixes database updates for CveXplore on Windows.

The database now contains all the necessary information, e.g.,

>>> from CveXplore import CveXplore
>>> cvx = CveXplore()
>>> result = cvx.get_multi_store_entries([("CWE", {"id": "78"}), ("cves", {"id": "CVE-2024-6652"})])
>>> result
[<< Cwe:78 >>, << Cves:CVE-2024-6652 >>]

cve-search / CveXplore