Getting OE layer list does not work when working behind a proxy

AlexanderAduda commented 2 days ago

Calling the script like this will download information from the OE layer index

bd-scan-yocto-via-sbom \
--blackduck_url <URL> \
--blackduck_api_token <TOKEN> \
--blackduck_trust_cert \
-t some-image \
-p some-project \
-v some-version \
--oe_data_folder ./build/oe-data \
--license_manifest ./build/tmp/deploy/licenses/some_target/some-image-some-target.rootfs/license.manifest  \
--cve_check_file ./build/tmp/deploy/images/some-target/some-image-some-target.rootfs.cve \
--download_dir ./build/downloads \
--package_dir ./build/tmp/deploy/rpm \
--build_dir ./build \
--image_package_type rpm

urllib raises an error when working behind a company proxy with some company specific certificate will not work due to unkown "self sigend"

INFO:root:--- PHASE 2 - GET OE DATA ------------------------------------------------
INFO:root:Processing OE recipes and layers ...
INFO:root:- Getting OE layers
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1000)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<project path>/.venv/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "<project path>/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "<project path>/.venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='layers.openembedded.org', port=443): Max retries exceeded with url: /layerindex/api/layerItems/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1000)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<project path>/.venv/bin/bd-scan-yocto-via-sbom", line 8, in <module>
    sys.exit(main.main())
             ^^^^^^^^^^^
  File "<project path>/.venv/lib/python3.12/site-packages/yocto_import_sbom/main.py", line 40, in main
    oe_class = OE(conf)
               ^^^^^^^^
  File "<project path>/.venv/lib/python3.12/site-packages/yocto_import_sbom/OEClass.py", line 14, in __init__
    self.layers = self.get_oe_layers(conf)
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "<project path>/.venv/lib/python3.12/site-packages/yocto_import_sbom/OEClass.py", line 35, in get_oe_layers
    r = requests.get(url)
        ^^^^^^^^^^^^^^^^^
  File "<project path>/.venv/lib/python3.12/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<project path>/.venv/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<project path>/.venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<project path>/.venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<project path>/.venv/lib/python3.12/site-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='layers.openembedded.org', port=443): Max retries exceeded with url: /layerindex/api/layerItems/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1000)')))

matthewb66 commented 2 days ago

Coding a custom utility to support the 20+ types of proxy deployments is not possible. Python requests is documented at https://requests.readthedocs.io/en/latest/user/advanced including how to support proxies. Options to resolve this include:

Downloading the OE data files manually and placing in a folder for reference by the utility (you could run the script externally to get the data and copy the files internally given they change slowly)
Use the HTTP_PROXY or HTTPS_PROXY environment variables as documented in the requests library

AlexanderAduda commented 2 days ago

this is not about support, it's about an option to ignore it. I have set HTTP(S)_PROXY but setting HTTP_PROXY does not fixing an SSL issue just have a look at the PR, this option won't hurt anyone

blackducksoftware / bd_scan_yocto_via_sbom

Getting OE layer list does not work when working behind a proxy #20