selenium-wier on Ubuntu server - Need Help

[monalisir]

Hi wkeeling, thanks for your code, I'm using it to intercept the request we sent. It runs well on my Mac with upstream proxy(luminati), But after I deployed it on the Ubuntu server, It hangs up on the sites we request, such as https://google.com. follow the instuctions, I have installed the ca.cert in ubuntu server cert folder /usr/local/share/ca-certificates and sudo update-ca-certificates it.and finally timed out. Do you have some experience work it on Ubuntu server? For there is no interface on the server, I can not handle the cert. I thought it would be the cert problem, so they can not connect with the proxy. How Can I check that If I installed it correctly? and the chromium will trust the cert? Thanks.

chromium version/driver: 75 selenium-wire: 3.0.5

[wkeeling]

I don't think it's an issue with Chromium trusting the certificate, because Selenium Wire instructs the browser to ignore certificate errors. There may be a few reasons for the "No such file or directory" error.

First, does the Ubuntu server have OpenSSL installed? Selenium Wire invokes the binary openssl command to generate certificates. You can confirm this by running sudo apt install openssl.

Second, is the default location of Selenium Wire's storage directory accessible on the Ubuntu server? You can change this with the request_storage_base_dir option. Try changing it to /tmp:

sw_options = {
    'request_storage_base_dir': '/tmp'
}
driver = webdriver.Chrome(seleniumwire_options=sw_options)

If you're still seeing an error after trying the above, try switching to the mitmproxy backend. This uses different code to capture the requests so you may find it resolves the issue, or gives us further clues.

Finally, make sure you have enabled logging in your code, as Selenium Wire may be generating a traceback that you're not seeing:

import logging
logging.basicConfig(level=logging.DEBUG)

[monalisir]

I don't think it's an issue with Chromium trusting the certificate, because Selenium Wire instructs the browser to ignore certificate errors. There may be a few reasons for the "No such file or directory" error.

First, does the Ubuntu server have OpenSSL installed? Selenium Wire invokes the binary openssl command to generate certificates. You can confirm this by running sudo apt install openssl.

Second, is the default location of Selenium Wire's storage directory accessible on the Ubuntu server? You can change this with the request_storage_base_dir option. Try changing it to /tmp:

sw_options = {
    'request_storage_base_dir': '/tmp'
}
driver = webdriver.Chrome(seleniumwire_options=sw_options)

If you're still seeing an error after trying the above, try switching to the mitmproxy backend. This uses different code to capture the requests so you may find it resolves the issue, or gives us further clues.

Finally, make sure you have enabled logging in your code, as Selenium Wire may be generating a traceback that you're not seeing:

import logging
logging.basicConfig(level=logging.DEBUG)

thanks for your reply, I have changed the storage_base_dir to '/tmp', it seems not work. also, I found the logging settings in the selenium-wire and I have not change it. which in the file main.py.

Is there any limitation of the mitmproxy version? I tried this, but it failed when launch it. Message: 'Created proxy listening on %s:%s' Arguments: ('::ffff:127.0.0.1', 55451, 0, 0) spider_opened_handler==>too many values to unpack (expected 2) the mitmproxy returned 4 parameters but selenium-wire used only 2. the _master.server.address returned a tuple with 4 parameters, so here might be error. addr, port = self.proxy.address()

mitmproxy version: 5.3.0 seleniume-wire: 4.0.0

[wkeeling]

Thanks. Selenium Wire has been tested with 5.3.0 and 6.0.2 of mitmproxy, so should be fine with the version you're using. It seems that the issue is caused by mitmproxy binding to an IPv6 socket which returns more than 2 values for the address. This will need a minor fix to resolve. I'll look at doing that ASAP and let you know once available.

[wkeeling]

@monalisir version 4.0.1 is now available which should fix the parameter error you're seeing. Are you able to give that a go?

[monalisir]

thanks, I tested it. it failed on Mac with 4.0.0 but works find on Ubuntu server. It works now. thanks ;-)