FileNotFoundError

[rllin-fathom]

----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58822)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/socketserver.py", line 650, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/local/lib/python3.7/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python3.7/site-packages/seleniumwire/proxy/proxy2.py", line 65, in __init__
    super().__init__(*args, **kwargs)
  File "/usr/local/lib/python3.7/socketserver.py", line 720, in __init__
    self.handle()
  File "/usr/local/lib/python3.7/http/server.py", line 428, in handle
    self.handle_one_request()
  File "/usr/local/lib/python3.7/http/server.py", line 414, in handle_one_request
    method()
  File "/usr/local/lib/python3.7/site-packages/seleniumwire/proxy/proxy2.py", line 207, in do_GET
    res_body_modified = self.response_handler(req, req_body, res, res_body_plain)
  File "/usr/local/lib/python3.7/site-packages/seleniumwire/proxy/handler.py", line 166, in response_handler
    self.server.storage.save_response(req.id, res, res_body)
  File "/usr/local/lib/python3.7/site-packages/seleniumwire/proxy/storage.py", line 118, in save_response
    self._save(response_data, request_dir, 'response')
  File "/usr/local/lib/python3.7/site-packages/seleniumwire/proxy/storage.py", line 90, in _save
    with open(os.path.join(request_dir, filename), 'wb') as out:
FileNotFoundError: [Errno 2] No such file or directory: '/root/.seleniumwire/storage-d63ae536-e5fe-4932-afba-99b27c1f99e7/request-05ad1193-431f-48e5-83a3-c56e25745f58/response'
----------------------------------------

I often get errors like this, sometimes they stop the show, sometimes they're shown as a result of an actual Selenium side exception. i'm running via a Docker image with chrome driver installed

[rllin-fathom]

seems like if i do driver.quit() instead of driver.close() i don't see this anymore

[wkeeling]

@rllin-fathom thanks for reporting this. We'll investigate why that condition sometimes occurs with .close()

[wkeeling]

@rllin-fathom - so there is an issue with driver.close() where responses from previous requests can sometimes be processed by Selenium Wire, even though the test has effectively ended, triggering the error you're seeing. driver.quit() handles the situation better by explicitly preventing delayed responses from being processed.

I have put a fix in which should resolve the issue for driver.close(). That is currently in master and will be available in the next release (1.0.3).

Thank you for raising the issue!

[rllin-fathom]

@wkeeling thanks for looking and addressing :)

[feal94]

Hello, I found a similar error but in a different part of the code. To give some background, I'm running several instances of Selenium-wire with firefox, each one of them opens a new webdriver, visits a page, does work and finally quits (driver.quit()). It all works fine for a while, but at some point the whole thing hangs with no error, but if I do an KeyboardInterrupt I can actually see this error:

Traceback (most recent call last):
File "/home/alvarofeal/.local/lib/python3.5/site-packages/joblib/externals/loky/process_executor.py", line 418, in _process_worker
r = call_item()
File "/home/alvarofeal/.local/lib/python3.5/site-packages/joblib/externals/loky/process_executor.py", line 272, in __call__
return self.fn(*self.args, **self.kwargs)
File "/home/alvarofeal/.local/lib/python3.5/site-packages/joblib/_parallel_backends.py", line 567, in __call__
return self.func(*args, **kwargs)
File "/home/alvarofeal/.local/lib/python3.5/site-packages/joblib/parallel.py", line 225, in __call__
for func, args, kwargs in self.items]
File "/home/alvarofeal/.local/lib/python3.5/site-packages/joblib/parallel.py", line 225, in <listcomp>
for func, args, kwargs in self.items]
File "crawl_pages.py", line 23, in crawl_page
driver = get_driver(browser)
File "crawl_pages.py", line 80, in get_driver
driver = webdriver.Firefox(executable_path=firefox_driver, options=options)
File "/home/alvarofeal/.local/lib/python3.5/site-packages/seleniumwire/webdriver/browser.py", line 27, in __init__
options=seleniumwire_options
File "/home/alvarofeal/.local/lib/python3.5/site-packages/seleniumwire/proxy/client.py", line 54, in create_proxy
proxy_config=proxy_config, options=options)
File "/home/alvarofeal/.local/lib/python3.5/site-packages/seleniumwire/proxy/server.py", line 15, in __init__
self.storage = RequestStorage()
File "/home/alvarofeal/.local/lib/python3.5/site-packages/seleniumwire/proxy/storage.py", line 41, in __init__
self._cleanup_old_dirs()
File "/home/alvarofeal/.local/lib/python3.5/site-packages/seleniumwire/proxy/storage.py", line 329, in _cleanup_old_dirs
if (os.path.getmtime(storage_dir) <
File "/usr/lib/python3.5/genericpath.py", line 55, in getmtime
return os.stat(filename).st_mtime
FileNotFoundError: [Errno 2] No such file or directory: '/home/alvarofeal/.seleniumwire/storage-dd467e88-9a0a-4352-bdfe-45901ee3021a'

This is a wild guess but could it be that calling several driver in parallel triggers the cleanup_old_dirs function in such a way that a previous driver removes the dirs and the next one fails to find them later?

Oh, in case this helps. Using chrome with driver.close() works like a charm but driver.quit() causes the same issue.

[wkeeling]

@feal94 yes you're absolutely right, selenium wire stores captured requests in a global per-user directory - by default ~/.seleniumwire. If you run multiple instances of selenium wire in separate threads, some threads will likely clean up the directory whilst other threads are still trying to use it.

Clean up takes place on driver.quit() as you've discovered, but also when the Python interpreter process exits - which can cause a separate issue in certain circumstances (see #56).

To fix this issue and also #56, I'm thinking it might be more sensible to perform cleanup on test start rather than quit/end, at the expense of leaving a few temporary files lying around. I'll look at implementing something asap.

Thanks for flagging this one.

[feal94]

@wkeeling what I've done as a temporary workaround is to just catch the Exception and pass. I know this is not the most elegant, but since the Exception is raised because the folder does not exist anymore, ignoring it should not cause any problem.

If anyone runs into the same issue they might want to do that while the issue is still open.

Thanks!

[wkeeling]

@feal94 there's a new version available, 1.0.7, which fixes this issue.

[ma-marco-mariotto]

Hi, I would like to reopen this. I get the following error, using seleniumwire 5.0.0. Error handling request Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/seleniumwire/handler.py", line 67, in request self.proxy.storage.save_request(request) File "/usr/local/lib/python3.10/dist-packages/seleniumwire/storage.py", line 90, in save_request os.mkdir(request_dir) FileNotFoundError: [Errno 2] No such file or directory: '/tmp/.seleniumwire/storage-4bb95fa8-9cba-4cee-ad2d-d98ee79bdb61/request-a6156adb-7a73-4475-9e04-ff9d5fc00d37'

My script uses a few threads each handling a Firefox driver; when an expected request is intercepted along with the response, each thread performs del driver.requests to clean up the requests. Once the thread is done, it calls driver.quit(). I'm not sure if this is the problem, but I have to do this clean up otherwise driver.wait_for_request will match to old requests.

[ma-marco-mariotto]

I think the reason is that you are trying to make a directory when the intermediate one does not exist: in my case storage-4bb95fa8-9cba-4cee-ad2d-d98ee79bdb61

[ma-marco-mariotto]

Again, a similar error pops up in _save:

Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/seleniumwire/handler.py", line 67, in request self.proxy.storage.save_request(request) File "/usr/local/lib/python3.10/dist-packages/seleniumwire/storage.py", line 93, in save_request self._save(request, request_dir, 'request') File "/usr/local/lib/python3.10/dist-packages/seleniumwire/storage.py", line 99, in _save with open(os.path.join(dirname, filename), 'wb') as out: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/.seleniumwire/storage-1b766480-0ee9-49e2-b293-8a43a7f2f1c3/request-2c58d886-f1a2-4a69-858e-f7fe09feee79/request'

I think it's for the same reason