Open Fogapod opened 6 years ago
same exception happens without using uvloop
Could you post the exact error message when not using uvloop? uvloop isn't supported right now because the last time I tried there were some issues that I couldn't work around.
Could you post the exact error message when not using uvloop?
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/arsenic/__init__.py", line 16, in __aenter__
self.session = await start_session(self.service, self.browser, self.bind)
File "/usr/local/lib/python3.6/dist-packages/arsenic/__init__.py", line 28, in start_session
driver = await service.start()
File "/usr/local/lib/python3.6/dist-packages/arsenic/services.py", line 99, in start
self.log_file
File "/usr/local/lib/python3.6/dist-packages/arsenic/services.py", line 34, in subprocess_based_service
process = await impl.start_process(cmd, log_file)
File "/usr/local/lib/python3.6/dist-packages/arsenic/subprocess.py", line 61, in start_process
stdin=DEVNULL,
File "/usr/lib/python3.6/asyncio/subprocess.py", line 225, in create_subprocess_exec
stderr=stderr, **kwds)
File "/usr/lib/python3.6/asyncio/base_events.py", line 1192, in subprocess_exec
bufsize, **kwargs)
File "/usr/lib/python3.6/asyncio/unix_events.py", line 200, in _make_subprocess_transport
**kwargs)
File "/usr/lib/python3.6/asyncio/base_subprocess.py", line 39, in __init__
stderr=stderr, bufsize=bufsize, **kwargs)
File "/usr/lib/python3.6/asyncio/unix_events.py", line 706, in _start
universal_newlines=False, bufsize=bufsize, **kwargs)
File "/usr/lib/python3.6/subprocess.py", line 667, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "/usr/lib/python3.6/subprocess.py", line 1184, in _get_handles
c2pwrite = stdout.fileno()
io.UnsupportedOperation: fileno
As a workaround, pass a log_file
to Geckodriver
(or whatever driver you are using). It should be an object that is valid for stdout
/stderr
as described in https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.AbstractEventLoop.subprocess_exec.
I added log_file
parameter, now I have different issue. I get the following output:
2018-05-28 05:42.02 request body={"desiredCapabilities": {"browserName": "firefox", "marionette": true, "acceptInsecureCerts": true, "firefoxOptions": {"args": ["-headless"]}}} method=POST url=http://localhost:43633/session
2018-05-28 05:42.02 response body={"desiredCapabilities": {"browserName": "firefox", "marionette": true, "acceptInsecureCerts": true, "firefoxOptions": {"args": ["-headless"]}}} data={'value': {'error': 'unknown error', 'message': 'Process unexpectedly closed with status 1', 'stacktrace': ''}} method=POST response=<ClientResponse(http://localhost:43633/session) [500 Internal Server Error]>
<CIMultiDictProxy('Connection': 'close', 'Content-Type': 'application/json; charset=utf-8', 'Cache-Control': 'no-cache', 'Content-Length': '105', 'Date': 'Mon, 28 May 2018 05:42:02 GMT')>
url=http://localhost:43633/session
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/arsenic/__init__.py", line 16, in __aenter__
self.session = await start_session(self.service, self.browser, self.bind)
File "/usr/local/lib/python3.6/dist-packages/arsenic/__init__.py", line 29, in start_session
return await driver.new_session(browser, bind=bind)
File "/usr/local/lib/python3.6/dist-packages/arsenic/webdriver.py", line 57, in new_session
raise SessionStartError(err_resp['error'], err_resp.get('message', ''), original_response)
arsenic.errors.SessionStartError: unknown error: Process unexpectedly closed with status 1
Log file says I don't have display variable defined, but I'm using headless mode
1527486122031 geckodriver INFO geckodriver 0.20.1
1527486122039 geckodriver INFO Listening on 127.0.0.1:43633 1527486122544 mozrunner::runner INFO Running command: "/usr/bin/firefox" "-marionette" "-profile" "/tmp/rust_mozprofile.INA3ouudlnMM"
Error: no DISPLAY environment variable specified
My code:
from arsenic import get_session, services, browsers
service = services.Geckodriver(log_file=open('file.txt','w'))
browser = browsers.Firefox(firefoxOptions={ 'args': ['-headless'] })
async with get_session(service, browser) as session:
await session.get('https://google.com')
Looks like the options changed (either in FF or geckodriver). this worked for me:
browser = browsers.Firefox(**{'moz:firefoxOptions': { 'args': ['-headless'] }})
New issue occurred. Log file output:
1527489279485 geckodriver INFO geckodriver 0.20.1
1527489279495 geckodriver INFO Listening on 127.0.0.1:54365
1527489279997 mozrunner::runner INFO Running command: "/usr/bin/firefox" "-marionette" "-headless" "-profile" "/tmp/rust_mozprofile.Tq4p7Bgte7hq"
*** You are running in headless mode.
Assertion failure: [unhandlable oom] Failed to allocate object while tenuring., at /build/firefox-lojCEZ/firefox-60.0.1+build2/js/src/vm/JSContext.cpp:1662
I'm able to get page with synchronous selenium
library, so Firefox browser and driver are working
unhandlable oom
You ran out of RAM? (OOM = Out Of Memory)
Yes, seems like previous firefox sessions remained in memory and ate it. Is this a bug?
Output ps -aux | grep fire
:
kiwi 385 0.1 10.2 2151272 103512 pts/1 Sl+ 06:29 0:04 /usr/lib/firefox/firefox -marionette --headless -profile /tmp/rust_mozprofile.IDL1dqR12NKe
kiwi 452 0.0 2.1 1538776 21312 pts/1 Sl+ 06:29 0:00 /usr/lib/firefox/firefox -contentproc -childID 1 -isForBrowser -intPrefs 42:0|44:0|74:0| -boolPrefs 5:1|61:1|276:0|299:0| -stringPrefs 285:36;df6fb09a-00ad-4dfd-a53c-ef0582a18525| -schedulerPrefs 0001,2 -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 385 true tab
kiwi 505 0.0 1.6 1524400 16328 pts/1 Sl+ 06:29 0:00 /usr/lib/firefox/firefox -contentproc -childID 2 -isForBrowser -intPrefs 42:0|44:0|74:0| -boolPrefs 5:1|61:1|276:0|299:0| -stringPrefs 285:36;df6fb09a-00ad-4dfd-a53c-ef0582a18525| -schedulerPrefs 0001,2 -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 385 true tab
kiwi 538 0.2 10.8 2179688 109572 pts/1 Sl+ 06:30 0:05 /usr/lib/firefox/firefox -marionette --headless -profile /tmp/rust_mozprofile.7monsLLmwYcX
kiwi 588 0.0 9.2 1813584 93352 pts/1 Sl+ 06:30 0:01 /usr/lib/firefox/firefox -contentproc -childID 1 -isForBrowser -intPrefs 42:0|44:0|74:0| -boolPrefs 5:1|61:1|276:0|299:0| -stringPrefs 285:36;2fd196cf-10f5-46d7-b129-6416e05f7d59| -schedulerPrefs 0001,2 -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 538 true tab
kiwi 627 0.0 1.6 1524400 16348 pts/1 Sl+ 06:30 0:00 /usr/lib/firefox/firefox -contentproc -childID 2 -isForBrowser -intPrefs 42:0|44:0|74:0| -boolPrefs 5:1|61:1|276:0|299:0| -stringPrefs 285:36;2fd196cf-10f5-46d7-b129-6416e05f7d59| -schedulerPrefs 0001,2 -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 538 true tab
kiwi 1569 0.0 0.1 14856 1120 pts/3 S+ 07:14 0:00 grep --color=auto fire
After killing these I was able to navigate and get screenshot. Thanks for the support
Yes, seems like previous firefox sessions remained in memory and ate it. Is this a bug?
I don't know. I ran the code and it worked fine. Maybe it failed to clean up in the cases where it triggered one of the bugs/problems you ran into? I had to use docker since I don't have a native linux handy, so it cleaned up after each run.
I managed to run session using uvloop setting log_file
argument to os.devnull
I managed to run session using uvloop setting log_file argument to os.devnull
Awesome! I'll try to add it as a fully supported loop then (with CI etc)
Versions
OS: Ubuntu 18.04 LTS x86_64 Python: 3.6.5 firefox: 60.0.1 geckodriver 0.20.1 arsenic: 1.0.0.dev8
My code
Traceback