search

scrapinghub / splash

Lightweight, scriptable browser as a service with an HTTP API
BSD 3-Clause "New" or "Revised" License
4.08k stars 515 forks source link

SystemError exception #481

Open animalmatsuzawa opened 8 years ago

animalmatsuzawa commented 8 years ago

It made ​​an error After PNG output at large web page.

server:AWS EC2 ubuntu14.04 LTS

Exception has occurred in the following.

splash/qtrender_image.py:514:qimage_to_pil_image
      buf = qimage.bits().asstring(qimage.byteCount())

SystemError: Bad call flags in PyCFunction_Call. METH_OLDARGS is no longer supported!

pyqt5 is installed in the following.

sudo apt-get install qt5-default
sudo apt-get install libqt5webkit5-dev
sudo apt-get install python3-pyqt5
sudo apt-get install python3-pyqt5.qtwebkit

Log

2016-07-14 14:54:29.725300 [render] [139971093668136] wait timeout for 139971094425032
2016-07-14 14:54:29.725565 [render] [139971093668136] [lua_runner] dispatch cmd_id=1
2016-07-14 14:54:29.725700 [render] [139971093668136] [lua_runner] arguments are for command 1, waiting for result of 1
2016-07-14 14:54:29.725859 [render] [139971093668136] [lua_runner] entering dispatch/loop body, args=(PyResult('return', True),)
2016-07-14 14:54:29.726004 [render] [139971093668136] [lua_runner] send PyResult('return', True)
2016-07-14 14:54:29.726160 [render] [139971093668136] [lua_runner] send (lua) (b'return', True)
2016-07-14 14:54:29.726331 [render] [139971093668136] getting history
2016-07-14 14:54:29.727217 [render] [139971093668136] getting HTML
2016-07-14 14:54:29.741626 [render] [139971093668136] HAR event: _onHtmlRendered
2016-07-14 14:54:29.743700 [render] [139971093668136] Getting PNG: width=None, height=None, render_all=True, scale_method=None, region=None
2016-07-14 14:54:29.743865 [render] [139971093668136] Rendering whole page contents (RENDER_ALL)
2016-07-14 14:54:29.744038 [render] [139971093668136] Contents size: PyQt5.QtCore.QSize(1024, 105100)
2016-07-14 14:54:29.744310 [render] [139971093668136] viewport size is set to 1024x105100
2016-07-14 14:54:29.744631 [render] [139971093668136] image render: output size=PyQt5.QtCore.QSize(1024, 105100), viewport=PyQt5.QtCore.QRect(0, 0, 1024, 105100)
2016-07-14 14:54:29.744776 [render] [139971093668136] image render (raster): rendering PyQt5.QtCore.QRect(0, 0, 1024, 105100) of the web page
2016-07-14 14:54:29.744915 [render] [139971093668136] image render (raster): rendering into PyQt5.QtCore.QRect(0, 0, 1024, 105100) of the canvas
2016-07-14 14:54:29.745054 [render] [139971093668136] image render (raster): canvas size=PyQt5.QtCore.QSize(1024, 105100)
2016-07-14 14:54:29.745229 [render] [139971093668136] image render: rendering PyQt5.QtCore.QRect(0, 0, 1024, 105100) of the web page
2016-07-14 14:54:29.745365 [render] [139971093668136] image render: rendering into PyQt5.QtCore.QRect(0, 0, 1024, 105100) of the canvas
2016-07-14 14:54:29.745504 [render] [139971093668136] image render: canvas size=PyQt5.QtCore.QSize(1024, 105100)
2016-07-14 14:54:29.745657 [render] [139971093668136] image render: draw region too large, rendering tile-by-tile
2016-07-14 14:54:30.002257 [render] [139971093668136] Rendering with viewport=PyQt5.QtCore.QRect(0, 0, 1024, 105100)
2016-07-14 14:54:30.043210 [render] [139971093668136] [lua_runner] instructions used: 257
2016-07-14 14:54:30.043453 [render] [139971093668136] [lua_runner] caught LuaError LuaError('[string "..."]:20: SystemError(\\'Bad call flags in PyCFunction_Call. METH_OLDARGS is no longer supported!\\',)',)
2016-07-14 14:54:30.043625 [render] [139971093668136] [lua] LuaError is caused by SystemError('Bad call flags in PyCFunction_Call. METH_OLDARGS is no longer supported!',)
2016-07-14 14:54:30.043758 [render] [139971093668136] [lua] returning Lua error as-is
2016-07-14 14:54:30.044061 [pool] [139971093668136] SLOT 0 finished with an error <splash.qtrender_lua.LuaRender object at 0x7f4d8f51c438>: [Failure instance: Traceback: <class 'splash.exceptions.ScriptError'>: {'source': '[string "..."]', 'type': 'LUA_ERROR', 'error': "SystemError('Bad call flags in PyCFunction_Call. METH_OLDARGS is no longer supported!',)", 'line_number': 20, 'message': 'Lua error: [string "..."]:20: SystemError(\'Bad call flags in PyCFunction_Call. METH_OLDARGS is no longer supported!\',)'}
    /home/ubuntu/github/splash/splash/browser_tab.py:497:_on_wait_timeout
    /home/ubuntu/github/splash/splash/qtrender_lua.py:476:success
    /home/ubuntu/github/splash/splash/lua_runner.py:29:return_result
    /home/ubuntu/github/splash/splash/qtrender.py:19:stop_on_error_wrapper
    --- <exception caught here> ---
    /home/ubuntu/github/splash/splash/qtrender.py:17:stop_on_error_wrapper
    /home/ubuntu/github/splash/splash/qtrender_lua.py:1525:dispatch
    /home/ubuntu/github/splash/splash/lua_runner.py:190:dispatch
    ]
2016-07-14 14:54:30.044281 [pool] [139971093668136] SLOT 0 is closing <splash.qtrender_lua.LuaRender object at 0x7f4d8f51c438>
2016-07-14 14:54:30.044433 [render] [139971093668136] [splash] clearing 0 objects
2016-07-14 14:54:30.044604 [render] [139971093668136] close is requested by a script
2016-07-14 14:54:30.044984 [render] [139971093668136] cancelling 0 remaining timers
2016-07-14 14:54:30.045128 [pool] [139971093668136] SLOT 0 done with <splash.qtrender_lua.LuaRender object at 0x7f4d8f51c438>
2016-07-14 14:54:30.046117 [events] {"user-agent": "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:16.0) Gecko/16.0 Firefox/16.0", "rendertime": 2.873365640640259, "path": "/execute", "timestamp": 1468475670, "qsize": 0, "load": [0.08, 0.18, 0.2], "fds": 21, "error": {"type": "ScriptError", "error": 400, "info": {"source": "[string \"...\"]", "type": "LUA_ERROR", "error": "SystemError('Bad call flags in PyCFunction_Call. METH_OLDARGS is no longer supported!',)", "line_number": 20, "message": "Lua error: [string \"...\"]:20: SystemError('Bad call flags in PyCFunction_Call. METH_OLDARGS is no longer supported!',)"}, "description": "Error happened while executing Lua script"}, "maxrss": 525932, "client_ip": "127.0.0.1", "_id": 139971093668136, "active": 0, "method": "POST", "args": {"cookies": [], "proxy": "socks5://127.0.0.1:9050", "url": "http://skunksworkedp2cg.onion/sites.html", "headers": {"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.3 Safari/534.53.10"}, "lua_source": "\n    function main(splash)\n      splash:init_cookies(splash.args.cookies)\n      assert(splash:go{\n          splash.args.url,\n          headers=splash.args.headers,\n          http_method=splash.args.http_method,\n          body=splash.args.body,\n        })\n      assert(splash:wait{time=1, cancel_on_redirect=false})\n\n      local entries = splash:history()\n      local last_response = entries[#entries].response\n      return {\n        url = splash:url(),\n        headers = last_response.headers,\n        http_status = last_response.status,\n        cookies = splash:get_cookies(),\n        html = splash:html(),\n        png = splash:png{render_all=true},\n      }\n    end\n    ", "uid": 139971093668136}, "status_code": 400}
2016-07-14 14:54:30.046336 [-] "127.0.0.1" - - [14/Jul/2016:05:54:29 +0000] "POST /execute HTTP/1.0" 400 413 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:16.0) Gecko/16.0 Firefox/16.0"
2016-07-14 14:54:30.046535 [pool] SLOT 0 is available
2016-07-14 14:54:30.046699 [QAbstractEventDispatcher] awake; block time: 0.5379
pawelmhm commented 8 years ago

can you add some steps to reproduce this exception? How are you making request to Splash? Is it happening when crawling some specific website, or happens for all requests?

animalmatsuzawa commented 8 years ago

Thanks for your quick response @pawelmhm .

It occurs when you request the "http://skunksworkedp2cg.onion.link/sites.html" in "splash:png{render_all=true}".

http://XXX.XXX.XXX.XXX:8050/info?wait=0.5&images=1&expand=1&timeout=60.0&url=http%3A%2F%2Fskunksworkedp2cg.onion.link%2Fsites.html&lua_source=function+main%28splash%29%0D%0A++local+url+%3D+splash.args.url%0D%0A++assert%28splash%3Ago%28url%29%29%0D%0A++assert%28splash%3Await%280.5%29%29%0D%0A++return+%7B%0D%0A++++html+%3D+splash%3Ahtml%28%29%2C%0D%0A++++png+%3D+splash%3Apng%7Brender_all%3Dtrue%7D%2C%0D%0A++++har+%3D+splash%3Ahar%28%29%2C%0D%0A++%7D%0D%0Aend

lua script

function main(splash)
  local url = splash.args.url
  assert(splash:go(url))
  assert(splash:wait{time=1, cancel_on_redirect=false})
  return {
    html = splash:html(),
    png = splash:png{render_all=true},
    har = splash:har(),
  }
end
Gallaecio commented 4 years ago

Did you manage to solve your issue?