search

jae-jae / QueryList

:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
https://querylist.cc
2.65k stars 441 forks source link

使用 PhantomJS 插件时在windows 下可以正常获取到信息，但是到了centos 后就获取不到信息 #52

Closed xbchome closed 5 years ago

xbchome commented 5 years ago

centos 下只返回：string(13) "" 一样的代码在Windows下就可以正常获取到数据

jae-jae commented 5 years ago

有个debug的参数，开启一下，然后看一下输出信息

xbchome commented 5 years ago

Network - Resource request error: QNetworkReply::NetworkError(OperationCanceledError) ( "Operation canceled" ) URL: "https://m.tuniu.com/flight/domestic/list/CAN_WUH_ST_1_0_0?deptDate=2018-10-27&isGo=0" 看到这个错误不知道怎么形成的

xbchome commented 5 years ago

2018-10-29T14:50:59 [DEBUG] Set "http" proxy to: "" : 1080 2018-10-29T14:50:59 [DEBUG] Phantom - execute: Configuration 2018-10-29T14:50:59 [DEBUG] 0 objectName : "" 2018-10-29T14:50:59 [DEBUG] 1 cookiesFile : "" 2018-10-29T14:50:59 [DEBUG] 2 diskCacheEnabled : "false" 2018-10-29T14:50:59 [DEBUG] 3 maxDiskCacheSize : "-1" 2018-10-29T14:50:59 [DEBUG] 4 diskCachePath : "" 2018-10-29T14:50:59 [DEBUG] 5 ignoreSslErrors : "true" 2018-10-29T14:50:59 [DEBUG] 6 localUrlAccessEnabled : "true" 2018-10-29T14:50:59 [DEBUG] 7 localToRemoteUrlAccessEnabled : "false" 2018-10-29T14:50:59 [DEBUG] 8 outputEncoding : "UTF-8" 2018-10-29T14:50:59 [DEBUG] 9 proxyType : "http" 2018-10-29T14:50:59 [DEBUG] 10 proxy : ":1080" 2018-10-29T14:50:59 [DEBUG] 11 proxyAuth : ":" 2018-10-29T14:50:59 [DEBUG] 12 scriptEncoding : "UTF-8" 2018-10-29T14:50:59 [DEBUG] 13 webSecurityEnabled : "true" 2018-10-29T14:50:59 [DEBUG] 14 offlineStoragePath : "" 2018-10-29T14:50:59 [DEBUG] 15 localStoragePath : "" 2018-10-29T14:50:59 [DEBUG] 16 localStorageDefaultQuota : "-1" 2018-10-29T14:50:59 [DEBUG] 17 offlineStorageDefaultQuota : "-1" 2018-10-29T14:50:59 [DEBUG] 18 printDebugMessages : "true" 2018-10-29T14:50:59 [DEBUG] 19 javascriptCanOpenWindows : "true" 2018-10-29T14:50:59 [DEBUG] 20 javascriptCanCloseWindows : "true" 2018-10-29T14:50:59 [DEBUG] 21 sslProtocol : "default" 2018-10-29T14:50:59 [DEBUG] 22 sslCiphers : "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-RC4-SHA:ECDHE-RSA-RC4-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:RC4-SHA:RC4-MD5" 2018-10-29T14:50:59 [DEBUG] 23 sslCertificatesPath : "" 2018-10-29T14:50:59 [DEBUG] 24 sslClientCertificateFile : "" 2018-10-29T14:50:59 [DEBUG] 25 sslClientKeyFile : "" 2018-10-29T14:50:59 [DEBUG] 26 sslClientKeyPassphrase : "" 2018-10-29T14:50:59 [DEBUG] 27 webdriver : ":" 2018-10-29T14:50:59 [DEBUG] 28 webdriverLogFile : "" 2018-10-29T14:50:59 [DEBUG] 29 webdriverLogLevel : "INFO" 2018-10-29T14:50:59 [DEBUG] 30 webdriverSeleniumGridHub : "" 2018-10-29T14:50:59 [DEBUG] Phantom - execute: Script & Arguments 2018-10-29T14:50:59 [DEBUG] script: "/tmp/6502b9c26b2c19818a9c" 2018-10-29T14:50:59 [DEBUG] Phantom - execute: Starting normal mode 2018-10-29T14:50:59 [DEBUG] WebPage - setupFrame "" 2018-10-29T14:50:59 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r"))) 2018-10-29T14:50:59 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r"))) 2018-10-29T14:50:59 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r"))) 2018-10-29T14:50:59 [DEBUG] WebPage - updateLoadingProgress: 10 2018-10-29T14:50:59 [DEBUG] CookieJar - Saved "locale=en; domain=flight.tuniu.com; path=/" 2018-10-29T14:50:59 [DEBUG] CookieJar - Saved "connect.sid=s%3Ak-90KEltVHlNuPONge8Xfj4fiGhDBC6m.v4fyrjd%2B3QdD0DhWhxZrb3ifuyyvuSEuhlhxwd6jJPI; HttpOnly; domain=flight.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "locale=en; domain=flight.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "connect.sid=s%3Ak-90KEltVHlNuPONge8Xfj4fiGhDBC6m.v4fyrjd%2B3QdD0DhWhxZrb3ifuyyvuSEuhlhxwd6jJPI; HttpOnly; domain=flight.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "connect.sid=s%3A7CfepSs2bhwUk0iyvtnp1KRZwEYFsENx.Kpv%2Be8i%2BeRFZ2ubWeWcTirEF2lbLt6sTGe52ytDrSQs; HttpOnly; domain=m.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] Network - Resource request error: QNetworkReply::NetworkError(RemoteHostClosedError) ( "Connection closed" ) URL: "https://m.tuniu.com/flight/domestic/list/CAN_WUH_ST_1_0_0?deptDate=2018-10-27&isGo=0" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "connect.sid=s%3Ak-90KEltVHlNuPONge8Xfj4fiGhDBC6m.v4fyrjd%2B3QdD0DhWhxZrb3ifuyyvuSEuhlhxwd6jJPI; HttpOnly; domain=flight.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "connect.sid=s%3A7CfepSs2bhwUk0iyvtnp1KRZwEYFsENx.Kpv%2Be8i%2BeRFZ2ubWeWcTirEF2lbLt6sTGe52ytDrSQs; HttpOnly; domain=m.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "locale=en; domain=flight.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] Network - Resource request error: QNetworkReply::NetworkError(RemoteHostClosedError) ( "Connection closed" ) URL: "https://m.tuniu.com/flight/domestic/list/CAN_WUH_ST_1_0_0?deptDate=2018-10-27&isGo=0" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "connect.sid=s%3Ak-90KEltVHlNuPONge8Xfj4fiGhDBC6m.v4fyrjd%2B3QdD0DhWhxZrb3ifuyyvuSEuhlhxwd6jJPI; HttpOnly; domain=flight.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "connect.sid=s%3A7CfepSs2bhwUk0iyvtnp1KRZwEYFsENx.Kpv%2Be8i%2BeRFZ2ubWeWcTirEF2lbLt6sTGe52ytDrSQs; HttpOnly; domain=m.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "locale=en; domain=flight.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] Network - Resource request error: QNetworkReply::NetworkError(RemoteHostClosedError) ( "Connection closed" ) URL: "https://m.tuniu.com/flight/domestic/list/CAN_WUH_ST_1_0_0?deptDate=2018-10-27&isGo=0" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "connect.sid=s%3Ak-90KEltVHlNuPONge8Xfj4fiGhDBC6m.v4fyrjd%2B3QdD0DhWhxZrb3ifuyyvuSEuhlhxwd6jJPI; HttpOnly; domain=flight.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "connect.sid=s%3A7CfepSs2bhwUk0iyvtnp1KRZwEYFsENx.Kpv%2Be8i%2BeRFZ2ubWeWcTirEF2lbLt6sTGe52ytDrSQs; HttpOnly; domain=m.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "locale=en; domain=flight.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] Network - Resource request error: QNetworkReply::NetworkError(RemoteHostClosedError) ( "Connection closed" ) URL: "https://m.tuniu.com/flight/domestic/list/CAN_WUH_ST_1_0_0?deptDate=2018-10-27&isGo=0" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "connect.sid=s%3Ak-90KEltVHlNuPONge8Xfj4fiGhDBC6m.v4fyrjd%2B3QdD0DhWhxZrb3ifuyyvuSEuhlhxwd6jJPI; HttpOnly; domain=flight.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "connect.sid=s%3A7CfepSs2bhwUk0iyvtnp1KRZwEYFsENx.Kpv%2Be8i%2BeRFZ2ubWeWcTirEF2lbLt6sTGe52ytDrSQs; HttpOnly; domain=m.tuniu.com; path=/" 2018-10-29T14:51:00 [DEBUG] CookieJar - Saved "locale=en; domain=flight.tuniu.com; path=/" 2018-10-29T14:51:01 [DEBUG] WebPage - updateLoadingProgress: 100 2018-10-29T14:51:01 [DEBUG] Network - Resource request error: QNetworkReply::NetworkError(OperationCanceledError) ( "Operation canceled" ) URL: "https://m.tuniu.com/flight/domestic/list/CAN_WUH_ST_1_0_0?deptDate=2018-10-27&isGo=0" 2018-10-29T14:51:01 [WARNING] QNetworkReplyImplPrivate::error: Internal problem, this method must only be called once. 2018-10-29T14:51:01 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function () {\n \n var styles = {};\n \n for(var property in styles) {\n document.body.style[property] = styles[property];\n }\n})(); })()" 2018-10-29T14:51:01 [DEBUG] WebPage - setupFrame "" 2018-10-29T14:51:01 [DEBUG] WebPage - evaluateJavaScript result QVariant(Invalid) \n2018-10-29T14:51:01 [DEBUG] WebPage - setupFrame "" 2018-10-29T14:51:01 [DEBUG] WebPage - updateLoadingProgress: 10 2018-10-29T14:51:01 [DEBUG] WebPage - setupFrame "" 2018-10-29T14:51:01 [DEBUG] WebPage - updateLoadingProgress: 100 2018-10-29T14:51:01 [DEBUG] WebPage - setupFrame "" 2018-10-29T14:51:01 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r"))) 2018-10-29T14:51:01 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r"))) 2018-10-29T14:51:01 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r"))) 2018-10-29T14:51:01 [DEBUG] WebPage - updateLoadingProgress: 10 2018-10-29T14:51:01 [DEBUG] WebPage - setupFrame "" 2018-10-29T14:51:01 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r"))) 2018-10-29T14:51:01 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r"))) 2018-10-29T14:51:01 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r"))) 2018-10-29T14:51:01 [DEBUG] WebPage - updateLoadingProgress: 100 2018-10-29T14:51:01 [DEBUG] WebPage - setupFrame "" 2018-10-29T14:51:01 [DEBUG] CookieJar - Purged (session) "locale=en; domain=flight.tuniu.com; path=/" 2018-10-29T14:51:01 [DEBUG] CookieJar - Purged (session) "connect.sid=s%3A7CfepSs2bhwUk0iyvtnp1KRZwEYFsENx.Kpv%2Be8i%2BeRFZ2ubWeWcTirEF2lbLt6sTGe52ytDrSQs; HttpOnly; domain=m.tuniu.com; path=/" 2018-10-29T14:51:01 [DEBUG] CookieJar - Purged (session) "connect.sid=s%3Ak-90KEltVHlNuPONge8Xfj4fiGhDBC6m.v4fyrjd%2B3QdD0DhWhxZrb3ifuyyvuSEuhlhxwd6jJPI; HttpOnly; domain=flight.tuniu.com; path=/"

jae-jae commented 5 years ago

看报错信息猜测可能是你服务器上无法访问目标网址

xbchome commented 5 years ago

直接用curl 或者get 方法可以正常访问就是获取不到异步数据而已

xbchome commented 5 years ago

发现在linux 系统直接用火狐或者谷歌浏览器都打不开途牛的机票页面