sitespeedio / sitespeed.io

sitespeed.io is an open-source tool for comprehensive web performance analysis, enabling you to test, monitor, and optimize your website’s speed using real browsers in various environments.
https://www.sitespeed.io/
MIT License
4.75k stars 603 forks source link

Can't get Navigation Timing #502

Closed pydubreucq closed 9 years ago

pydubreucq commented 10 years ago

Hi, I'm trying to test sitespeed.io beta 3 and I can't collect Navigation Timing Metrics. I'm testing it on Ubuntu Server 14.04.1 LTS

sitespeed.io -u http://www.monsiteestlent.com -b firefox
info: OS: linux 3.13.0-39-generic sitespeed:3.0.0-beta  phantomJs:1.9.8 java:java version "1.7.0_65"
info: Will crawl from start point http://www.monsiteestlent.com with crawl depth 1
info: Will analyze 18 pages
info: Running YSlow for http://www.monsiteestlent.com
info: Running YSlow for http://www.monsiteestlent.com/2014/02/
info: Running YSlow for http://www.monsiteestlent.com/mentions-legales/
info: Running YSlow for http://www.monsiteestlent.com/dossiers/les-pires-pratiques-performance-web
info: Running YSlow for http://www.monsiteestlent.com/2014/04/
info: Running YSlow for http://www.monsiteestlent.com/dossiers/haproxy-nouvelle-version-majeure-1-5-0
info: Running YSlow for http://www.monsiteestlent.com/actualites/
info: Running YSlow for http://www.monsiteestlent.com/2014/06/
info: Running YSlow for http://www.monsiteestlent.com/a-propos/
info: Running YSlow for http://www.monsiteestlent.com/2014/10/
info: Running YSlow for http://www.monsiteestlent.com/cms-framework/
info: Running YSlow for http://www.monsiteestlent.com/blog/
info: Running YSlow for http://www.monsiteestlent.com/2014/08/
info: Running YSlow for http://www.monsiteestlent.com/actualites/sortie-rc2-gatling-2-0-0
info: Running YSlow for http://www.monsiteestlent.com/2014/03/
info: Running YSlow for http://www.monsiteestlent.com/cms-framework/magento/quanta-computing-suivi-performances-profiling-magento
info: Running YSlow for http://www.monsiteestlent.com/ecommerce/
info: Running YSlow for http://www.monsiteestlent.com/dossiers/
info: Queueing browsertime for http://www.monsiteestlent.com firefox
info: Queueing browsertime for http://www.monsiteestlent.com/2014/02/ firefox
info: Queueing browsertime for http://www.monsiteestlent.com/mentions-legales/ firefox
info: Queueing browsertime for http://www.monsiteestlent.com/dossiers/les-pires-pratiques-performance-web firefox
info: Queueing browsertime for http://www.monsiteestlent.com/2014/04/ firefox
info: Queueing browsertime for http://www.monsiteestlent.com/dossiers/haproxy-nouvelle-version-majeure-1-5-0 firefox
info: Queueing browsertime for http://www.monsiteestlent.com/actualites/ firefox
info: Queueing browsertime for http://www.monsiteestlent.com/2014/06/ firefox
info: Queueing browsertime for http://www.monsiteestlent.com/a-propos/ firefox
info: Queueing browsertime for http://www.monsiteestlent.com/2014/10/ firefox
info: Queueing browsertime for http://www.monsiteestlent.com/cms-framework/ firefox
info: Queueing browsertime for http://www.monsiteestlent.com/blog/ firefox
info: Queueing browsertime for http://www.monsiteestlent.com/2014/08/ firefox
info: Queueing browsertime for http://www.monsiteestlent.com/actualites/sortie-rc2-gatling-2-0-0 firefox
info: Queueing browsertime for http://www.monsiteestlent.com/2014/03/ firefox
info: Queueing browsertime for http://www.monsiteestlent.com/cms-framework/magento/quanta-computing-suivi-performances-profiling-magento firefox
info: Queueing browsertime for http://www.monsiteestlent.com/ecommerce/ firefox
info: Queueing browsertime for http://www.monsiteestlent.com/dossiers/ firefox
info: Running browsertime for firefox http://www.monsiteestlent.com

Sitespeed blocked when that run browsertime...

In browsertime.log file, I have: 2014-10-30T12:45:20.543Z - info: Start the proxy, will wait 3000 ms In browsermobproxy.log file, I have: {"level":"info","message":"Starting browsermobproxy ...","timestamp":"2014-10-30T12:45:20.544Z"} {"level":"info","message":"stderr:INFO 10/30 12:45:22 n.s.u.i.d.DataBuild~ - No pattern available for 'Other'.\n","timestamp":"2014-10-30T12:45:22.598Z"} {"level":"info","message":"stderr:INFO 10/30 12:45:22 n.s.u.i.d.DataBuild~ - No pattern available for 'Personal computer'.\n","timestamp":"2014-10-30T12:45:22.599Z"} {"level":"info","message":"stderr:INFO 10/30 12:45:23 n.s.u.i.d.DataBuild~ - No pattern available for 'Personal computer'.\n","timestamp":"2014-10-30T12:45:23.532Z"} {"level":"error","message":"Error: connect ECONNREFUSED\n at errnoException (net.js:901:11)\n at Object.afterConnect as oncomplete","timestamp":"2014-10-30T12:45:23.568Z"} {"level":"error","message":"Error: socket hang up\n at createHangUpError (http.js:1472:15)\n at Socket.socketCloseListener (http.js:1522:23)\n at Socket.EventEmitter.emit (events.js:117:20)\n at TCP.close (net.js:466:12)","timestamp":"2014-10-30T12:45:23.571Z"} {"level":"error","message":"Error: connect ECONNREFUSED\n at errnoException (net.js:901:11)\n at Object.afterConnect as oncomplete","timestamp":"2014-10-30T12:45:23.572Z"} {"level":"error","message":"Error: connect ECONNREFUSED\n at errnoException (net.js:901:11)\n at Object.afterConnect as oncomplete","timestamp":"2014-10-30T12:45:23.573Z"} {"level":"error","message":"Error: socket hang up\n at createHangUpError (http.js:1472:15)\n at Socket.socketCloseListener (http.js:1522:23)\n at Socket.EventEmitter.emit (events.js:117:20)\n at TCP.close (net.js:466:12)","timestamp":"2014-10-30T12:45:23.574Z"} {"level":"error","message":"Error: socket hang up\n at createHangUpError (http.js:1472:15)\n at Socket.socketCloseListener (http.js:1522:23)\n at Socket.EventEmitter.emit (events.js:117:20)\n at TCP.close (net.js:466:12)","timestamp":"2014-10-30T12:45:23.574Z"} {"level":"error","message":"Error: connect ECONNREFUSED\n at errnoException (net.js:901:11)\n at Object.afterConnect as oncomplete","timestamp":"2014-10-30T12:45:23.576Z"} {"level":"error","message":"Error: connect ECONNREFUSED\n at errnoException (net.js:901:11)\n at Object.afterConnect as oncomplete","timestamp":"2014-10-30T12:45:23.576Z"} {"level":"error","message":"Error: connect ECONNREFUSED\n at errnoException (net.js:901:11)\n at Object.afterConnect as oncomplete","timestamp":"2014-10-30T12:45:23.576Z"} {"level":"error","message":"Error: connect ECONNREFUSED\n at errnoException (net.js:901:11)\n at Object.afterConnect as oncomplete","timestamp":"2014-10-30T12:45:23.577Z"} {"level":"error","message":"Error: socket hang up\n at createHangUpError (http.js:1472:15)\n at Socket.socketCloseListener (http.js:1522:23)\n at Socket.EventEmitter.emit (events.js:117:20)\n at TCP.close (net.js:466:12)","timestamp":"2014-10-30T12:45:23.578Z"}

I've testing with another website blog.admin-linux.org and it's the same. Do you have an idea ?

Best Regards Pierre-Yves

soulgalore commented 10 years ago

Hi Pierre-Yves, thanks for the report and really good that you included all the logs, lovely! It seems like the proxy that starts couldn't be started (no rights to open the port or the port occupied already). We've made the ports configurable but it is not yet propagated to the sitespeed.io. I will test it asap on Ubuntu and see if I get the same thing and will try to do a fix during the weekend (busy today and tomorrow). Will keep the thread up to date asap when I do a fix.

Best Peter

P.S thanks again for the good bug report

pydubreucq commented 10 years ago

Hi, Thanks for your reply. What is the default port ? Thanks by advance Pierre-Yves

soulgalore commented 10 years ago

Hi Pierre-Yves, by default the ports are (theres one for the proxy that is an REST API where you configure/handle other proxy and one for the actual proxy): 10800 9093

Would be lovely if you could check if something running there or if it is the user that can't open on that port.

Best Peter

pydubreucq commented 10 years ago

Hi, I have nothing on their port. netstat -atn Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 36 10.1.31.217:22 192.168.100.205:54346 ESTABLISHED tcp 0 0 10.1.31.217:22 192.168.100.205:54341 ESTABLISHED tcp6 0 0 :::80 :::* LISTEN
tcp6 0 0 :::22 :::* LISTEN

When browsertime is running, I can see 9093's port

netstat -atn | grep 9093 tcp 0 0 127.0.0.1:9093 127.0.0.1:48225 TIME_WAIT
tcp 0 0 10.1.31.217:59093 173.194.45.239:80 TIME_WAIT
tcp 0 0 127.0.0.1:9093 127.0.0.1:48224 TIME_WAIT
tcp 0 0 127.0.0.1:9093 127.0.0.1:48222 TIME_WAIT
tcp 0 0 127.0.0.1:9093 127.0.0.1:48221 TIME_WAIT

My firewall could block the dest port. I'm also surprised that it does not use the HTTP port

soulgalore commented 10 years ago

had some problems with my ubuntu box but now is everything up and running so will hopefully fix it before tomrrow night, when I plan to do a new beta release.

soulgalore commented 10 years ago

Hi @pydubreucq , so now I have it up and running on Ubuntu, please try it with the new beta-2 version: npm install sitespeed.io@3.0.0-beta-2 -g

Best Peter

pydubreucq commented 10 years ago

Hi @soulgalore Thanks for this update. I've the same problem. I've updated sitespeed.io

root@webperf1:~# sitespeed.io -V 3.0.0-beta-2

When I lanch sitespeed.io, I'm blocked on browsertime with this error log :

{"level":"error","message":"Error: socket hang up\n    at createHangUpError (http.js:1472:15)\n    at Socket.socketCloseListener (http.js:1522:23)\n    at Socket.EventEmitter.emit (events.js:117:20)\n    at TCP.close (net.js:466:12)","timestamp":"2014-11-03T22:34:38.570Z"}
{"level":"error","message":"Error: socket hang up\n    at createHangUpError (http.js:1472:15)\n    at Socket.socketCloseListener (http.js:1522:23)\n    at Socket.EventEmitter.emit (events.js:117:20)\n    at TCP.close (net.js:466:12)","timestamp":"2014-11-03T22:34:38.571Z"}
{"level":"error","message":"Error: socket hang up\n    at createHangUpError (http.js:1472:15)\n    at Socket.socketCloseListener (http.js:1522:23)\n    at Socket.EventEmitter.emit (events.js:117:20)\n    at TCP.close (net.js:466:12)","timestamp":"2014-11-03T22:34:38.571Z"}
{"level":"error","message":"Error: socket hang up\n    at createHangUpError (http.js:1472:15)\n    at Socket.socketCloseListener (http.js:1522:23)\n    at Socket.EventEmitter.emit (events.js:117:20)\n    at TCP.close (net.js:466:12)","timestamp":"2014-11-03T22:34:38.572Z"}
{"level":"error","message":"Error: socket hang up\n    at createHangUpError (http.js:1472:15)\n    at Socket.socketCloseListener (http.js:1522:23)\n    at Socket.EventEmitter.emit (events.js:117:20)\n    at TCP.close (net.js:466:12)","timestamp":"2014-11-03T22:34:38.572Z"}
{"level":"error","message":"Error: connect ECONNREFUSED\n    at errnoException (net.js:901:11)\n    at Object.afterConnect [as oncomplete] (net.js:892:19)","timestamp":"2014-11-03T22:34:38.573Z"}
{"level":"error","message":"Error: connect ECONNREFUSED\n    at errnoException (net.js:901:11)\n    at Object.afterConnect [as oncomplete] (net.js:892:19)","timestamp":"2014-11-03T22:34:38.574Z"}
{"level":"error","message":"Error: connect ECONNREFUSED\n    at errnoException (net.js:901:11)\n    at Object.afterConnect [as oncomplete] (net.js:892:19)","timestamp":"2014-11-03T22:34:38.574Z"}
{"level":"error","message":"Error: connect ECONNREFUSED\n    at errnoException (net.js:901:11)\n    at Object.afterConnect [as oncomplete] (net.js:892:19)","timestamp":"2014-11-03T22:34:38.574Z"}
{"level":"error","message":"Error: connect ECONNREFUSED\n    at errnoException (net.js:901:11)\n    at Object.afterConnect [as oncomplete] (net.js:892:19)","timestamp":"2014-11-03T22:34:38.575Z"}

Is I print list of open files by this process, I've that :

root@webperf1:~# lsof -p 19217
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
node    19217 root  cwd    DIR  253,1     4096 50332164 /root
node    19217 root  rtd    DIR  253,1     4096      128 /
node    19217 root  txt    REG  253,1  1462424  1307582 /usr/bin/nodejs
node    19217 root  mem    REG  253,1    47712 50332374 /lib/x86_64-linux-gnu/libnss_files-2.19.so
node    19217 root  mem    REG  253,1    90080 50331992 /lib/x86_64-linux-gnu/libgcc_s.so.1
node    19217 root  mem    REG  253,1  1071552 50332314 /lib/x86_64-linux-gnu/libm-2.19.so
node    19217 root  mem    REG  253,1  1845024 50332299 /lib/x86_64-linux-gnu/libc-2.19.so
node    19217 root  mem    REG  253,1   141574 50332380 /lib/x86_64-linux-gnu/libpthread-2.19.so
node    19217 root  mem    REG  253,1   979056 16778308 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
node    19217 root  mem    REG  253,1    31792 50332286 /lib/x86_64-linux-gnu/librt-2.19.so
node    19217 root  mem    REG  253,1    14664 50332379 /lib/x86_64-linux-gnu/libdl-2.19.so
node    19217 root  mem    REG  253,1  1926432 50332097 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
node    19217 root  mem    REG  253,1   383112 50332068 /lib/x86_64-linux-gnu/libssl.so.1.0.0
node    19217 root  mem    REG  253,1  4219152 50347751 /usr/lib/libv8.so.3.14.5
node    19217 root  mem    REG  253,1    68168 18579668 /usr/lib/x86_64-linux-gnu/libcares.so.2.1.0
node    19217 root  mem    REG  253,1   100728 50332161 /lib/x86_64-linux-gnu/libz.so.1.2.8
node    19217 root  mem    REG  253,1   149120 50332301 /lib/x86_64-linux-gnu/ld-2.19.so
node    19217 root    0u   CHR  136,0      0t0        3 /dev/pts/0
node    19217 root    1u   CHR  136,0      0t0        3 /dev/pts/0
node    19217 root    2u   CHR  136,0      0t0        3 /dev/pts/0
node    19217 root    3r  FIFO    0,8      0t0  1240499 pipe
node    19217 root    4w  FIFO    0,8      0t0  1240499 pipe
node    19217 root    5u  0000    0,9        0     6405 anon_inode
node    19217 root    6r  FIFO    0,8      0t0  1240500 pipe
node    19217 root    7w  FIFO    0,8      0t0  1240500 pipe
node    19217 root    8u  0000    0,9        0     6405 anon_inode
node    19217 root    9r   DIR  253,1     4096      128 /
node    19217 root   12w   REG  253,1      143 51584605 /root/browsertime.log
node    19217 root   13u  sock    0,7      0t0  1379135 can't identify protocol
node    19217 root   14w   REG  253,1  9053260 51584606 /root/browsermobproxy.log
node    19217 root   15u  sock    0,7      0t0  1379136 can't identify protocol
node    19217 root   16u  sock    0,7      0t0  1379137 can't identify protocol

I've found this post about this error and lsof : https://idea.popcount.org/2012-12-09-lsof-cant-identify-protocol/ I don't know if it could help you.

Best Regards, Pierre-Yves

soulgalore commented 10 years ago

Hi @pydubreucq ok thanks will look into it again. Best Peter

soulgalore commented 10 years ago

Hi again @pydubreucq , if you could try the latest beta-3 and then add this parameter to your run:

--btConfig {\"proxySleepBeforeStart\":\"8000\"}

It will make browsertime/sitespeed.io wait 8 seconds before to start access the URL through the proxy, giving the proxy time to start (it is slow on some systems).

Best Peter

pydubreucq commented 10 years ago

Hi, We advance ;) When I launch sitespeed.io, browsertime didn't had connection problem, but it didn't work.

This is browsermobproxy.log with firefox :

{"level":"info","message":"Starting browsermobproxy ...","timestamp":"2014-11-05T11:35:41.101Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:43 n.s.u.i.d.DataBuild~ - No pattern available for 'Other'.\n","timestamp":"2014-11-05T11:35:43.068Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:43 n.s.u.i.d.DataBuild~ - No pattern available for 'Personal computer'.\n","timestamp":"2014-11-05T11:35:43.068Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:44 n.s.u.i.d.DataBuild~ - No pattern available for 'Personal computer'.\n","timestamp":"2014-11-05T11:35:44.020Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:44 n.l.b.p.Main         - Starting BrowserMob Proxy version %s\n","timestamp":"2014-11-05T11:35:44.922Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:44 o.e.j.u.log          - jetty-0.0.1-SNAPSHOT\n","timestamp":"2014-11-05T11:35:44.987Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:45 o.e.j.u.log          - started o.e.j.s.ServletContextHandler{/,null}\n","timestamp":"2014-11-05T11:35:45.087Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:46 o.e.j.u.log          - Started SelectChannelConnector@0.0.0.0:9092\n","timestamp":"2014-11-05T11:35:46.586Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:49 n.l.b.p.j.h.HttpSer~ - Version Jetty/0.0.1-SNAPSHOT\n","timestamp":"2014-11-05T11:35:49.484Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:49 n.l.b.p.j.u.Contain~ - Started HttpContext[/,/]\n","timestamp":"2014-11-05T11:35:49.485Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:49 n.l.b.p.j.h.SocketL~ - Started SocketListener on 0.0.0.0:9091\n","timestamp":"2014-11-05T11:35:49.491Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:49 n.l.b.p.j.u.Contain~ - Started net.lightbody.bmp.proxy.jetty.jetty.Server@3fdaf238\n","timestamp":"2014-11-05T11:35:49.491Z"}
{"level":"error","message":"Error: Failed to install profile; firefox terminated with Result(code=1, signal=null)\n    at Error (<anonymous>)\n    at /usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/firefox/binary.js:227:15\n    at /usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/lib/goog/base.js:1582:15\n    at webdriver.promise.ControlFlow.runInNewFrame_ (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/lib/webdriver/promise.js:1640:20)\n    at notify (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/lib/webdriver/promise.js:444:12)\n    at notifyAll (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/lib/webdriver/promise.js:422:7)\n    at resolve (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/lib/webdriver/promise.js:400:7)\n    at fulfill (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/lib/webdriver/promise.js:512:5)\n    at ChildProcess.onExit (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/io/exec.js:134:12)\n    at ChildProcess.g (events.js:180:16)\n==== async task ====\nWebDriver.createSession()\n    at Function.webdriver.WebDriver.acquireSession_ (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:148:22)\n    at Function.webdriver.WebDriver.createSession (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:122:30)\n    at new Driver (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/firefox/index.js:189:36)\n    at Builder.build (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/builder.js:308:14)\n    at Object.exports.getDriver (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/browsers/firefox.js:41:56)\n    at /usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/browsertime.js:93:91\n    at /usr/local/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:254:17\n    at /usr/local/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:157:25\n    at /usr/local/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:251:21","timestamp":"2014-11-05T11:35:51.183Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:51 n.l.b.p.j.u.Threade~ - Stopping Acceptor ServerSocket[addr=/0.0.0.0,localport=9091]\n","timestamp":"2014-11-05T11:35:51.198Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:51 n.l.b.p.j.h.SocketL~ - Stopped SocketListener on 0.0.0.0:9091\n","timestamp":"2014-11-05T11:35:51.296Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:51 n.l.b.p.j.u.Contain~ - Stopped HttpContext[/,/]\n","timestamp":"2014-11-05T11:35:51.464Z"}
{"level":"info","message":"stderr:INFO 11/05 11:35:51 n.l.b.p.j.u.Contain~ - Stopped net.lightbody.bmp.proxy.jetty.jetty.Server@3fdaf238\n","timestamp":"2014-11-05T11:35:51.464Z"}

For firefox, I'm using Ubuntu package :

# dpkg -l | grep firefox
ii  firefox                             33.0+build2-0ubuntu0.14.04.1           amd64        Safe and easy web browser from Mozilla

This is browsermobproxy.log with chrome :

{"level":"info","message":"stderr:INFO 11/05 11:41:17 n.s.u.i.d.DataBuild~ - No pattern available for 'Other'.\n","timestamp":"2014-11-05T11:41:17.771Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:17 n.s.u.i.d.DataBuild~ - No pattern available for 'Personal computer'.\n","timestamp":"2014-11-05T11:41:17.772Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:18 n.s.u.i.d.DataBuild~ - No pattern available for 'Personal computer'.\n","timestamp":"2014-11-05T11:41:18.685Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:19 n.l.b.p.Main         - Starting BrowserMob Proxy version %s\n","timestamp":"2014-11-05T11:41:19.675Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:19 o.e.j.u.log          - jetty-0.0.1-SNAPSHOT\n","timestamp":"2014-11-05T11:41:19.755Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:19 o.e.j.u.log          - started o.e.j.s.ServletContextHandler{/,null}\n","timestamp":"2014-11-05T11:41:19.828Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:21 o.e.j.u.log          - Started SelectChannelConnector@0.0.0.0:9092\n","timestamp":"2014-11-05T11:41:21.353Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:24 n.l.b.p.j.h.HttpSer~ - Version Jetty/0.0.1-SNAPSHOT\n","timestamp":"2014-11-05T11:41:24.249Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:24 n.l.b.p.j.u.Contain~ - Started HttpContext[/,/]\n","timestamp":"2014-11-05T11:41:24.250Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:24 n.l.b.p.j.h.SocketL~ - Started SocketListener on 0.0.0.0:9091\n","timestamp":"2014-11-05T11:41:24.262Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:24 n.l.b.p.j.u.Contain~ - Started net.lightbody.bmp.proxy.jetty.jetty.Server@3f545b48\n","timestamp":"2014-11-05T11:41:24.262Z"}
{"level":"error","message":"Error: The ChromeDriver could not be found on the current PATH. Please download the latest version of the ChromeDriver from http://chromedriver.storage.googleapis.com/index.html and ensure it can be found on your PATH.\n    at Error (<anonymous>)\n    at new ServiceBuilder (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/chrome.js:51:11)\n    at getDefaultService (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/chrome.js:216:22)\n    at new Driver (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/chrome.js:470:32)\n    at Builder.build (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/selenium-webdriver/builder.js:302:14)\n    at Object.exports.getDriver (/usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/browsers/chrome.js:31:56)\n    at /usr/local/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/browsertime.js:93:91\n    at /usr/local/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:254:17\n    at /usr/local/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:157:25\n    at /usr/local/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:251:21","timestamp":"2014-11-05T11:41:24.789Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:24 n.l.b.p.j.u.Threade~ - Stopping Acceptor ServerSocket[addr=/0.0.0.0,localport=9091]\n","timestamp":"2014-11-05T11:41:24.805Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:24 n.l.b.p.j.h.SocketL~ - Stopped SocketListener on 0.0.0.0:9091\n","timestamp":"2014-11-05T11:41:24.905Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:25 n.l.b.p.j.u.Contain~ - Stopped HttpContext[/,/]\n","timestamp":"2014-11-05T11:41:25.101Z"}
{"level":"info","message":"stderr:INFO 11/05 11:41:25 n.l.b.p.j.u.Contain~ - Stopped net.lightbody.bmp.proxy.jetty.jetty.Server@3f545b48\n","timestamp":"2014-11-05T11:41:25.101Z"}

For chrome, I'm using Ubuntu package :

# dpkg -l | grep chrome
ii  chromium-chromedriver               37.0.2062.120-0ubuntu0.14.04.1~pkg1049 amd64        WebDriver driver for the Chromium Browser

Best Regards And Many thanks to try to help me ;) Bye

soulgalore commented 10 years ago

Hi @pydubreucq , thanks for the great bug reports! I will need to test Firefox again and check why the profile isn't working (when I tested on my machine it did, but often don't you hear that, hehe). I will get back on that later this week.

For chrome, thinking about the log "couldn't find the chromedriver". You have it installed right? When I tested, I just followed these instructions on my Ubuntu (but I'm using 14.04 Desktop): http://yizeng.me/2014/04/20/install-chromedriver-and-phantomjs-on-linux-mint/

Best Peter

pydubreucq commented 10 years ago

Hi, I'm just stupid... My Xvfb wasn't started... :( I'm so sorry... It's working with firefox and chromium.

I have another problem, sometime (mostly) sitespeed.io stop their check with browsertime like that :

info: Running browsertime for chrome http://www.monsiteestlent.com/actualites/
info: Start the proxy [9091], will wait 8000 ms
info: Fetch http://www.monsiteestlent.com/actualites/
info: Fetch http://www.monsiteestlent.com/actualites/

Maybe a timeout could be usefull if browsertime stop to work ?

Best Regards

soulgalore commented 10 years ago

Hi @pydubreucq , no problem, that happens to everyone :)

yes good, the error handling today from browsertime is not perfect, I will try to to make it better the coming day,

Best Peter

JeroenVdb commented 9 years ago

I have a similar errors as @pydubreucq. Maybe I'm using Xvfb wrong?

Xvfb is in /usr/bin/Xvfb and I also have xvfb-run available.

When I run Sitespeed.io in Ubuntu (headless) (via Docker):

root@31e987d525fc:/sitespeed.io# /usr/bin/Xvfb & sitespeed.io -u http://www.demorgen.be --noYslow -b firefox -m 1
[1] 164
Initializing built-in extension Generic Event Extension
Initializing built-in extension SHAPE
Initializing built-in extension MIT-SHM
Initializing built-in extension XInputExtension
Initializing built-in extension XTEST
Initializing built-in extension BIG-REQUESTS
Initializing built-in extension SYNC
Initializing built-in extension XKEYBOARD
Initializing built-in extension XC-MISC
Initializing built-in extension SECURITY
Initializing built-in extension XINERAMA
Initializing built-in extension XFIXES
Initializing built-in extension RENDER
Initializing built-in extension RANDR
Initializing built-in extension COMPOSITE
Initializing built-in extension DAMAGE
Initializing built-in extension MIT-SCREEN-SAVER
Initializing built-in extension DOUBLE-BUFFER
Initializing built-in extension RECORD
Initializing built-in extension DPMS
Initializing built-in extension Present
Initializing built-in extension DRI3
Initializing built-in extension X-Resource
Initializing built-in extension XVideo
Initializing built-in extension XVideo-MotionCompensation
Initializing built-in extension SELinux
Initializing built-in extension GLX
info: OS: linux 3.16.4-tinycore64 sitespeed:3.0.0-beta-4  phantomJs:1.9.8 java:java version "1.7.0_65"
info: Will crawl from start point http://www.demorgen.be with crawl depth 1
info: Will analyze 1 pages
info: Queueing browsertime for http://www.demorgen.be firefox
info: Running browsertime for firefox http://www.demorgen.be

/usr/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/browsertime.js:79
        var driver = self.browsers.getBrowser(options.browser).getDriver(optio
                                   ^
TypeError: Cannot call method 'getBrowser' of undefined
    at /usr/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/browsertime.js:79:36
    at /usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:254:17
    at /usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:157:25
    at /usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:251:21
    at /usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:615:34
    at Browsertime.emit (/usr/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/async-node-events/lib/async-node-events.js:77:20)
    at /usr/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/browsertime.js:75:16
    at /usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:610:21
    at /usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:249:17
    at iterate (/usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:149:13)

When I run via Browsertime, the browsermobproxy.log file (for browsertime -u http://www.demorgen.be):

{"level":"error","message":"stderr:INFO 12/02 10:22:24 n.s.u.i.d.DataBuild~ - No pattern available for 'Other'.\n","timestamp":"2014-12-02T10:22:24.196Z"}
{"level":"error","message":"stderr:INFO 12/02 10:22:24 n.s.u.i.d.DataBuild~ - No pattern available for 'Personal computer'.\n","timestamp":"2014-12-02T10:22:24.200Z"}
{"level":"error","message":"stderr:INFO 12/02 10:22:24 n.s.u.d.AbstractUpd~ - Can not check for an updated version. Are you sure you have an established internet connection?\n","timestamp":"2014-12-02T10:22:24.362Z"}
{"level":"error","message":"stderr:INFO 12/02 10:22:25 n.l.b.p.Main         - Starting BrowserMob Proxy version %s\n","timestamp":"2014-12-02T10:22:25.193Z"}
{"level":"error","message":"stderr:INFO 12/02 10:22:25 o.e.j.u.log          - jetty-0.0.1-SNAPSHOT\n","timestamp":"2014-12-02T10:22:25.272Z"}
{"level":"error","message":"stderr:INFO 12/02 10:22:25 o.e.j.u.log          - started o.e.j.s.ServletContextHandler{/,null}\n","timestamp":"2014-12-02T10:22:25.340Z"}
{"level":"error","message":"stderr:WARN 12/02 10:22:25 n.s.u.d.UpdateOpera~ - The data can not be read from the specified URL: Can not open stream to the given URL: http://user-agent-string.info/rpc/get_data.php?key=free&format=xml\n","timestamp":"2014-12-02T10:22:25.469Z"}
{"level":"error","message":"stderr:INFO 12/02 10:22:25 n.s.u.d.UpdateOpera~ - Reading fallback data...\n","timestamp":"2014-12-02T10:22:25.470Z"}
{"level":"info","message":"Stopped proxy process.","timestamp":"2014-12-02T10:22:25.855Z"}

Some extra information

whereis firefox
firefox: /usr/bin/firefox /etc/firefox /usr/lib/firefox /usr/bin/X11/firefox /usr/share/man/man1/firefox.1.gz
dpkg -l | grep firefox
ii  firefox                          33.0+build2-0ubuntu0.14.04.1      amd64        Safe and easy web browser from Mozilla
xvfb-run firefox

(process:675): GLib-CRITICAL **: g_slice_set_config: assertion 'sys_page_size == 0' failed
Xlib:  extension "RANDR" missing on display ":99".
soulgalore commented 9 years ago

Hey @JeroenVdb , we will publish new releases tonight of sitespeed & browsertime that fix a couple of the common bugs.

When I run it on Ubuntu I've used xvfb-run:

xvfb-run sitespeed.io -u http://www.sitespeed.io -b chrome

About the current problem, I will check more tonight. Best Peter

JeroenVdb commented 9 years ago

I have the same error when I run you command. I will recheck my testcases with the new release tomorrow. Thanks!

xvfb-run sitespeed.io -u http://www.sitespeed.io -b firefox
info: OS: linux 3.16.4-tinycore64 sitespeed:3.0.0-beta-4  phantomJs:1.9.8 java:java version "1.7.0_65"
info: Will crawl from start point http://www.sitespeed.io with crawl depth 1
info: Will analyze 6 pages
info: Running YSlow for http://www.sitespeed.io
info: Running YSlow for http://www.sitespeed.io/logo/
info: Running YSlow for http://www.sitespeed.io/documentation/
info: Running YSlow for http://www.sitespeed.io/rules/
info: Running YSlow for http://www.sitespeed.io/faq/
info: Running YSlow for http://www.sitespeed.io/example/
info: Queueing browsertime for http://www.sitespeed.io firefox
info: Queueing browsertime for http://www.sitespeed.io/logo/ firefox
info: Queueing browsertime for http://www.sitespeed.io/documentation/ firefox
info: Queueing browsertime for http://www.sitespeed.io/rules/ firefox
info: Queueing browsertime for http://www.sitespeed.io/faq/ firefox
info: Queueing browsertime for http://www.sitespeed.io/example/ firefox
info: Running browsertime for firefox http://www.sitespeed.io

/usr/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/browsertime.js:79
        var driver = self.browsers.getBrowser(options.browser).getDriver(optio
                                   ^
TypeError: Cannot call method 'getBrowser' of undefined
    at /usr/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/browsertime.js:79:36
    at /usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:254:17
    at /usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:157:25
    at /usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:251:21
    at /usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:615:34
    at Browsertime.emit (/usr/lib/node_modules/sitespeed.io/node_modules/browsertime/node_modules/async-node-events/lib/async-node-events.js:77:20)
    at /usr/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/browsertime.js:75:16
    at /usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:610:21
    at /usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:249:17
    at iterate (/usr/lib/node_modules/sitespeed.io/node_modules/async/lib/async.js:149:13)
soulgalore commented 9 years ago

Hi again @JeroenVdb , so I think there is different issues, your issue seems like a bug, but I haven't been able to reproduce.

Install the latest version: npm install sitespeed.io@3.0.0-beta-5 -g

and lets hope that it works :) Best Peter

JeroenVdb commented 9 years ago

Works like a charm! Thanks @soulgalore!

xvfb-run sitespeed.io -u http://www.demorgen.be -m 1 --noYslow -b firefox
info: OS: linux 3.16.4-tinycore64 sitespeed:3.0.0-beta-5  phantomJs:1.9.8 java:java version "1.7.0_65"
info: Will crawl from start point http://www.demorgen.be with crawl depth 1
info: Will analyze 1 pages
info: Starting proxy on port 9092, will wait at most 15000 ms
info: Queueing browsertime for http://www.demorgen.be firefox
info: Running browsertime for firefox http://www.demorgen.be
info: Fetching http://www.demorgen.be (1 of 3)
info: Fetching http://www.demorgen.be (2 of 3)
info: Fetching http://www.demorgen.be (3 of 3)
info: Storing /sitespeed.io/sitespeed-result/www.demorgen.be/2014-12-02-16-06-19/data/har/firefox/www.demorgen.be.har
info: Storing /sitespeed.io/sitespeed-result/www.demorgen.be/2014-12-02-16-06-19/data/browsertime/firefox/www.demorgen.be-browsertime.json
info: Stopping proxy
info: Done analyzing urls
info: Wrote file assets.html
info: Wrote file domains.html
info: Wrote file hotlist.html
info: Wrote file pages.html
info: Wrote file rules.html
info: Wrote file index.html
info: Wrote file errors.html
info: Wrote file detailed-site-summary.html
info: Wrote file www.demorgen.be.html
info: Wrote results to /sitespeed.io/sitespeed-result/www.demorgen.be/2014-12-02-16-06-19
soulgalore commented 9 years ago

perfect, it is @tobli that has done all the hard work :)