Troubleshooting general

[adolfintel]

If you need troubleshooting, send me an email at info@fdossena.com, or use this thread. Please don't make new issues if you just need help.

[moeller0]

Hi adolftintel,

first of, nice project!

I have two smallish nits to pick: 1) Please have a look at line 38 of speedtest_worker.js: overheadCompensationFactor: 1048576/925000, //compensation for HTTP+TCP+IP+ETH overhead. 925000 is how much data is actually carried over 1048576 (1mb) bytes downloaded/uploaded. This default value assumes HTTP+TCP+IPv4+ETH with typical MTUs over the Internet. You may want to change this if you're going through your local network with a different MTU or if you're going over IPv6 (see doc.md for some other values)

This really is just a guestimate as the speedtest has no real idea about the true overhead on the link. The HTTP part will be under control of the speedtest, sure, and probably also the IP part (assuming you force IPv4) but how about TCP options? Also I believe you do not measure the actual MSS/MTU so this part really should be reported as additional data IMHO; so present the real measured HTTP/TCP/IP Goodput first and then also present estimates about gross rates at lower OSI layers, that way the user can choose which numbers to trust.

2) And lines 235: dlStatus = ((speed 8 settings.overheadCompensationFactor)/1048576).toFixed(2) // speed is multiplied by 8 to go from bytes to bits, overhead compensation is applied, then everything is divided by 1048576 to go to megabits/s and 351: ulStatus = ((speed 8 settings.overheadCompensationFactor)/1048576).toFixed(2) // speed is multiplied by 8 to go from bytes to bits, overhead compensation is applied, then everything is divided by 1048576 to go to megabits/s

actually assume Mebibytes/Mebibits (base 2^10) instead of Megabytes/Megabits (base 1000), as far as I can see in networking all rates are traditionally defined is SI units. It would make it much easier to compare this speedtests results with other tests if the conventional measures would be used (or at least the correct unit descriptors)

Best Regards

[adolfintel]

Yes, the overhead is an estimate of course. I estimated it from my ADSL, which seems to be the most common type over here. Also yes, I could add a MB/MiB setting.

[moeller0]

Ah, I believe the fact that it is an estimate should be reported prominently to the enduser, so one can understand the results better. By the way, have a look at https://github.com/moeller0/ATM_overhead_detector for getting empirical data on the true overhead on your ADSL link, it is way more complex than one naively assumes... Also the actual encapsulations in use are quite diverse, too diverse, I believe, to list them all in the results. BTW, can you get the number of packets used for the test, because it would be quite helpful to also show the user the number of packets and the volume of data transferred per direction.

[moeller0]

Side note: Overhead compensation/reporting is I believe quite an interesting feature for endusers, especially since many ISPs only "promise" gross rates an so the ability to get a decent estimate of the gross rate from a speed test is quite helpful. The challenge for doing this generically is simply that there are a lot of different options that are hard to deduce empirically from the speedtest server side, and making the user manually select the used encapsulation will mostly confuse the "target" audience of normal users (this is not meant as an offense, nobody except a few axperts should have to know this ;))

[adolfintel]

It's probably overkill. A good compromise could be to ignore overhead and output in MiB. Either way, it's a miniscule difference, I doubt anyone cares.

[moeller0]

Sure, reporting the measured rate seems like a decent idea, and maybe add the IP-level rate as well (as anyone is bound to use IP to reach the server), but anything higher like ethernet/ATM-ADSL/PTM-VDSL2 will just be confusing. But the number of packets sent would be great ;). The difference between Mb and Mib (100 - 100*1000^2/1024^2) = 4.6% will be ignored by a lot of users (those that look at the order of magnitude of the test result and their contracted rates) but there are users who will obsess over roughly 5%. Whether one wants to cater to those is i different question though ;).

[geekism]

Not really a issue, But I love the HTML5 Speedtest, I have about 50 mirrors. And i wanted something to show the test records from using 'example-telemetry.html'. So i wrote one.. You can view it here..

http://code.justla.me/index.php?file=php/html5speedtesttelemetry.php

[adolfintel]

@geekism I have an almost identical code that I use on my server. I haven't published it because it's very basic and generates very long pages. If you have time, you could make some page system and send a PR.

Also, a couple of suggestions for improving your page:

• you can mix HTML and PHP, you don't need to echo HTML tags
• don't ever ever ever use px, use em or another modern unit

[cayxxx]

Hello , thanks of your project . I've run it on my http website very well. But recently I turned it to https , the speedtest page became abnormal. IP Address and Jitter cannot be displayed ,and the speed became very very slow. How can i fix it ?

[adolfintel]

@cayxxx Do you get an error 500 if you try to open getIp.php or any other PHP file?

[cayxxx]

@adolfintel Thank you for reminding! It works perfectly after I installed php. But I am wondering why it could work before without php.

[adolfintel]

With the default settings, the test requires PHP. It's possible that your web server was configured to use PHP only with HTTP, and when you used HTTPS something changed.

[mattorola7]

I really like this project. I would like to contribute back a command line client (instead of web browser) that could perform speed tests against the server. Has anybody developed a command line client for it already before we embark on the task?

[adolfintel]

This is a speedtest for browsers, if you can use the command line, iperf is a better solution

[mattorola7]

We probably have a corner use case. I thought iPerf would enter the discussion :) I've been using iperf but I don't think it is possible to do exactly what we are trying to accomplish with it. If iperf would stream its JSON output rather than only output it at the end of the test, it would make it easier for us to build a "speed test API". Ultimately, we want to animate the speedtest result, even though it is run behind the scenes between server to server. This animation is just not going to happen in enough "real time" as we hoped.

[adolfintel]

Hmm, I don't know much about iperf, but I really can't recommend this project for your use case. This test runs over HTTP, the result is not as accurate as iperf, and also the amount of processing that has to be done on both sides is greater by at least an order of magnitude, which means that if you have a really fast link like a 10 Gbit/s, you'll get a completely wrong result. If you still want to use this, there is nothing stopping you, but I strongly recommend something else. Look at iperf's documentation, maybe there is a real time output that you can parse.

[ketterling]

Browser choice makes a difference. For a client, I'm using a recent iMac, and the server is on our local network (all gigabit Ethernet). I noticed that using Firefox, my reported download/upload speeds were consistently 990/600 Mbps. After troubleshooting for awhile, I tried Chrome and consistently get 995/995 Mbps. Something to consider if you're testing a local network.

[adolfintel]

@ketterling Yes, some browsers are a bit heavier on the CPU than others, therefore the speed may be limited on very fast connections. Still, I wouldn't recommend this tool to test a local network, iperf would be better unless you absolutely want to run the test from a browser

[ketterling]

@adolfintel I was trying to understand what appeared to be a slow speed test for my new 200Mbit Internet service provided by my cable company. Having a solution such as yours (great solution, by the way) allowed me to test over the Internet and also to perform the same test locally and across my firewall. With this test, I was able to conclusively pinpoint my firewall, and a bit of trial and error pointed me to a particular setting. I'm very grateful for your solution! And honestly, I'd never heard of iperf until finding this page, but I'll be reading up about it tonight.

[niko2]

Great tool @adolfintel !

I would like to share a strange behavior with Chrome (whatever OS I use, Linux, android or Win) : each time it runs, my web server (apache mpm_event php7-fpm) answers (exactly and always) 3 bad requests atfer test is finished. I don't have this behavior with Firefox. I tried to disable parallel_downloading in chrome flags but got poor speed results and same apache errors:

access_log
xx.yy.zz.qq - [18/Jul/2018:01:53:22 +0200] "POST /empty.php?r=0.37227131738601105 HTTP/1.1" 400 226 "http://**********/speedtest_worker.js" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"
error_log
[Wed Jul 18 01:53:44.285925 2018] [proxy_fcgi:error] [pid 8111:tid 139725453113088] (70008)Partial results are valid but processing is incomplete: [client xx.yy.zz.qq:57824] AH01075: Error dispatching request to : (reading input brigade), referer: http://**********/speedtest_worker.js

looking at the log timestamp, you may notice the access log with error 400 is older than telemetry.php (which is not shown here but trust me) but only appears at the same time as the php error, which is stamped after telemetry.php. So it looks like, the 3 requests are blocked somewhere in apache and never finish until error. There is about 35s between the requests and their php errors, which is not related to any (apache or php) timeout configured.

Strange isn't it ?

[adolfintel]

@niko2 I'm 99% sure that those are the incomplete upload XHRs that have been terminated at the end of the test. Nothing to worry about I'd say. Is it causing you problems?

[niko2]

thank you @adolfintel for your quick answer; no it is not causing me problems except flooding my error logs which I prefer to avoid. Do you have any clue about the different behavior between Chrome and Firefox ?

[adolfintel]

@niko2 I don't know the implementation details. I guess Firefox just closes the socket so the server treats it as a request that the client failed to send, but Chromium just leaves it hanging. Pure speculation of course, but if you're interested you can use wireshark to see exactly what's being sent (or not sent).

[GuojlWeb]

Why can not upload when speed is detected in mobile browser?

[adolfintel]

@GuojlWeb what browser?

[sistemabc]

hey I can't get ISP data, there is written "unknown ISP", can you help me? @adolfintel

[dunklesToast]

@sistemabc Do you use this in your local network or is it hosted on a server outside your networ? If it’s in your local network the Speedtest can’t determine your ISP since we use the IP for that.

[sistemabc]

So how do I run so I get data ISP? I mean when running in the browser, is it "localhost/speedtest/" ?

[dunklesToast]

If you want to use the ISP Feature, you need to have the Speedtest Instance running on a Device outside your local network. Inside your network (or when you host it on the same machine where you test it) the Speedtest will use the internal IP and this can't be used to determine the ISP. Since you wrote localhost/speedtest I assume you host it on the same machine you make the speedtest. This is a wrong way of using the Speedtest software since it can't display correct Network data, as the traffic won't mpve through your Network.

[sistemabc]

oh I see. where I can find a configuration directory server who located to running the speed test? so I can configure that to my server. @dunklesToast

[dunklesToast]

@sistemabc What do you mean with "configuration directory server"? You can simply clone this git repo to your server git clone https://github.com/adolfintel/speedtest.git. If this is done you may or may not need to change your webserver (apache, nginx, lighttpd etc.) config. So it points to the speedtest.

[sistemabc]

I mean is a script that can do the testing, and I see your speedtest project is located in milan. how do i configure it so i can change it to my server in jakarta? @dunklesToast

[dunklesToast]

It’s that what I wrote you. If it’s a v/root server you can connect via ssh and clone this Repo. The Repo contains all the code needed for the page to run. After you cloned the Repo you need to configure your Webserver, so it uses the speedtest folder for a specific servername. Contact me via dunklestoast@gmail.com if you need further assistance

[adolfintel]

@sistemabc @dunklesToast before you go too deep into troubleshooting, make sure that the web server has permissions to connect to other servers, because it's needed to connect to the ipinfo service. This problem was already reported by another user whose server had this kind of protection and PHP wasn't allowed to go online.

[dunklesToast]

It is not the Webserver. He tried running this on localhost...

[adolfintel]

Yea I just noticed, sorry. If you run the test on localhost of course you won't get ISP info. @sistemabc The part on my test where it says that the server is in Milan is just something I wrote on the page to inform users from other countries of why they may get high pings, etc.

[sistemabc]

do you have a tool to calculate the packet loss? it's some kind like ping on cmd. send the packet with load and then the server give back the packet. so we can know how many packet will return to the client

[adolfintel]

No, you can't measure packet loss over HTTP since TCP hides retransmissions.

On August 9, 2018 1:41:50 PM GMT+02:00, sistemabc notifications@github.com wrote:

do you have a tool to calculate the packet loss? it's some kind like ping on cmd. send the packet with load and then the server give back the packet. so we can know how many packet will return to the client

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/adolfintel/speedtest/issues/75#issuecomment-411729083

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Brijendrasial]

It would be great if you can add this feature somehow :) Much needed

[dunklesToast]

It’s not possible since TCP ensures that all packets are delivered - the packet loss would be 0 anytime

[sistemabc]

@adolfintel may i ask you about the logic of ping calculation and jitter calculation?

[adolfintel]

@sistemabc The test works like this:

• An empty file (empty.php) is downloaded repeatedly. This requires 1 packet for the request and 1 packet for the response
• The response contains a keep-alive header so the TCP connection only needs to be established once
• The first response is discarded because it takes longer to do it (3 way handshake for TCP, request, response)
• The other responses (35 by default) are averaged to calculate the ping. Ping is measured as the amount of time between the request being sent and the response being received. If the browser supports the Performance API, this timing can be quite accurate.
• The jitter basically tells how variable the ping times are and it's calculated during the ping test

[sistemabc]

I have an aws server in Singapore then I tried the test using this tool. and I get a fairly large ping 284 ms. and I tried again using speedtest.net by pointing to the Singapore server as well (but it's just some server not mine) I get a ping only 23 ms. is there something wrong with my configuration? btw i'm on Indonesia

[adolfintel]

It's probably just the server being slow, I assume ookla has some badass servers for their speedtest, but if you can give me the URL I can see if it's responding correctly.

[adolfintel]

@sistemabc it's configured correctly, the server is just responding slowly. I'm getting 300ms ping from Italy.

[dunklesToast]

Same here. Server is fine and Ping measured by speedtest is nearly the same as the one provided by a normal ping request. Possible the Amazon Instance is slow? Try to make a speedtest there.

apt install speedtest -y && speedtest

[adolfintel]

@dunklesToast It's pretty odd, I wouldn't expect an aws server to have such an awful ping, but maybe there's other stuff hosted on that server

[frizurd]

Hey @adolfintel

Awesome project.

Was just wondering how it works compared to fast.com and speedtest.net. What I noticed was that the amount of data transferred (checked via Chrome developer tools) with your application increases depending on the speed of the internet. I understand that the PHP file gets downloaded in a loop. If I understand correctly, this may result in high data transfer server costs if the application gets used by many.

If I compare it with services such as fast and speed test, their data transfer stays below 1 MB. May I ask what the difference is? I apologize for my ignorance.

[adolfintel]

@frizurd It's a perfectly legitimate question. The reason why my speedtest uses more or less data depending on the connection speed is because the tests have a default length of 15 seconds instead of transferring a known amount of data. During the test, the data gets downloaded (or uploaded) repeatedly until the time passes. At the beginning of each test there's a short "grace time" during which timing info is discarded so that buffering done by your network card's driver doesn't affect the results of the test.

It's just one way of doing it. I think speedtest.net stops when it reaches a certain stability threshold, but I haven't reverse engineerd it, my test works completely differently from theirs as far as I know.

[adolfintel]

@MohamedHajji Can you tell me more about it? What database are you using? Anything in the PHP log?