offspot / imager-service

Create Kiwix Hotspot microSD cards online
https://imager.kiwix.org/
GNU General Public License v3.0
11 stars 6 forks source link

What affects download speeds? #276

Open Popolechien opened 2 years ago

Popolechien commented 2 years ago

This is sort of related to #266 I guess: there's a user that is located on the US East Coast that tried downloading an image from our servers. It is however pretty slow, eventhough they claim download speeds over 100+ Mbps Download Speed I've tried downloading one of their images and actually did get much better DL speeds. Any idea why/how this could be and how we could remediate this? Capture d’écran 2021-10-19 à 10 36 10

rgaudin commented 2 years ago

We've discussed that several times, but it might not have been captured in a ticket:

Of course, the user's bandwidth and route congestion between user and the server play a big role but we've seen that frequently, users face slow download speeds, in the orders shown above: ~650KB/s.

The fact that it's roughly the same speed could indicate some sort of throttling on Wasabi side. We don't know if they reduce bandwidth after intensive use – IP? – (they don't mention it on their doc) nor if we've gone over the limits of that policy.

At this point, I'd suggest you contact them to ask why we're so frequently getting limited speeds.

Popolechien commented 2 years ago

Will do, thanks.

Popolechien commented 2 years ago

Ok Wasabi asked us to run the following tests:

https://wasabi-support.zendesk.com/hc/en-us/articles/115003033352-I-am-not-seeing-the-network-performance-I-m-expecting-what-should-I-do-?

rgaudin commented 2 years ago

Thanks, how I understand it is, we should follow the following procedure when encountering issues:

I suggest we do this the next time one of us is affected. Seems a bit much to ask from our clients… as this won't lead to an immediate resolution.


Sample runs from my 35/2Mbps connection (no issue ATM). Confirms #266 would benefit people from EU (or routing from EU).

Speedtest and route to eu-central-1 (⚠️ not what we're using!)

Screen Shot 2021-10-20 at 08 36 46
sudo mtr -T -P 443 -rw s3.eu-central-1.wasabisys.com
Start: 2021-10-20T09:00:55+0000
HOST: faku.local                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- yeleman-camel                           0.0%    10    1.6   1.9   1.6   2.1   0.1
  2.|-- 197.155.137.101                         0.0%    10    1.9   2.2   1.9   2.9   0.3
  3.|-- 213.154.92.106                          0.0%    10   26.8  86.8  26.8  99.5  21.4
        193.251.248.175
        81.52.188.117
  4.|-- 196.207.255.193                        40.0%    10   18.8  54.5  17.3  90.6  39.1
        hundredgige0-11-0-11.pastr4.paris.opentransit.net
  5.|-- 213.144.187.194                         0.0%    10   81.1  81.5  54.5  91.4  11.2
        be2868.ccr41.lon13.atlas.cogentco.com
        be2871.ccr42.lon13.atlas.cogentco.com
  6.|-- be3183.ccr41.par01.atlas.cogentco.com   0.0%    10   84.2  69.7  54.3  93.6  15.6
        e0-4.switch1.lis1.he.net
        be3184.ccr42.par01.atlas.cogentco.com
        be12488.ccr42.ams03.atlas.cogentco.com
  7.|-- be12266.ccr42.ams03.atlas.cogentco.com  0.0%    10   86.5  79.6  60.2  93.6  13.3
        be2278.rcr21.b038092-0.ams03.atlas.cogentco.com
        e0-6.switch1.mad2.he.net
        be2283.rcr21.b038092-0.ams03.atlas.cogentco.com
  8.|-- 149.11.201.210                          0.0%    10   88.8  90.2  88.3  95.6   2.4
        be2278.rcr21.b038092-0.ams03.atlas.cogentco.com
        100ge10-1.core1.par2.he.net
        be2283.rcr21.b038092-0.ams03.atlas.cogentco.com
  9.|-- 149.11.201.210                          0.0%    10   87.3  96.1  86.7 141.5  16.6
        130.117.252.16
        port-channel13.core2.ams1.he.net
 10.|-- 130.117.252.16                          0.0%     6   96.2  97.8  93.0 100.8   2.9
        wasabi-technologies-inc.port-channel3.switch1.ams4.he.net

Speedtest and route to us-west-1 (where our servers are)

Screen Shot 2021-10-20 at 08 39 26
sudo mtr -T -P 443 -rw s3.us-west-1.wasabisys.com
Start: 2021-10-20T08:32:23+0000
HOST: faku.local                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- yeleman-camel                           0.0%    10    1.8   1.8   1.6   2.4   0.2
  2.|-- 197.155.137.101                         0.0%    10    2.0   3.4   1.8  15.8   4.4
  3.|-- 213.154.92.106                          0.0%    10   24.5  79.4  24.5  98.2  28.6
        81.52.188.117
        193.251.248.175
  4.|-- 196.207.255.193                        70.0%    10   20.0  43.4  20.0  89.9  40.3
        hundredgige0-11-0-11.pastr4.paris.opentransit.net
  5.|-- 213.144.187.194                         0.0%    10   52.1  71.2  50.8  86.8  16.8
        be2868.ccr41.lon13.atlas.cogentco.com
        be2871.ccr42.lon13.atlas.cogentco.com
  6.|-- e0-4.switch1.lis1.he.net                0.0%    10   57.7 184.2  57.0 1072. 314.4
        be2099.ccr31.bos01.atlas.cogentco.com
        be2101.ccr32.bos01.atlas.cogentco.com
        be3184.ccr42.par01.atlas.cogentco.com
  7.|-- be12489.ccr42.lon13.atlas.cogentco.com  0.0%    10   87.1 108.9  60.3 158.8  40.3
        be12497.ccr41.lon13.atlas.cogentco.com
        be3600.ccr22.alb02.atlas.cogentco.com
        e0-6.switch1.mad2.he.net
        be3599.ccr21.alb02.atlas.cogentco.com
  8.|-- be2879.ccr22.cle04.atlas.cogentco.com   0.0%    10  165.3 124.0  66.7 168.4  44.7
        be2878.ccr21.cle04.atlas.cogentco.com
        100ge0-34.core1.bio1.he.net
        be2101.ccr32.bos01.atlas.cogentco.com
        be2099.ccr31.bos01.atlas.cogentco.com
  9.|-- 100ge0-30.core1.orf2.he.net             0.0%    10  159.2 156.3 149.0 163.7   4.5
        be3600.ccr22.alb02.atlas.cogentco.com
        be2718.ccr42.ord01.atlas.cogentco.com
        be3599.ccr21.alb02.atlas.cogentco.com
 10.|-- be2831.ccr21.mci01.atlas.cogentco.com   0.0%    10  175.5 167.3 154.7 176.6  10.1
        be2832.ccr22.mci01.atlas.cogentco.com
        be2878.ccr21.cle04.atlas.cogentco.com
        100ge15-1.core2.ash1.he.net
 11.|-- be2718.ccr42.ord01.atlas.cogentco.com  10.0%    10  164.4 171.7 158.1 189.8  13.3
        100ge1-2.core1.ash1.he.net
        be3036.ccr22.den01.atlas.cogentco.com
        be3035.ccr21.den01.atlas.cogentco.com
 12.|-- be3037.ccr21.slc01.atlas.cogentco.com   0.0%    10  205.5 188.1 174.1 205.6  14.2
        be2832.ccr22.mci01.atlas.cogentco.com
        be3038.ccr32.slc01.atlas.cogentco.com
        be2831.ccr21.mci01.atlas.cogentco.com
 13.|-- be3035.ccr21.den01.atlas.cogentco.com   0.0%    10  188.4 194.7 187.6 218.1  12.4
        be3036.ccr22.den01.atlas.cogentco.com
        be2029.ccr22.sea02.atlas.cogentco.com
        100ge14-2.core1.msp1.he.net
        be2042.ccr21.sea02.atlas.cogentco.com
 14.|-- be2670.ccr21.pdx01.atlas.cogentco.com   0.0%    10  223.6 224.0 206.2 240.0   9.4
        be3038.ccr32.slc01.atlas.cogentco.com
        100ge11-2.core1.sea1.he.net
        be3037.ccr21.slc01.atlas.cogentco.com
        be2671.ccr21.pdx01.atlas.cogentco.com
 15.|-- 100ge0-35.core1.pdx3.he.net             0.0%    10  234.0 231.5 215.6 249.8  10.1
        100ge15-1.core1.pdx1.he.net
        be2042.ccr21.sea02.atlas.cogentco.com
        be2216.ccr51.pdx02.atlas.cogentco.com
        be2029.ccr22.sea02.atlas.cogentco.com
 16.|-- 38.104.108.138                          0.0%    10  222.7 222.5 213.1 235.5   6.5
        be2671.ccr21.pdx01.atlas.cogentco.com
        wasabi-technologies-inc.e0-53.switch1.pdx3.he.net
        100ge0-36.core1.pdx2.he.net
 17.|-- 76.191.80.53                            0.0%    10  225.6 222.1 213.6 231.3   5.1
        100ge0-35.core1.pdx3.he.net
        be2216.ccr51.pdx02.atlas.cogentco.com

10MiB file retrieval test

for count in {1..5}; do dig +noall +answer s3.us-west-1.wasabisys.com | tail -1; echo -n `date`" -- ";curl -w 'bytesPerSecondDownload %{speed_download}\n' -so /dev/null http://s3.us-west-1.wasabisys.com/org-kiwix-hotspot-cardshop-download/10Mio.dat; done
us-west-1.wasabisys.com. 120    IN  A   76.191.80.53
Wed Oct 20 08:50:58 GMT 2021 -- bytesPerSecondDownload 397639.000
us-west-1.wasabisys.com. 120    IN  A   76.191.80.53
Wed Oct 20 08:51:25 GMT 2021 -- bytesPerSecondDownload 348040.000
us-west-1.wasabisys.com. 120    IN  A   76.191.80.53
Wed Oct 20 08:51:55 GMT 2021 -- bytesPerSecondDownload 490470.000
us-west-1.wasabisys.com. 120    IN  A   76.191.80.53
Wed Oct 20 08:52:17 GMT 2021 -- bytesPerSecondDownload 704688.000
us-west-1.wasabisys.com. 120    IN  A   76.191.80.53
Wed Oct 20 08:52:31 GMT 2021 -- bytesPerSecondDownload 515372.000
kelson42 commented 1 year ago

My feedback, download a log of Zimfar, so save object-cache provider, bucket located in Europe (my guess), my download moves beetween 6MB/s and 13MB/s, so basically between 50 and 100Mb/s.

rgaudin commented 1 year ago

zimfarm logs bucket, as almost all of our buckets are in us-west-1 (portland)