search

hachyderm / community

Hachyderm Community Resources
https://community.hachyderm.io
Creative Commons Attribution Share Alike 4.0 International
490 stars 54 forks source link

Slow in east Asia #529

Open ppromet opened 6 months ago

ppromet commented 6 months ago

Please describe the issue you are seeing.

A couple questions to help you get started:

  1. Are you seeing error codes? If so please include the full error message and, if it makes sense to, a screenshot.
  2. When are you seeing the issue? When you are submitting posts? Uploading media?
  3. Is the issue only happening in certain circumstances? For example, only for uploading certain file sizes, or only when using a specific browser / 3rd party app. If so, please include which and what versions.

Please note that if there are issues with individual apps or with Mastodon core that we can work with you to determine that to be the case, but we can not fix third party apps and do not directly contribute to the Mastodon project at this time.

How long have you noticed this issue?

2-3 days

What geographic region are you accessing Hachyderm from?

east Asia

Is there anything else that we should know?

I obeserve that connection to Hackyderm is a bit slow in Asia, esp at night here Here are same random ping results comparing with other popular Mastodon instances:

ping -c 3 hachyderm.io PING hachyderm.io (172.104.253.29): 56 data bytes 64 bytes from 172.104.253.29: icmp_seq=0 ttl=47 time=355.414 ms 64 bytes from 172.104.253.29: icmp_seq=1 ttl=47 time=399.674 ms 64 bytes from 172.104.253.29: icmp_seq=2 ttl=47 time=315.311 ms

ping -c 3 mastodon.social PING mastodon.social (151.101.1.91): 56 data bytes 64 bytes from 151.101.1.91: icmp_seq=0 ttl=58 time=6.996 ms 64 bytes from 151.101.1.91: icmp_seq=1 ttl=58 time=3.431 ms 64 bytes from 151.101.1.91: icmp_seq=2 ttl=58 time=14.756 ms

ping -c 3 mas.to PING mas.to (172.67.166.96): 56 data bytes 64 bytes from 172.67.166.96: icmp_seq=0 ttl=57 time=3.494 ms 64 bytes from 172.67.166.96: icmp_seq=1 ttl=57 time=7.733 ms 64 bytes from 172.67.166.96: icmp_seq=2 ttl=57 time=11.232 ms

If I put c=10, there is even packet loss every time.

Preskton commented 5 months ago

howdy, @ppromet - we are starting to look at options on how to improve service in Asia in general. I was recently traveling through India & Japan and observed the same behavior you reported above.

I'm going to spend a bit this morning testing an edge in Japan to see how it helps out -- I'll let you know how it's looking and when it's available for testing.

Preskton commented 5 months ago

Alright, we are testing a set up that routes through Japan/Tokyo. You're welcome to test along with us by overriding your operating system's hosts file with these entries:

172.104.114.37 hachyderm.io
172.104.114.37 media.hachyderm.io

Once we have a bit more data, we'll identify which geo-locations will route through this edge by default. We'll then change those regions to pin through our new edge, at which point you won't need the hosts override.

agrmohit commented 5 months ago

In India, both old and new endpoints have the same ping at 160 ms. For comparision google is 60ms.

without editing hosts (ipv6)
Minimum = 159ms, Maximum = 164ms, Average = 160ms

after editing hosts (ipv4)
Minimum = 158ms, Maximum = 162ms, Average = 159ms
Preskton commented 5 months ago

Thanks for the data, @agrmohit!

kaos-ocs commented 5 months ago

From Melbourne, Australia using Optus NBN. hachyderm.io is 17 hops and is faster. 172.104.114.37 is 20 hops and slower.

$ mtr -r -n -c 10 hachyderm.io
HOST: ocs15                       Loss%   Snt   Last   Avg  Best  Wrst StDev
...
 17.|-- 74.207.251.5               0.0%    10  203.4 183.1 180.0 203.4   7.2
$ mtr -r -n -c 10 172.104.114.37
HOST: ocs15                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 20.|-- 172.104.114.37             0.0%    10  279.1 198.1 188.4 279.1  28.5
th3noname commented 5 months ago

I thought it would be helpfull if i run a few ripe atlas ping measurements against the new edge location and hachyderm in general. I hope this helps.

Ping from worldwide probes against 172.104.114.37: hachyderm_measurement_1

Ping from probes in the asia region against 172.104.114.37: hachyderm_measurement_3

Ping from worldwide probes against hachyderm.io: hachyderm_measurement_2

Preskton commented 1 week ago

Thank you all for this awesome data. It seems to tell me that cdn-tokyo-1 is great for japan but didn't help anywhere else much. One of the items on my to do list is to make our edge node spin up much quicker. Once we do this, then we'll look at some other edge pops. Seems like Melbourne and and maybe Singapore could be good; we'll also have to look at traffic patterns.

Thanks again - this is awesome.