Chinese mirror of the package registry

chubbyginger commented 2 years ago

What kind of issue is this?

[x] Feature Request. Start by telling us what problem you’re trying to solve. Often a solution already exists! Don’t send pull requests to implement new features without first getting our support. Sometimes we leave features out on purpose to keep the project small.

Configuration

Operating system: Windows 10 Home 21H2 64-bit

PlatformIO Version (platformio --version): PlatformIO Core, version 6.0.2

Description of problem

I am a PlatformIO user from China. It is all known to devs in China that there is a You-Know-What system that will cause accessing github.com becoming very, very, very slow or even unreachable (ERR_CONNECTION_RESET).

When I create a PlatformIO project, it will download a lot of data, including toolchains, SDKs, etc from github or some places like that. Thus the process is extremely slow and it takes hours to install that. In many situations there is completely no network activity.

internet

I think it is appropriate for PlatformIO developers to add a import function which can import pre-downloaded toolchains, SDKs, etc (NOT the ones downloaded from the "Platform" page. Seems that it is only a project framework.). There should also be a place to manually download the files, so that I can call my friends to help me download them.

Steps to Reproduce

Create a new project.

Actual Results

The project wizard stuck there showing

Please wait...
The first initialization requires Internet connection and may take a while (need to install dependent toolchains, frameworks, SDKs).
Please be patient and let the initialization complete.

Expected Results

Project can be created successfully

ps. My board: Espressif ESP32 Devkit

ivankravets commented 2 years ago

Could you provide a download speed for each mirror specified below according to their number?

This information will help us to improve our CDN. We had a plan to mirror our packages to the China region but we could not do this because most cloud providers require us to be a citizen of China. If you have ideas on how to help us here, please email me@ikravets.com.

chubbyginger commented 2 years ago

Mirror 1: 31.9 kB/s Mirror 2: 20.8 kB/s Mirror 3: 1.2 MB/s Mirror 4: 30.4 kB/s

Seems that Mirror 3 is the fastest. How can I switch to this mirror in PlatformIO?

It is difficult for non-Chinese citizens to build a website in China. You may provide some information on how to build a PlatformIO mirror so that people who are interested in PlatformIO may build there own mirrors in China.

ivankravets commented 2 years ago

Could you try our new mirror in Hong Kong?

https://dl.registry.ns2.platformio.org/tools/84/b8/d7e68fd716ef2a2a7a7f915418fe8c5e77debca2f52314d3d2ddfadb9b49/contrib-pysite-windows_amd64-2.37.191020.tar.gz

chubbyginger commented 2 years ago

17.6 kB/s, quite slow.

ivankravets commented 2 years ago

Do you use a proxy?

chubbyginger commented 2 years ago

Yes. Without the proxy it is even slower, at 8 kB/s

ivankravets commented 2 years ago

I see. This is the problem of your proxy provider, please contact it. Hundreds of thousands of users from Chinese use PlatformIO every day and this is the first case.

We are going to close this issue. If someone experiences it, please leave a comment here. We will reopen.

chubbyginger commented 2 years ago

It seems that it is a common problem in China. Creating PlatformIO projects is slow even with a proxy. Type "platformio" into the search box of Baidu, the most used search engine in China, and you will find a lot of devs encountering this problem. platformio very slow Not many people mention this problem in GitHub issues, because many users just don't want to ask a question in GitHub.

I think it is still quite necessary to give out instructions on how to build a mirror as it enables users and, sometimes universities (many universities run public mirrors available to everyone) to build their own ones. They can provide faster mirror services.

After I went through the docs carefully, it seems that PlatformIO does allow users to install ZIP packages. My mistake

Anyway, thanks for your help.

ivankravets commented 2 years ago

@yangshunhuai, @424778940z, @jiladahe1997, @LONGER3D, @zsm666, @Gianthard-cyh could you test the next links without any VPNs, or proxies? Thanks in advance!

CN1: https://hk03.layerstack.com/50MB.test CN2: http://103.136.146.146/100MB.test CN3: https://sin1.contabostorage.com/ba5dbd0aeacc4876a4a275a0b31216ca:pioregmirrorsg/tools/e2/52/6bc7480216c01b464603619a3840b8e5ec6118c8118d8a79e909b1b82960/toolchain-xtensa-windows-1.40802.0.tar.gz

Gianthard-cyh commented 2 years ago

CN1: less than 1MB/s CN2 : about 3MB/s @ivankravets

mujinsong commented 2 years ago

@yangshunhuai, @424778940z, @jiladahe1997, @LONGER3D, @zsm666, @Gianthard-cyh could you test the next links without any VPNs, or proxies? Thanks in advance!

CN1: https://hk03.layerstack.com/50MB.test CN2: http://103.136.146.146/100MB.test

CN1:about 500 kB/s CN2: about 3.5MB/s pretty great!!!THANKS for your team @ivankravets

ivankravets commented 2 years ago

I've just added 3rd link, Singapore, what is the speed for it? Hope we will finally significantly improve download speed for Chinese customers.

Gianthard-cyh commented 2 years ago

I've just added 3rd link, Singapore, what is the speed for it? Hope we will finally significantly improve download speed for Chinese customers.

CN3 about 17.5kB/s not so good

Gianthard-cyh commented 2 years ago

I've just added 3rd link, Singapore, what is the speed for it? Hope we will finally significantly improve download speed for Chinese customers.

CN2 is good enough thank you for all you have done to Chinese users @ivankravets

ivankravets commented 2 years ago

It is strange, CN2 - Sydney - Australia, where CN1 - China Direct CN2 Route. It is not clear to me how CN2 is better than CN1.

424778940z commented 2 years ago

CN1: stable 10M CN2: jump between 3-5M CN3: jump between 2.5-3.5M

Please note, I'm using China Unicom (no, it's not unicorn lol) network, usually is better for access servers that outside China. There are also other major ISPs: China Telecom usually good for anything with in China, but really sucks for global servers, unless you pay for additional optimized plans. China Mobile which always sucks lol Education Network and Radio and TV Broadband are not regular ISPs (and usually sucks), but some people use them because they are free or very cheap.

Gianthard-cyh commented 2 years ago

I've just added 3rd link, Singapore, what is the speed for it? Hope we will finally significantly improve download speed for Chinese customers.

CN2 is good enough thank you for all you have done to Chinese users @ivankravets

note I'm using China Mobile which always sucks that mentioned by @424778940z China Mobile may have a slow link to Honkong area

Gianthard-cyh commented 2 years ago

We use this service to test download speed https://www.17ce.com/

So, which location in China do you recommend in this case?

Usually Guangdong or Shanghai has the fastest access Yunnan or Guizhou etc. for the average level

424778940z commented 2 years ago

Few recommendations:

Try get in touch with TsingHua University mirror hosting see if the would like to host platformio packages
Tencent Cloud and Ali Cloud are providing very good hosting services, like the Chinese version of AWS
If decide to host on self own/rented servers, make sure it has China Unicom and China Telecom dual connections.
Cloudflare should also be an option

Hosting location I recommend Shanghai or other city that have CN2 (I mean the CN2 backbone network, not the speed test server) network access

ivankravets commented 2 years ago

Could I ask to test another Sydney DC - https://proof.ovh.us/

Switch to the Syndey on the "Server Selection".

It warns about HTTPs, I've just skipped it, it looks like they have a broken certificate.

ivankravets commented 2 years ago

LayerStack provides CN2, could you try their test files? https://hk03.layerstack.com/

424778940z commented 2 years ago

Could I ask to test another Sydney DC - https://proof.ovh.us/

Switch to the Syndey on the "Server Selection".

https://www.nperf.com/en/r/3394835265503124-BuZnnQHk

ivankravets commented 2 years ago

Tencent Cloud and Ali Cloud, etc

They require Chinese data, we don't have any roots in China. Also, their pricing model does not fit our requirements. We need unlimited network traffic, where they charge per gigabyte. We serve dozens of terabytes of data per day.

ivankravets commented 2 years ago

@424778940z, thanks for your results! So! What do we have! The best download speeds were reported from Sydney's data centers (Australia). I don't have explanations for this :)

424778940z commented 2 years ago

LayerStack provides CN2, could you try their test files? https://hk03.layerstack.com/

starting at 10M, then slowly drop to 3.5m and until finished.

chubbyginger commented 2 years ago

CN1: 60.3 kB/s CN2: 5.1 MB/s CN3: 21 kB/s

CN2 is also the fastest here. ISP: China Mobile Loc: Shanghai

Gianthard-cyh commented 2 years ago

Tencent Cloud and Ali Cloud, etc

They require Chinese data, we don't have any roots in China. Also, their pricing model does not fit our requirements. We need unlimited network traffic, where they charge per gigabyte. We serve dozens of terabytes of data per day.

https://www.nperf.com/r/3394835807609173-wHt1wz5h

Also we can actually ask if Tsinghua University mirror would like to host platformio packages. They provide mirrors of npm and linux distributions for millions of users for free.The mirror is often used in China.

424778940z commented 2 years ago

Also we can actually ask if Tsinghua University mirror would like to host platformio packages. They provide mirrors of npm and linux distributions for millions of users for free.The mirror is often used in China.

Agreed, if they could do it, that will be awesome

ivankravets commented 2 years ago

So, the final test. This is the 3rd data center in Sydney, I search them randomly. This time - Vultr:

https://syd-au-ping.vultr.com/

What is the speed for https://syd-au-ping.vultr.com/vultr.com.100MB.bin ?

424778940z commented 2 years ago

What is the speed for https://syd-au-ping.vultr.com/vultr.com.100MB.bin ?

start at 8M, then stable at 7M, drop to 5M at the very end

ivankravets commented 2 years ago

Thanks to all! We will start searching for a good proposal in Sydney according to our needs. We will be back here soon and ask finally to test mirrored PlatformIO packages.

chubbyginger commented 2 years ago

So, the final test. This is the 3rd data center in Sydney, I search them randomly. This time - Vultr:

https://syd-au-ping.vultr.com/

What is the speed for https://syd-au-ping.vultr.com/vultr.com.100MB.bin ?

Stable at 2.2 MB/s.

Gianthard-cyh commented 2 years ago

So, the final test. This is the 3rd data center in Sydney, I search them randomly. This time - Vultr:

https://syd-au-ping.vultr.com/

What is the speed for https://syd-au-ping.vultr.com/vultr.com.100MB.bin ?

Stable 1.5MB/s

ivankravets commented 2 years ago

Hi there,

Good news. We added a new mirror to the Asia continent. Could I ask you to re-test this file? Which is download speed without any proxies or VPNs?

Example file: https://dl.registry.ns2.platformio.org/tools/34/0c/1bb3ce94604147dd87b7cb891a39637da5624ed991fe6fb9477b373fab2c/toolchain-gccarmnoneeabi-windows_x86-1.50201.0.tar.gz

chubbyginger commented 2 years ago

Hi there,

Good news. We added a new mirror to the Asia continent. Could I ask you to re-test this file? Which is download speed without any proxies or VPNs?

Example file: https://dl.registry.ns2.platformio.org/tools/34/0c/1bb3ce94604147dd87b7cb891a39637da5624ed991fe6fb9477b373fab2c/toolchain-gccarmnoneeabi-windows_x86-1.50201.0.tar.gz

Stable 2.4 MB/s.

Gianthard-cyh commented 2 years ago

Hi there,

Good news. We added a new mirror to the Asia continent. Could I ask you to re-test this file? Which is download speed without any proxies or VPNs?

Example file: https://dl.registry.ns2.platformio.org/tools/34/0c/1bb3ce94604147dd87b7cb891a39637da5624ed991fe6fb9477b373fab2c/toolchain-gccarmnoneeabi-windows_x86-1.50201.0.tar.gz

2.4MB/s too

ivankravets commented 2 years ago

The additional mirror does not solve the problem for all Chinese users. We will look for a better data center.

ivankravets commented 2 years ago

@yangshunhuai, @Gianthard-cyh are you from the "green" zones?

Gianthard-cyh commented 2 years ago

@yangshunhuai, @Gianthard-cyh are you from the "green" zones?

yep i'm in Yunnan

ivankravets commented 2 years ago

@yangshunhuai, could you re-test China Direct Route (CN2)? Does it still provide a small speed?

CN2 Route: https://hk03.layerstack.com/100MB.test

chubbyginger commented 2 years ago

@yangshunhuai, could you re-test China Direct Route (CN2)? Does it still provide a small speed?

CN2 Route: https://hk03.layerstack.com/100MB.test

Avg speed 418 kB/s

@yangshunhuai, @Gianthard-cyh are you from the "green" zones?

yep I am in Shanghai

chubbyginger commented 2 years ago

YESSSSS! PlatformIO is working well now using China Mobile network.

chubbyginger commented 2 years ago

真香

ivankravets commented 2 years ago

Happy to hear that we finally improved the developer experience for Chinese users.

P.S: If someone in China has any issues working with installing PlatformIO, please post a comment here. We will reopen this issue.

captainstdin commented 2 years ago

This feeling is too bad. The chinese mainland users are restricted by the firewall （GFW），

I can't even initialize the project。

Mirrors can be set in other languages，I also tried setting

C:\Users\admin\.platformio\penv\pip.conf

[global]
index-url = https://mirrors.aliyun.com/pypi/simple/

[install]
trusted-host=mirrors.aliyun.com

You are not Chinese citizens ，cannot provide services in China。 but，You can contact the Aliyun official( Aliyun.com is a chinese cloud computing equivalent to Amazon cloud computing in the United States) ,aliyun can provide free mirros for chinese mainland ，https://mirrors.aliyun.com/

424778940z commented 2 years ago

Mirrors can be set in other languages，I also tried setting

not really sure what you means... pio is no a language

You are not Chinese citizens ，cannot provide services in China。

I don't think it'is required, sure the GFW blocked lots of things, but its not opreated in whitelist mode exclusively

aliyun can provide free mirros for chinese mainland

they could dose not means they will offer it for free, and you are mixing the Aliyun mirror that offers mirror like Tsinghua University, and Aliyun service which offer hosting/computing service like Amazon AWS

ivankravets commented 2 years ago

@captainstdin , could you try to download this file in the browser https://dl.registry.ns2.platformio.org/tools/34/0c/1bb3ce94604147dd87b7cb891a39637da5624ed991fe6fb9477b373fab2c/toolchain-gccarmnoneeabi-windows_x86-1.50201.0.tar.gz ? What is your speed?

captainstdin commented 2 years ago

@captainstdin , could you try to download this file in the browser https://dl.registry.ns2.platformio.org/tools/34/0c/1bb3ce94604147dd87b7cb891a39637da5624ed991fe6fb9477b373fab2c/toolchain-gccarmnoneeabi-windows_x86-1.50201.0.tar.gz ? What is your speed?

4.4M/s with (VPN,v2ray)，but，This error occurs when I use vpn

424778940z commented 2 years ago

https://dl.registry.ns2.platformio.org/tools/34/0c/1bb3ce94604147dd87b7cb891a39637da5624ed991fe6fb9477b373fab2c/toolchain-gccarmnoneeabi-windows_x86-1.50201.0.tar.gz

I got 3M without vpn, @captainstdin are you using China Telecom / China Mobile?

captainstdin commented 2 years ago

https://dl.registry.ns2.platformio.org/tools/34/0c/1bb3ce94604147dd87b7cb891a39637da5624ed991fe6fb9477b373fab2c/toolchain-gccarmnoneeabi-windows_x86-1.50201.0.tar.gz

I got 3M without vpn, @captainstdin are you using China Telecom / China Mobile?

Telcom ,obvious

如图

If I don't use vpn, I won't be able to download

platformio / platformio-core