Chinese mirror of the package registry

chubbyginger commented 2 years ago

What kind of issue is this?

[x] Feature Request. Start by telling us what problem you’re trying to solve. Often a solution already exists! Don’t send pull requests to implement new features without first getting our support. Sometimes we leave features out on purpose to keep the project small.

Configuration

Operating system: Windows 10 Home 21H2 64-bit

PlatformIO Version (platformio --version): PlatformIO Core, version 6.0.2

Description of problem

I am a PlatformIO user from China. It is all known to devs in China that there is a You-Know-What system that will cause accessing github.com becoming very, very, very slow or even unreachable (ERR_CONNECTION_RESET).

When I create a PlatformIO project, it will download a lot of data, including toolchains, SDKs, etc from github or some places like that. Thus the process is extremely slow and it takes hours to install that. In many situations there is completely no network activity.

internet

I think it is appropriate for PlatformIO developers to add a import function which can import pre-downloaded toolchains, SDKs, etc (NOT the ones downloaded from the "Platform" page. Seems that it is only a project framework.). There should also be a place to manually download the files, so that I can call my friends to help me download them.

Steps to Reproduce

Create a new project.

Actual Results

The project wizard stuck there showing

Please wait...
The first initialization requires Internet connection and may take a while (need to install dependent toolchains, frameworks, SDKs).
Please be patient and let the initialization complete.

Expected Results

Project can be created successfully

ps. My board: Espressif ESP32 Devkit

ivankravets commented 1 year ago

@cainiaoxiao , do you use any firewall that blocks 122.128.106.144 ?

cainiaoxiao commented 1 year ago

@ivankravets no,but sometime early it could use,althought the speed not fast ,about several hundreds KB/s,I think a better way is user can change platformIO's CDN servers manual or provide a way to manual download these packages,maybe you can use CI/CD system auto push packages to github? but maybe that chould make repository mess and this packages too big,but I think this maybe help to resolve these relative problem.

ivankravets commented 1 year ago

A temporary workaround to skip/disable some mirrors is to point PlatformIO CDN hosts in your operating system to 127.0.0.1.

If you are on Windows, see https://www.nublue.co.uk/guides/edit-hosts-file/

For example, to skip the Chinese mirror ( dl.registry.ns3.platformio.org ), please add the following line:

127.0.0.1    dl.registry.ns3.platformio.org

Did it help you?

cainiaoxiao commented 1 year ago

A temporary workaround to skip/disable some mirrors is to point PlatformIO CDN hosts in your operating system to 127.0.0.1.

If you are on Windows, see https://www.nublue.co.uk/guides/edit-hosts-file/

For example, to skip the Chinese mirror ( dl.registry.ns3.platformio.org ), please add the following line:
127.0.0.1    dl.registry.ns3.platformio.org
Did it help you?

all right, I use linux and I maybe search a way to configure hosts in my system by myself when the problem appear again.Thanks a lot!

ivankravets commented 1 year ago

@CyberFoxHax, could you try this URL https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz ? Does it work for you? This is a Chinese mirror.

elysia-best commented 1 year ago

@CyberFoxHax, could you try this URL https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz ? Does it work for you? This is a Chinese mirror.

I've tried this link in Chengdu with China Telecom. Unfortunately, the downloads speed (direct access) seems to be very slow. It will reach approximately 1M Byte/s but quickly falls to around 20 K Bytes/s. However, my ISP claims that my theoretically maximum download speed is 1000M bits/s though.

P.S. I also recommend to contact universities in China to see whether they have the interest in serving the mirror. They have provided many commonly used repos like Linux-distros, pypi etc.

For example, Peking University's site has around 20T free space and can email to mirror@pku.edu.cn or create a pr in https://github.com/PKUOSC/issues/issues to ask for there opinion

ivankravets commented 1 year ago

Could you try Github? https://github.com/winpython/winpython/releases/download/6.4.20230625final/Winpython64-3.11.4.0dot.exe

What is your download speed for the URL above?

elysia-best commented 1 year ago

Could you try Github? https://github.com/winpython/winpython/releases/download/6.4.20230625final/Winpython64-3.11.4.0dot.exe

What is your download speed for the URL above?

Unfortunately，it's the same form me. About 20 kb/s.

ivankravets commented 1 year ago

@villivateur , could you try https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz without proxy? What is your ISP?

villivateur commented 1 year ago

@ivankravets Thank you for your reply, Here is the result:

$ wget https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz
--2023-09-14 09:03:00--  https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz
Resolving dl.registry.ns3.platformio.org (dl.registry.ns3.platformio.org)... 122.128.106.144
Connecting to dl.registry.ns3.platformio.org (dl.registry.ns3.platformio.org)|122.128.106.144|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 61429219 (59M) [application/octet-stream]
Saving to: ‘framework-espidf-3.40401.0.tar.gz’

framework-espidf-3.40401.0.tar.gz                    100%[====================================================================================================================>]  58.58M   669KB/s    in 2m 3s

2023-09-14 09:05:03 (488 KB/s) - ‘framework-espidf-3.40401.0.tar.gz’ saved [61429219/61429219]

My ISP is China Unicom.

However, my bandwidth is 1000Mbps. And the download speed from platformio may vary. For example, on morning, it's speed is 500KB/s, but at night, it is only 50KB/s.

BTW, could you tell me how platformio choose CDN domains? I wonder why my PlatformIO use dl.registry.ns3.platformio.org but not dl.registry.ns1.platformio.org. Thank you.

ivankravets commented 1 year ago

@villivateur, PlatformIO CDN picks the best mirror based on your IP (usually country detection). If you have problems with NS3 mirror, you can try to add this line to your hosts file

127.0.0.1      dl.registry.ns3.platformio.org

wc7086 commented 11 months ago

@ivankravets Is it possible to synchronize registry repository using the rsync protocol (or a dedicated synchronization tool)? If it is possible to synchronize using the rsync protocol (or a dedicated synchronization tool) and you agree, I can request that https://mirrors.tuna.tsinghua.edu.cn/ mirrors the registry repository, their mirroring service is completely free.

The reason I'm here is that I waited a whole day without successfully downloading the first package.

(PS: Whether or not it is illegal to use proxy software to connect to the Internet in China depends on the attitude of local law enforcement officials towards this matter.)

villivateur commented 11 months ago

@ivankravets Is it possible to synchronize registry repository using the rsync protocol (or a dedicated synchronization tool)? If it is possible to synchronize using the rsync protocol (or a dedicated synchronization tool) and you agree, I can request that https://mirrors.tuna.tsinghua.edu.cn/ mirrors the registry repository, their mirroring service is completely free.

The reason I'm here is that I waited a whole day without successfully downloading the first package.

(PS: Whether or not it is illegal to use proxy software to connect to the Internet in China depends on the attitude of local law enforcement officials towards this matter.)

It should be the best solution. But it seems that platformio's maintainers dislike it and never response to the solution directly.

ivankravets commented 11 months ago

PlatformIO Registry is built on top of its own CDN infrastructure that has support for public and private packages. It is not enough "to copy files".

We need a fully functional server or VPS to deploy a new PlatfomIO Registry's CDN node.

ivankravets commented 11 months ago

@villivateur , do you also have problem with downloading https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz ?

wc7086 commented 11 months ago

@villivateur , do you also have problem with downloading https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz ?

ivankravets commented 11 months ago

@wc7086 , what is your ISP? Could you try without a proxy?

wc7086 commented 11 months ago

PlatformIO Registry is built on top of its own CDN infrastructure that has support for public and private packages. It is not enough "to copy files".

OKAY, I see, platformio's library is similar to dockerhub.

wc7086 commented 11 months ago

@wc7086 , what is your ISP? Could you try without a proxy?

China Telecom, no proxy is used.

Most people do not have access to legitimate proxy software, so it is not always possible to use proxies for downloads.

Penalties for illegal proxies are entirely at the discretion of local authorities, so many people need an avenue to download dev kits without using proxies.

https://www.theguardian.com/world/2023/oct/09/chinese-programmer-ordered-to-pay-1m-yuan-for-using-virtual-private-network

wc7086 commented 11 months ago

In the future I may try to contact someone who can build a PlatformIO Registry node.

ivankravets commented 11 months ago

@wc7086 , could you share the result of a speed for https://hk03.layerstack.com/50MB.test ?

wc7086 commented 11 months ago

@wc7086 , could you share the result of a speed for https://hk03.layerstack.com/50MB.test ?

ivankravets commented 11 months ago

Could you also check the speed for https://as-speedtest.kamatera.com/ ?

villivateur commented 11 months ago

@villivateur , do you also have problem with downloading https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz ?

@ivankravets It is useless to test a single URL download speed, because the speed can vary a lot. The ISP, location, IP zones, date & time, and even your luck will affect the download speed, as long as the server is outside China.

ivankravets commented 11 months ago

People in this thread who are located in China please report the speed for this URL: https://dl.registry.ns3.platformio.org/50MB.test

wc7086 commented 11 months ago

People in this thread who are located in China please report the speed for this URL: https://dl.registry.ns3.platformio.org/50MB.test

ivankravets commented 11 months ago

We have deployed a new PlatformIO Registry CDN mirror to China. It has direct peering with China Unicom, China Mobile & China Telecom.

Anyone in this thread from China please confirm the speed. Thanks!

villivateur commented 11 months ago

We have deployed a new PlatformIO Registry CDN mirror to China. It has direct peering with China Unicom, China Mobile & China Telecom.

Anyone in this thread from China please confirm the speed. Thanks!

Wow, thank you very much. Could you please send me a test URL here so I can test with my ISP (China Unicom and China Telecom)?

ivankravets commented 11 months ago

You can check the same URL https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz

villivateur commented 11 months ago

It is still extremely slow. 80kB/s.

Jiangsu China Unicom, just now.

wc7086 commented 11 months ago

We have deployed a new PlatformIO Registry CDN mirror to China. It has direct peering with China Unicom, China Mobile & China Telecom.

Apart from the independent proprietary lines, it seems that only China Telecom's GIA (https://bgp.he.net/AS4809) provides direct triple play (China Unicom, China Mobile & China Telecom.) connections.

https://dl.registry.ns3.platformio.org

GIA (https://bgp.he.net/AS4809) VPS

ivankravets commented 11 months ago

We have contacted LayerStack support and asked them to explain the reason for the slow download speed for Chinese customers.

ivankravets commented 11 months ago

@villivateur , @wc7086 , please send me email to me@ikravets.com. I'll need more details (network configuration, etc) from your side. It is better to share private data via email instead of Github. Thanks in advance!

bsdayo commented 11 months ago

You can check the same URL https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz

~50KB/s download speed without using proxy. My ISP is China Mobile, and I'm located in Shenzhen.

ivankravets commented 9 months ago

Sorry for bothering you. Could I ask people in this thread from China to re-test the download speed for this URL https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz ? Please turn off any proxies.

villivateur commented 9 months ago

@ivankravets Here is the test:

Snipaste_2023-12-30_19-44-22

It was too slow and I cancelled it.

My ISP is China Unicom.

ivankravets commented 9 months ago

Kinchin, could you email me at me@ikravets.com? I want to ask you to provide additional information which is better to keep private (traceroute, etc).

jiladahe1997 commented 9 months ago

Hi @ivankravets , thanks for your efforts in getting Chinese people to use platformio.

I have tried this link:https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar .gz . The speed is very slow, only 20kb/s.

By the way, I have some other suggesstions:

A large part of programmers in China have their own proxy, because we can not even visit github without proxy : ( .
Most proxy is base on Clash or Shadowsocks, which usually provide system level proxy, thus maybe platformio could just use system proxy.
On the other hand, I suggest you use Alibaba Cloud or Huawei Cloud, They are China's largest cloud service provider, providing object storage, static storage and other services.
You can also use Alibaba Cloud's time-based billing server in China (cheap, about < 1 USD/hour) to test download speeds in China If you want build something like continuous-integration to test download speed in China.

ivankravets commented 9 months ago

Hi @jiladahe1997, could you email me at me@ikravets.com? I want to ask you to provide additional information which is better to keep private (traceroute, etc).

wc7086 commented 9 months ago

On the other hand, I suggest you use Alibaba Cloud or Huawei Cloud, They are China's largest cloud service provider, providing object storage, static storage and other services. You can also use Alibaba Cloud's time-based billing server in China (cheap, about < 1 USD/hour) to test download speeds in China If you want build something like continuous-integration to test download speed in China.

这个我跟他说过了，海外公司想用大陆服务器提供内容分发服务应该不容易。

bsdayo commented 9 months ago

其实允许改源也是一种可行的方案，#4781

elysia-best commented 9 months ago

I support adding the auto detection of proxy setting in operating system. This could be a better way to handle a variety of cases.

发自我的 iPhone

在 2023年12月31日，02:11，wc7086 @.***> 写道：

CAUTION: External Message. Use caution opening links and attachments.

On the other hand, I suggest you use Alibaba Cloudhttps://www.alibabacloud.com/ or Huawei Cloudhttps://www.huaweicloud.com/intl/en-us/, They are China's largest cloud service provider, providing object storage, static storage and other services. You can also use Alibaba Cloud's time-based billing server in China (cheap, about < 1 USD/hour) to test download speeds in China If you want build something like continuous-integration to test download speed in China.

这个我跟他说过了，海外公司想用大陆服务器提供内容分发服务应该不容易。

— Reply to this email directly, view it on GitHubhttps://github.com/platformio/platformio-core/issues/4345#issuecomment-1872576908, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJJXE2TUVJHLNUWQEPJTJGDYMBKNXAVCNFSM53BHD642U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBXGI2TONRZGA4A. You are receiving this because you commented.Message ID: @.***>

ivankravets commented 9 months ago

Currently, the download speed should be very good. Could confirm this with https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz without any proxies?

If it is true, we will discuss it with LayerStack (the Chinese server provider) on how to reserve this bandwidth for the future.

elysia-best commented 9 months ago

Yes that is reasonable. It is better to let pio server read settings of server or proxy url from a spectific environment variable.

发自我的 iPhone

在 2023年12月31日，18:18，不是 @.***> 写道：

CAUTION: External Message. Use caution opening links and attachments.

其实允许改源也是一种可行的方案，#4781https://github.com/platformio/platformio-core/issues/4781

— Reply to this email directly, view it on GitHubhttps://github.com/platformio/platformio-core/issues/4345#issuecomment-1872913046, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJJXE2RCYF4OFOIRR7MJ2CDYME3YDAVCNFSM53BHD642U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBXGI4TCMZQGQ3A. You are receiving this because you commented.Message ID: @.***>

jiladahe1997 commented 9 months ago

Currently, the download speed should be very good. Could confirm this with https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz without any proxies?

If it is true, we will discuss it with LayerStack (the Chinese server provider) on how to reserve this bandwidth for the future.

@ivankravets Test just now:

wget https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz 
--2023-12-31 23:32:58--  https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz
Connecting to 192.168.64.1:7890... connected.
Length: 61429219 (59M) [application/octet-stream]
Saving to: ‘framework-espidf-3.40401.0.tar.gz’

01.0.tar.gz           2%[                    ]   1.68M  37.3KB/s    eta 21m 56s^

The GFW(Please google search it) really works in some magical way that makes the network work sometimes and sometimes not, please don't try to challenge it....

villivateur commented 9 months ago

Connecting to 192.168.64.1:7890... connected.

@jiladahe1997 It seems that you are using proxy.

elysia-best commented 9 months ago

Here is my download speed

~# curl https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e7 47e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz -o /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 1 58.5M 1 704k 0 0 33729 0 0:30:21 0:00:21 0:30:00 35327

发自我的 iPhone

在 2023年12月31日，19:50，Ivan Kravets @.***> 写道：

CAUTION: External Message. Use caution opening links and attachments.

Currently, the download speed should be very good. Could confirm this with https://dl.registry.ns3.platformio.org/tools/08/c7/7511ca5c84f8e747e04d6d0021c98d0c8ffef7c588782b0aec5b92ff5691/framework-espidf-3.40401.0.tar.gz without any proxies?

If it is true, we will discuss it with LayerStackhttps://www.layerstack.com/ (the Chinese server provider) on how to reserve this bandwidth for the future.

— Reply to this email directly, view it on GitHubhttps://github.com/platformio/platformio-core/issues/4345#issuecomment-1872928745, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJJXE2XAENNY3RQT63RASO3YMFF5PAVCNFSM53BHD642U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBXGI4TEOBXGQ2Q. You are receiving this because you commented.Message ID: @.***>

wdq4587 commented 7 months ago

All of these download speed tests are meaningless. They don't know what happened, so there are no real solutions!

Download speed is not the real download speed. There is Something continues to send TCP reset package to break you HTTP connection, and after so many brakes, a large file for example more than 100 to 300 MB will eventually fail to download. Even using a 56Kb/s Hayes modem I can download such a file within 20 hours. But sometime you can not download it even take a week (without proxy).
The download program does not support resuming downloading leads the real failure (Not only Python, many of them). About 25 years ago, among the software that could download files, only the browsers does not support resuming downloading (Now some browsers support it). Every download software support it (for example NetVampire), but now all of you think this technical is obsolete and useless.

wdq4587 commented 7 months ago

Continue, many users who responded here they have proxies.There is always a way for them. But for those who without, the real problem is not slow download, but the inability to successfully download large files. This means that updating large packages is impossible.

Every file sources outside the mainland is likely to be blocked, or at least you may assume they will block it.
Smaller file transfers have a higher chance of surviving TCP reset attacks, which is why I am able to post here. For example, files smaller than 30 MB has a success rate of over 50%. One solution is to break large files into smaller files, such as 14 MB in size. It's an acceptable solution especially many of your web servers do not support resuming downloading. But I can't expect you people to do that.

platformio / platformio-core