search

pyamsoft / tetherfi

TetherFi - Internet sharing without Root
Apache License 2.0
276 stars 24 forks source link

Inconsistent HTTP request response on Linux software repository mirrors when using vs. not-using TetherFi #280

Closed brlin-tw closed 1 month ago

brlin-tw commented 3 months ago

I recently found that using TetherFi to fetch specific resources from some Linux distribution software archives (mirrors) will generate spurious HTTP 404 Not found errors:

brlin@brlin-fw13:~$ sudo docker run --env http_proxy --env https_proxy --rm -it centos:7
[root@abbda6318740 /]# yum makecache
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: centos.ccns.ncku.edu.tw
 * extras: centos.ccns.ncku.edu.tw
 * updates: centos.ccns.ncku.edu.tw
http://centos.ccns.ncku.edu.tw/centos/7.9.2009/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.

   ...stripped...

failure: repodata/repomd.xml from base: [Errno 256] No more mirrors to try.
http://centos.ccns.ncku.edu.tw/centos/7.9.2009/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
http://ftp.ksu.edu.tw/pub/CentOS/7.9.2009/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
http://ftp.sjtu.edu.cn/centos/7.9.2009/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found

   ...stripped...

http://mirrors.cqu.edu.cn/CentOS/7.9.2009/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
http://mirrors.qlu.edu.cn/centos/7.9.2009/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
http://mirrors.ustc.edu.cn/centos/7.9.2009/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found

I originally thought that it was the ISP who implemented faulty transparent proxies in between the connection that caches the erratic response however the problem is no longer reproduced after I unset the proxy settings and use the native USB tethering network sharing:

[root@abbda6318740 /]# unset http_proxy https_proxy
[root@abbda6318740 /]# yum makecache
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: centos.ccns.ncku.edu.tw
 * extras: centos.ccns.ncku.edu.tw
 * updates: centos.ccns.ncku.edu.tw
base                                                                                                | 3.6 kB  00:00:00     
extras                                                                                              | 2.9 kB  00:00:00     
updates                                                                                             | 2.9 kB  00:00:00     
(1/10): base/7/x86_64/group_gz                                                                      | 153 kB  00:00:00     
(2/10): extras/7/x86_64/filelists_db                                                                | 303 kB  00:00:00     
(3/10): extras/7/x86_64/other_db                                                                    | 150 kB  00:00:00     
(4/10): extras/7/x86_64/primary_db                                                                  | 250 kB  00:00:00     
(5/10): base/7/x86_64/other_db                                                                      | 2.6 MB  00:00:01     
(6/10): updates/7/x86_64/other_db                                                                   | 1.5 MB  00:00:01     
(7/10): base/7/x86_64/filelists_db                                                                  | 7.2 MB  00:00:02     
(8/10): updates/7/x86_64/filelists_db                                                               |  14 MB  00:00:03     
(9/10): base/7/x86_64/primary_db                                                                    | 6.1 MB  00:00:06     
(10/10): updates/7/x86_64/primary_db                                                                |  25 MB  00:00:21     
Metadata Cache Created

The issue can also be reproduced when I use the web browser to access the same URL(left: w/o TetherFi; right: w/ TetherFi):

Screenshot of the comparison of the HTTP request and responses

I wonder whether there's any difference in the HTTP requests that are generated w/, w/o TetherFi that may trigger the different result?

brlin-tw commented 3 months ago

Debug log with excess "Proxy Request" lines stripped: debug.txt

pyamsoft commented 3 months ago

I will look. TetherFi generally speaking does not change requests, but this may be sending extra data over as part of the HTTP connect step.

I will attempt to reproduce the issue on my end first.

pyamsoft commented 3 months ago

Hi,

Testing against this URL: https://mirror01.idc.hinet.net/centos/7.9.2009/os/x86_64/repodata/repomd.xml using Firefox 123 on ArchLinux and MacOS, I was not able to recreate the 404.

While the page certainly loads slightly slower because of the proxy, I was able to get the XML file to load using both the latest main branch and the release build 40

Are you consistently able to reproduce the 404? Could there be something else at play like the server blocking an IP address, or some network configuration on your Android device preventing the connection?

brlin-tw commented 1 month ago

@pyamsoft

First of all, apologies for the ignorance.

Are you consistently able to reproduce the 404?

Mostly yes, there is a minor case where the download will temporarily succeed after several refresh attempts on the web browser, however, most access does return 404.

Could there be something else at play like the server blocking an IP address

Nothing I would know of.

some network configuration on your Android device preventing the connection?

Not something I can think of.

pyamsoft commented 1 month ago

Are you sure that yum in the container is using the http_proxy variables, and that the proxy is set?

If you curl the file outside of docker (with the proxy env set) does it work?

trying to isolate if this is a "docker network namespace" thing, or a "proxy http in the terminal" thing, thanks!

brlin-tw commented 1 month ago

@pyamsoft

Are you sure that yum in the container is using the http_proxy variables, and that the proxy is set?

Absolutely:

Screenshot with the top pane has the proxy variables set but not the bottom pane

If you curl the file outside of docker (with the proxy env set) does it work?

Nope:

Screenshot

brlin-tw commented 1 month ago

I have manually built and installed version 41 from the source, and the problem is no longer reproduced.

Not sure what exactly fixes the issue, though, will it be helpful if I do a git bisect to locate the revision that changes the result?

brlin-tw commented 1 month ago

All the HTTP requests that can readily reproduce the problem using the old version:

Not that it is insightful or anything.

brlin-tw commented 1 month ago

All the HTTP requests that can readily reproduce the problem using the old version

I can verify that all these URLs can be successfully downloaded using the 41 version.

pyamsoft commented 1 month ago

A slight change was made to how TCP connections were exchanged in 41. Maybe this has also fixed your problem?

Either way, its good to hear that 41 seems to have resolved the issue. I hope in the next few days Google will approve the build so it can release on the Play Store.

pyamsoft commented 1 month ago

Can this be closed since the issue seems to be fixed in version 41?

brlin-tw commented 1 month ago

Can this be closed since the issue seems to be fixed in version 41?

Yes, certainly.