Older versions of curl are unable to download large files using newer SSL/HTTP technologies

lorentedford commented 5 years ago

User Story

As a [user description], I want [desired action] so that [desired outcome]. https://linuxgsm.com/lgsm/cod4server/

Basic info

Distro: [Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-22-generic x86_64)]
Game: [COD4]
Command: [install]

Further Information

A clear description of what the bug is and any ideas on how to resolve it. Plus any further context that might be relevant to the issue.

To Reproduce

add user follow directions on website and type ./cod4server install This is where the error occurs. curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2) downloading cod4x18_1772_dedrun.tar.bz2...OK verifying cod4x18_1772_dedrun.tar.bz2 with MD5...FAIL cod4x18_1772_dedrun.tar.bz2 returned MD5 checksum: 809b16e34396d6ef588d28748962dc3f expected MD5 checksum: de29f29d79f9cc24574b838daa501e46 fetching core_exit.sh...OK

Here is logs:

./cod4server install

cod4server@games:~$ ./cod4server install
    fetching core_functions.sh...OK
    fetching core_legacy.sh...OK
    fetching core_messages.sh...OK
    fetching core_dl.sh...OK
    fetching core_trap.sh...OK
    fetching _default.cfg...OK
    copying _default.cfg...OK
    fetching common.cfg...OK
    fetching cod4server.cfg...OK
    fetching linuxgsm.sh...OK
    fetching core_getopt.sh...OK
    fetching command_install.sh...OK
    fetching check.sh...OK
    fetching check_tmuxception.sh...OK
    fetching check_permissions.sh...OK
    fetching check_glibc.sh...OK
    fetching info_distro.sh...OK
    fetching check_system_requirements.sh...OK
    fetching install_header.sh...OK

                                mdMMMMbm
                              mMMMMMMMMMMm
                              mMMMMMMMMMMMMm
                             mMMMMMMMMMMMMMMm
                             hMMMV^VMMV^VMMMh
                             MMMMM  MM  MMMMM
                             hMMs   vv   sMMh
                            hMMM:        :MMMh
                          .hMMMh          hMMMh.
                         -dMMMh     __     hMMMd-
                        :mMMMs      ||      sMMMm:
                       :MMMM+       || _     +NMMN:
                      .mMMM+     ========     +MMMm.
                      yMMMy   ##############   yMMMy
                      mMMM:   ##############   :MMMm
                      mMM   nn   nn    nn   nn   MMm
                      o   nNNNNNNNn    nNNNNNNNn   o
                         nNNNNNNNNNn  nNNNNNNNNNn
                        nNNNNNNNNNNN  NNNNNNNNNNNn
                         +NNNNNNNNN:  :NNNNNNNNN+
                           nNNNNNNN /\ NNNNNNNn
                             nnnnn  db  nnnnn

888      d8b                             .d8888b.   .d8888b.  888b     d888
888      Y8P                            d88P  Y88b d88P  Y88b 8888b   d8888
888                                     888    888 Y88b.      88888b.d88888
888      888 88888b.  888  888 888  888 888          Y888b.   888Y88888P888
888      888 888  88b 888  888  Y8bd8P  888  88888      Y88b. 888 Y888P 888
888      888 888  888 888  888   X88K   888    888        888 888  Y8P  888
888      888 888  888 Y88b 88Y .d8pq8b. Y88b  d88P Y88b  d88P 888   *   888
LinuxGSM 888 888  888  Y8888Y  888  888   Y2012P88   Y8888P   888       888

=================================
LinuxGSM_
by Daniel Gibbs
Game:Call of Duty 4
Website: https://linuxgsm.com
Contributors: https://linuxgsm.com/contrib
Donate: https://linuxgsm.com/donate
=================================
    fetching install_server_dir.sh...OK

Server Directory
=================================
/home/cod4server

Continue? [Y/n] Y
mkdir: created directory '/home/cod4server/serverfiles'
    fetching install_logs.sh...OK

Creating log directories
=================================
installing log dir: /home/cod4server/log...OK
installing LinuxGSM log dir: /home/cod4server/log/script...OK
creating LinuxGSM log: /home/cod4server/log/script/cod4server-script.log...OK
installing console log dir: /home/cod4server/log/console...OK
creating console log: /home/cod4server/log/console/cod4server-console.log...OK
installing game log dir: /home/cod4server/serverfiles/Logs...OK
creating symlink to game log dir: /home/cod4server/log/server -> /home/cod4server/serverfiles/Logs...OK
    fetching check_deps.sh...OK

Checking Dependencies
=================================
tmux
wget
ca-certificates
file
bsdmainutils
util-linux
python3
tar
bzip2
gzip
unzip
binutils
bc
jq
lib32gcc1
libstdc++6:i386
Information! Required dependencies already installed
    fetching install_server_files.sh...OK

Installing Call of Duty 4 Server
=================================
                                          ##O=#   #                                                                                                                  ##########################                                                                                                                  19.4%
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
downloading cod4x18_1772_dedrun.tar.bz2...OK
verifying cod4x18_1772_dedrun.tar.bz2 with MD5...FAIL
cod4x18_1772_dedrun.tar.bz2 returned MD5 checksum: 809b16e34396d6ef588d28748962dc3f
expected MD5 checksum: de29f29d79f9cc24574b838daa501e46
    fetching core_exit.sh...OK

End of Logs

Steps to reproduce the behaviour:

After fresh user created.
su - NewUsername
wget -O linuxgsm.sh https://linuxgsm.sh && chmod +x linuxgsm.sh && bash linuxgsm.sh cod4server
./cod4server install
Error occurs through the installation.

Expected behaviour

Installation of cod4.

issue-label-bot[bot] commented 5 years ago

Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.

dgibbs64 commented 5 years ago

Looks like the download didnt complete so you have a corrupted download. you will need to delete the download and try again

lorentedford commented 5 years ago

Looks like the download didnt complete so you have a corrupted download. you will need to delete the download and try again

I have done this twice.. Looks like its an installation script issue.

dgibbs64 commented 5 years ago

Just tested this myself with no issues. The download is large at 4GB in size. The curl error shows the download failed at 19%. The download simply didn't complete.

                                          ##O=#   #                                                                                                                  ##########################                                                                                                                  19.4%
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

Here are my results.

Installing Call of Duty 4 Server
=================================
################################################################################################################################################################################################################################# 100.0%
downloading cod4x18_1772_dedrun.tar.bz2...OK
verifying cod4x18_1772_dedrun.tar.bz2 with MD5...OK

lorentedford commented 5 years ago

Trust me this isn't adding up.

verifying cod4x18_1772_dedrun.tar.bz2 with MD5...FAIL
cod4x18_1772_dedrun.tar.bz2 returned MD5 checksum: 809b16e34396d6ef588d28748962dc3f
expected MD5 checksum: de29f29d79f9cc24574b838daa501e46

dgibbs64 commented 5 years ago

Your error clearly states the download failed at 19.4%. The md5 sum WILL fail with a file that's only 19% downloaded. For whatever reason, the connection dropped and the download failed

Here is your original error see where it stated 19.4%

##########################19.4%
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
downloading cod4x18_1772_dedrun.tar.bz2...OK
verifying cod4x18_1772_dedrun.tar.bz2 with MD5...FAIL
cod4x18_1772_dedrun.tar.bz2 returned MD5 checksum: 809b16e34396d6ef588d28748962dc3f
expected MD5 checksum: de29f29d79f9cc24574b838daa501e46
    fetching core_exit.sh...OK

lorentedford commented 5 years ago

Clearly it has to be the server where its being downloaded possibly?

https://youtu.be/lNE2Hbw-9l8

YouTube
Linux GSM Failed to install Cod4 fresh installation nothing in user folder Ubuntu 18.04LTS Server.
This is in response to a bug i posted with the Linuxgsm community the user didn't believe me that I was working with a fresh user account and installation. I...

dgibbs64 commented 5 years ago

My monitoring software has reported no outages or downtime for over a week. I host all the files on my dedicated server. I have tested with my own test servers with no issues. It appears that your server connection is dropping the download for some reason. Is the server hosted on a local network or the internet? As it might be a poor connection. Here is the url if you want to download the using a browser https://files.linuxgsm.com/

Index of /

lorentedford commented 5 years ago

The server is hosted in a data center environment even I have to connect to it from the outside.. We are doing a full fresh re-installation of this box. At this time I will keep you posted.

lorentedford commented 5 years ago

Even with a fresh installation of Ubuntu 18.04 LTS this fails

#############################################                                                                                                                                                                                        20.0%
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
downloading cod4x18_1772_dedrun.tar.bz2...OK
verifying cod4x18_1772_dedrun.tar.bz2 with MD5...FAIL
cod4x18_1772_dedrun.tar.bz2 returned MD5 checksum: 4ba4c1d95d5612e0e3c3b3851a2480e8
expected MD5 checksum: de29f29d79f9cc24574b838daa501e46
    fetching core_exit.sh...OK

Oddest thing ever.

lorentedford commented 5 years ago

So I have tried installing this on my server at home at my home network which i know is up and working correctly I got this...

curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
downloading cod4x18_1772_dedrun.tar.bz2...OK
verifying cod4x18_1772_dedrun.tar.bz2 with MD5...FAIL
cod4x18_1772_dedrun.tar.bz2 returned MD5 checksum: c63d303be76f8aebdba4258865525199
expected MD5 checksum: de29f29d79f9cc24574b838daa501e46
    fetching core_exit.sh...OK
cod4server@server3:~$

I have now verified that this is an issue with either your connection where the files are located or an issue with some thing posted on on your cod 4 installation page maybe its downloading an outdated version of linuxgsm when i run the commands idk..

Here is the directions i have followed. https://linuxgsm.com/lgsm/cod4server/

Here is the video of the dedicated box re installation from a fresh install Its at 2x the speed for you.

https://youtu.be/AsftEIomanU

LinuxGSM_
Call of Duty 4 Linux Game Server - LinuxGSM
LinuxGSM is the command-line tool for quick, simple deployment and management of Linux Call of Duty 4 dedicated game servers.

YouTube
Format Reinstall of the Ubuntu 18.04LTs box still failed to install cod4 linuxgsm
This is a full format reinstall of an Ubuntu 18.04LTs box still failed to install cod4 linuxgsm.. Here is where the posting has taken place. https://github.c...

UltimateByte commented 5 years ago

@lorentedford The issue here is rather a network issue than anything else. What about trying with iptables off? Also, what server provider are you using and which country is your server located in? Maybe runing some "ping" commands to the file hosting server would provide us some useful info.

dgibbs64 commented 5 years ago

Potential issue related to fail2ban blocking Cloudflare IP addresses. I have made a change. Please test again.

lorentedford commented 5 years ago

@lorentedford The issue here is rather a network issue than anything else. What about trying with iptables off? Also, what server provider are you using and which country is your server located in? Maybe runing some "ping" commands to the file hosting server would provide us some useful info.

Well since this happened with my home data center servers I this is not the issue. Besides iptables wouldn't affect downloads by default even on a fresh install.. You would have to program that into iptables..

Now..

Potential issue related to fail2ban blocking Cloudflare IP addresses. I have made a change. Please test again.

No this didn't work on a fresh install again. I have dedicated business class fiber with five static ip's at my data center here in my basement where I actually host web and email servers along with many other gamer servers that run linuxgsm with out any issues. I also play with asterisk phone systems and many other projects in the amateur radio community.. I can assure you this is a download issue from your servers.. Especially since I have verified this on an outside data center that is not at or near my network.

Here are my static ip ranges. 157.185.79.34 to 157.185.79.38. According to pf-sense its pulling from 157.185.79.38 non of my other projects have any issues and my download stream has been solid over the past 24 hrs to our gateway.

Here is what Pfsense is showing on traffic information over the past 8 hours. From what i can see on the router side is that some thing from the download server side disconnects us.

lorentedford commented 5 years ago

Potential issue related to fail2ban blocking Cloudflare IP addresses. I have made a change. Please test again.

Please read my above posts and i am off to bed will be up anywhere between 6 to 8 hours from now I will check my email through out my sleep to see If i get another notice from github of reply on this.. Chat in the morning Best Wishes

dgibbs64 commented 5 years ago

Just to be clear. I have identified a potential issue on my server. It appears fail2ban is blocking Cloudflare IP addresses (instead of the offending bad IP address) This might cause connections to be dropped. I have disabled the fail2ban jail that is causing this so the issue might be resolved. Please test again to see if this has helped.

A potential solution that I will investigate later to allow log files to see original visitor IPs addresses that will allow fail2ban to work correctly and not block Cloudflare IP ranges

https://support.cloudflare.com/hc/en-us/articles/200170786-Restoring-original-visitor-IPs-Logging-visitor-IP-addresses-with-mod-cloudflare-

lorentedford commented 5 years ago

Just to be clear. I have identified a potential issue on my server. It appears fail2ban is blocking Cloudflare IP addresses (instead of the offending bad IP address) This might cause connections to be dropped. I have disabled the fail2ban jail that is causing this so the issue might be resolved. Please test again to see if this has helped.

A potential solution that I will investigate later to allow log files to see original visitor IPs addresses that will allow fail2ban to work correctly and not block Cloudflare IP ranges

https://support.cloudflare.com/hc/en-us/articles/200170786-Restoring-original-visitor-IPs-Logging-visitor-IP-addresses-with-mod-cloudflare-

Nope Its still denying it.. I do have to wipe the directory and start fresh each time when I go to install..

Here is a screen shot..

Cloudflare Support
Restoring original visitor IPs: Logging visitor IP addresses with mod_cloudflare
Learn how to configure mod_cloudflare to log your visitor’s original IP address based on your origin web server type (including Apache, nginx, Microsoft IIS and others). Cloudflare no longer update...

h3o66 commented 5 years ago

some time ago i tried to do a "hack" to do a auto resume of the download.

https://github.com/h3o66/LinuxGSM/commit/ff48b8dee686a30dea06af7967980e2bb4ec4884

Do you think this could help as addition for lgsm ?

dgibbs64 commented 5 years ago

@h3o66 yeah something ensure that the download retries would be good. A potential solution will need to be investigated. I will be doing maintenance and testing to make sure that the potential solution resolves the problem.

lorentedford commented 5 years ago

@h3o66 yeah something ensure that the download retries would be good. A potential solution will need to be investigated. I will be doing maintenance and testing to make sure that the potential solution resolves the problem.

A linode server is like 5 dollars a month to test it on.. Just saying.. I wouldn't use it to necessarily host the server on but installation wise it should work and tell you if you found out why the server doesn't like to download and install correctly.

dgibbs64 commented 5 years ago

@lorentedford Linode already sponsor me with a free account ;-).

lorentedford commented 5 years ago

@lorentedford Linode already sponsor me with a free account ;-).

Well there ya go :) So what's wrong with the downloading from your server? Do you have any ideas yet?

lorentedford commented 5 years ago

Keep in mind i have tried on several systems here in our network and at the data center network and still no go on downloading and installation its so odd...

lorentedford commented 5 years ago

Are my ip's blacklisted?

dgibbs64 commented 5 years ago

Yes I'm working on a fix as we speak. I think some Cloudflare's IP are being blocked by my server which users proxy though

lorentedford commented 5 years ago

Yeah personally I haven't been a fan of cloud flare services yet.. Just my thoughts.

dgibbs64 commented 5 years ago

Right, I think things might have settled down. I have reconfigured fail2ban and it now correctly sends bans to the cloudflare firewall. This will have to be monitored.

lorentedford commented 5 years ago

Well this server on my network failed.. Here are my static ip ranges. 157.185.79.34 to 157.185.79.38.

And the one at the data center eventually failed as well at 29%

dgibbs64 commented 5 years ago

Found the request in the logs and it's not being blocked. Even shows me how far the download got. I no longer think the issue is with fail2ban blocking anything. I'm stumped with this one currently, especially since I can't replicate this.

Whats possibly non-standard with your server is you are using link aggregation across 4 NIC's. Maybe the file transfer has an issue with that...but I'm just guessing as it has been a while since I have used such a configuration. Possibly doesn't like the particular bonding mode as there are 7 different modes. :man_shrugging:

https://www.interserver.net/tips/kb/network-bonding-types-network-bonding/

Interserver Tips
What is Network Bonding? Types of Network Bonding
Network bonding is a process of combing or joining two or more network interfaces together into a single interface. Network bonding offers performance improvements and redundancy by increasing the network throughput and bandwidth. If one interface is down or unplugged the other one will work. It can be used in situations where you need fault

lorentedford commented 5 years ago

Let m boot up a linode really quick and try one from linode and see what happens.. Again if its not working on my personal data center network or the dedicated box and you can see where both ips pulled for the data example my server from my personal data center should have been pulling from 157.185.79.38 and not sure on the data center side.. Can you tell me if you see the 157.185.79.38? Why does the server drop this connection?

lorentedford commented 5 years ago

I created a linode vps really quickly to try out your theory that it might be on both of my networks this was spun up literally like 10 minutes ago as stated and it failed.. Here is the ip of my linode server.

198.58.123.110 and 2600:3c00::f03c:92ff:feb0:3d0e

This re affirms my theory that its on your network or your server connections.

See if there is any logs of this ip trying to connect up to your server.

dgibbs64 commented 5 years ago

Possible lead. Tested on 7 different distros and the download only worked on newer versions of curl, researching found a possible issue with curl using certain types https encryption. Bed for me now but I will continue investigating this. I think I will have to tweak https settings for compatibility.

CentOS 8 - curl 7.61.1 - PASS
CentOS 7 - curl 7.29.0 - FAIL
Debian 10 - curl 7.64.0 - PASS
Debian 9 - curl 7.52.1 - FAIL
Debian 8 - curl 7.38.0 - FAIL
Ubuntu 18.04 - curl 7.58.0 - FAIL
Ubuntu 16.04 - curl 7.47.0 - FAIL

markie83 commented 5 years ago

probobly has outdated Certificate Authorities in the older versions of cURL......try it with a newer version of cURL from backport repos.

markie83 commented 5 years ago

ok turns out there is not a backport for cURL on 18.04 so I manually added the ppa for 18.10

deb http://ppa.launchpad.net/xapienz/curl34/ubuntu cosmic main deb-src http://ppa.launchpad.net/xapienz/curl34/ubuntu cosmic main

and then installed curl from there and then it worked.........not something I really wanted to do in a production environment. Ubuntu REALLY needs to do a backport for 18.04 LTS

Index of /xapienz/curl34/ubuntu

lorentedford commented 5 years ago

ok turns out there is not a backport for cURL on 18.04 so I manually added the ppa for 18.10

deb http://ppa.launchpad.net/xapienz/curl34/ubuntu cosmic main deb-src http://ppa.launchpad.net/xapienz/curl34/ubuntu cosmic main

and then installed curl from there and then it worked.........not something I really wanted to do in a production environment. Ubuntu REALLY needs to do a backport for 18.04 LTS

Index of /xapienz/curl34/ubuntu

Yes this actually worked Waiting for @dgibbs64 to respond before i do this i don't want to mess this up on my friends server that I am installing for him..

Index of /xapienz/curl34/ubuntu

dgibbs64 commented 5 years ago

Yup updating curl would be a good fix for this. I will, however, look at making changes to my webserver to ensure compatibility with older versions of curl.

lorentedford commented 5 years ago

Yup updating curl would be a good fix for this. I will, however, look at making changes to my webserver to ensure compatibility with older versions of curl.

Me and Markie talked about this last night this should have been back ported because this is going to cause issues down the line.. Markie is my boyfriend I brought him over to my case because I wanted to make sure I wasn't missing anything..

dgibbs64 commented 5 years ago

I agree curl should be backported. Especially with the speed that new web technologies and certificate authorities are changing. Looking through the settings I think the easiest and most reliable option is to revert to HTTP. I would prefer to use https but since there is no sensitive data being transferred I think it makes sense in this case.

So, in summary, the quick fix is to upgrade curl to a newer version. On my end, I will make changes to move to HTTP, which I will make ready for the next release.

lorentedford commented 4 years ago

Well now we are back to this again...

lorentedford commented 4 years ago

@dgibbs64 What did you do from yesterday?? Because now I can't download the files either way..

Just trying to figure out what happened...

lorentedford commented 4 years ago

Now this is the same server that installed everything just fine yesterday after we modified with the latest curl.

Here is what is happening now after creating a new user and running the commands.

wget -O linuxgsm.sh https://linuxgsm.sh && chmod +x linuxgsm.sh && bash linuxgsm.sh cod4server

./cod4server install

lorentedford commented 4 years ago

Ok so where does that leave us?

Frisasky commented 4 years ago

i think its related to cloudflare's compatibility with older software, try turn off http2 and tls 1.3 in CF's website config

lorentedford commented 4 years ago

Hey this problem still exist 12-1-2019

Why is this closed????

Frisasky commented 4 years ago

did u updated to latest version of lgsm?

lorentedford commented 4 years ago

Yes

lorentedford commented 4 years ago

did u updated to latest version of lgsm?

Did you update your website??

Here is my reinstall script since i got tired of making user accounts..

#!/bin/sh
cd ~
rm -R ./cod4server ./lgsm/ ./linuxgsm.sh ./log/ ./serverfiles/
sleep 2s
wget -O linuxgsm.sh https://linuxgsm.sh && chmod +x linuxgsm.sh && bash linuxgsm.sh cod4server
sleep 4s
./cod4server install

I don't appreciate being treated like i don't know shit.. I get it working one time give you guys the information then you all break it.. Oh then close the ticket after I have been waiting patiently for over a week..

Here is what i am going off of... https://linuxgsm.com/lgsm/cod4server/

It would have been nice if you guys tagged me in my own ticket and asked me if the issue was fully resolved instead of just assuming that the problem is fixed and moving on..

Frisasky commented 4 years ago

kindly asking can you join our discord server so we could perform better diagnostic https://linuxgsm.com/discord

Discord
Join the LinuxGSM_ Discord Server!
Check out the LinuxGSM_ community on Discord - hang out with 1,674 other members and enjoy free voice and text chat.

lorentedford commented 4 years ago

kindly asking can you join our discord server so we could perform better diagnostic https://linuxgsm.com/discord

DiscordJoin the LinuxGSM_ Discord Server!Check out the LinuxGSM_ community on Discord - hang out with 1,674 other members and enjoy free voice and text chat.

I am in the general chat and waiting on a video to finish rendering with my fresh install https://youtu.be/re3enYFImjc

Discord
Join the LinuxGSM_ Discord Server!
Check out the LinuxGSM_ community on Discord - hang out with 1,674 other members and enjoy free voice and text chat.

YouTube
LinuxGSM closes my issues ticket with out even checking with me.. :(
So I have been battling it out with linuxgsm at this point my ticket has been open for over 13 days and I had a solution until they broke some thing back end...

dgibbs64 commented 4 years ago

Further testing is ongoing with this issue. Using http has helped improve reliability but is not currently working all the time. I have found a way for curl to continue if the connection drops for any reason so I will be adding that feature to the curl command. Testing is ongoing to confirm if cloud flare is causing a connection issue or not but is very hard to confirm either way.

GameServerManagers / LinuxGSM