UltimateByte commented 9 years ago

I always hated that FastDL thing... But things are what they are, and an SRCDS is limitting to 20KB/s while uploading content files from your addons to clients... And today, i can't bypass it using workshop anymore... One of my servers (Gmod DarkRP) needs it right now for some addons, and i don't wish making workshop addons for all of them.

So i'm letting you know that with my very poor knowledge about bash, i just started learning it better and working on a little script to configure FastDL, as i didn't find any other relevant script.

So, my target is to make a script that can :

Copy all needed files to upload to clients from the garrysmod folder into a fastdl folder at root (with gmodserver script from danielgibbs), maintaining the good folder structure
Do the same with the addons folder (which is stucking me for now for the addon part as all folders have a shitty different architecture)
Compress every single file using b2zip into its respective directory (i'll see if can make it so files under a certain size will be ignored).
Eventually, if needed (and if i'm able to do it) it'll make a fastdl_resource.lua file into garrysmod/lua/autorun/server that will - like workshop file - request all the files to be downloaded by the client.

And the reason why i'm telling you this is : I'd like to share my work with you, so maybe you could take some ideas from it and integrate a fn_fastdl :p

dgibbs64 commented 9 years ago

Decent. I have a function similar to this with unreal tournament that compresses all the files and saves them in a directory. The hard part is getting it to the web directory. I will talk about this more I a bit :-)

UltimateByte commented 9 years ago

I'm back on it,i just made a GitHub repository https://github.com/UltimateByte/fastdl

Don't pay attention to the script right now, it's crappy and not functional yet, i'm still working. I'll tell you once it works for me (if it does haha).

dgibbs64 commented 9 years ago

OK let me know when you want me to review it. Also i will be calling the function fn_srcds_fastdl

UltimateByte commented 9 years ago

Thanks Daniel. So my first code is working ! Champagne !

BTW, i don't really know how to use functions like you do, so for now i've just made a standalone script that needs your script's folder's structure to work and to be placed at the same folder lvl than yours. I have to say that i've learned somehow from your code, mostly from the way that you gather inputs or call folders from variables.

Now as this is my first code, it may some dirty code. For example, i'm not sure when i should put "" between a variable that is a folder. Please let me know if i you see things to change or if i can help you in any way if you want to integrate the function to your lgsm.

UltimateByte commented 9 years ago

Limitations to my script :

Some files are copied but are not required by client
There is nothing to create a fastdl_resource.lua file into garrysmod/lua/autorun/server that will request all the files to be downloaded by the client, but this could be an advantage.

Explaination for this second point. At least for gmod, i have to say coders of addons are smart enough to already include the resource.AddFile("") , and this is already sent to clients at connection. So if i do this recursively for all files that i recovered with my script, it will make client download files that are not needed.

dgibbs64 commented 9 years ago

Had a quick look. Not bad :-). One issue I have seen is the script would only work on Debian based systems. It will need to be a generic as possible.

UltimateByte commented 9 years ago

Mh... Got much more to learn... Is this the grep, the way i call variables, or the use of echo ? Maybe the 3 of them ?

Scarsz commented 9 years ago

@UltimateByte Grep and echo are both separate programs completely unrelated to variables in general, you can call variables anywhere you want as long as it's in good syntax (for example we call the AppID of a Garry's Mod server when installing that server from SteamCMD)

Grep finds strings inside of existing files, echo just says stuff at you

dgibbs64 commented 9 years ago

Handy tool http://www.shellcheck.net/

to define a variable use variable="hello"

to call a variable ${variable} (best practice) or $variable

examples

echo "${variable}"

grep ${variable} /tmp/foo.txt

if [ ${variable} == "hello" ]; then echo "the variable is hello" fi

you can do what you like with the variable you can print it to screen using echo or use it in an if statement or anything you need it for.

dgibbs64 commented 9 years ago

grep example

foo.txt contains the following lines

hello this is a test bye hello, how are you? see you soon

grep hello /dir/foo.txt

will output the following

hello this is a test hello,how are you?

dgibbs64 commented 9 years ago

Warning if bzip2 is not installed

if [ -z "$(command -v bzip2)" ]; then echo "WARNING bzip2 is not installed" echo "You can still continue but loading time won't be as best as possible for your players" while true; do read -p "Continue? [y/N]" yn case $yn in [Yy]* ) break;; [Nn]* ) exit;; * ) echo "Please answer yes or no.";; esac done fi

UltimateByte commented 9 years ago

God thanks guyz you rock ! I'll try that out when i get home (tomorrow) Then if i got time enough i'll try to learn using functions and to make it a function for your script.

UltimateByte commented 9 years ago

So if i understand your little code for warning, the grep check for bzip2 installed is not needed anymore, as it checks if there is a /bin/bzip2 ? I trust your code, but in this specific case with a command, you need the quotation marks ? " " if [ -z "$(command -v bzip2)" ]; then

UltimateByte commented 9 years ago

I also changed this to your code, as the variable bzip2 didn't exist anymore. End if bzip2 not installed if [ $bzip2 = 0 ]; then

=> if [ -z "$(command -v bzip2)" ]; then

UltimateByte commented 9 years ago

I'm a bit stuck right now, i need to generate a Lua file for resource but each file should be between a command : resource.AddFile ( " path/to/file.ext " )

So i already got the way to make a list (don't laugh, it took me a lot of time) : find fastdl ( -name "." ! -name "*.bz2" ) -print > fileslist.lua

But I can't find how to get every single line in between the command...

Scarsz commented 9 years ago

echo "resource.AddFile("\""${file}"\"")" >> fileslist.lua

UltimateByte commented 9 years ago

Thanks Scarz. I really don't have time for that these days, work + making music + administrating all the time... I'm not forgetting that thing but i don't know when i'll have more time for it.

dgibbs64 commented 9 years ago

Any update on this?

Scarsz commented 9 years ago

Was working on a script for this https://github.com/Scarsz/fastdl-sync

dgibbs64 commented 9 years ago

Awesome. Let me know when it's ready and I will integrate it :-)

UltimateByte commented 9 years ago

Nice ! I'm sorry for not having time to work on it yet (too much work and things to manage). I'm glad someone can ! Feel free to reuse some ideas in my script (like how to work with darkrp content)

dgibbs64 commented 8 years ago

@UltimateByte how complete is your fastdl module? if you think its ready it would be great if you could do a pull request and integrate this.

UltimateByte commented 8 years ago

Well i'm using it on a few gmod servers that have standalone addons (not workshop). It was able to help us for our FastDL servers. But it lacks a function to do a resource.addfile to every single file, for addons that doesn't have it already in their code (and it seems like it's most of them). More info : https://wiki.garrysmod.com/page/resource/AddFile Interesting note : "There's a 8192 downloadable file limit." It would require me get some more knowledge to be able to list every needed file and put it between resource.Addfile ("path/to/file.ext") in a generated lua file.

I guess servers like CSS don't need this command to add files, it's automatic. The last version can still be found here : https://github.com/UltimateByte/fastdl

I feel like as of now, it doesn't fit your quality standard. Would you feel like using it as a base and make something great with it ?

dgibbs64 commented 8 years ago

@UltimateByte I agree its a little rough but coudl work. There is another suggestion to add nginx webserver to LGSM to allow fastdl feature for Unreal servers however this would benefit source servers as well. But I think this would require a dedicated developer.

UltimateByte commented 8 years ago

Well, i don't even know if nginx can run as a user, which seems like a requirement for me, but it sounds like a great idea. BTW, once i'm done with Hurtworld, i can work on implementing this as a function. I'm gonna have to clean it before, and to try it with other source games.

UltimateByte commented 8 years ago

Well, i'm gonna try to implement this as a function once i got some other stuff done, then make one for servers using sourcemod such as CSS, CS GO, TF2...

UltimateByte commented 8 years ago

I'm on it !

I'm making the script so it will be easier to add support for other source games ! :)

UltimateByte commented 8 years ago

OK it's working ! Now i just gotta use your commands to set text forms and add some script logs ! Then when i got time i'm gonna work on the fn_gmod_force_download with scarz command as a start. echo "resource.AddFile("\""${file}"\"")" >> fileslist.lua Then the file will be copied into lua/autorun/server This will be optional, warned and explained, and i if user refuses it, it will check for existing file and ask if it should remove the previous one.

Shouldn't be too hard now, gonna have to get comfy with the bash while command though, which is a good thing. :)

jaredballou commented 8 years ago

This might be a good place to use Python or Perl. Text manipulation, and stuff like keeping the file list free of duplicates and properly ordered, is a pain in the ass with Bash. Is this a baseline for the FastDL/Nginx feature we were talking about before, or is this something different?

UltimateByte commented 8 years ago

As the list would be rebuilt from scratch every time you'd run the command, i guess bash would be enough. BTW, as i don't know python, unless somebody gives me a hand for it, i'll have to stick to bash.

UltimateByte commented 8 years ago

So, i'm back on it ! I have this to list all files : find fastdl ( -name "." ! -name "*.bz2" ) -print > fileslist.lua And this to write them using the syntax : echo "resource.AddFile("\""${file}"\"")" >> fileslist.lua

What i wanna do is make a while function for that i guess. I'm not used to recursive commands in bash, so i'm gonna have a look at it right now.

UltimateByte commented 8 years ago

So, after a huge pain in the ass, i got it working for gmod !

Main function : https://github.com/dgibbs64/linuxgsm/blob/fastdl/functions/command_fastdl.sh

Needs a bit more testing though, and need to adapt for other srcds games, which won't be that much of a problem :)

Screenshot : http://image.noelshack.com/fichiers/2016/07/1455965859-fastdl-screenshot2.png

lock[bot] commented 6 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

GameServerManagers / LinuxGSM

[Feature] FastDL #380

Warning if bzip2 is not installed