freeze and unfreeze server to avoid warming up delay and cpu spikes

[Br31zh]

Hello,

That’s not really an issue, but I wanted to share this and don’t know where I can say it: I have my server on my (Linux) PC, and starting the server create CPU usage pikes, and freezes my other programs. So, instead of shutdown the server, I freeze it with a kill -STOP <pid> and resume with kill -CONT <pid>. It constantly use the 2GB of assigned RAM, but CPU, disk and network usage are null during this time. And the resume is instant, without CPU spike.

I had to set max-tick-time to -1 in server.properties and start the server before your script (and, since your script expect the server to be down, I freeze it before launching your script). I set the startup time to 1 in the configuration, and I resume the server before stopping it in my shutdown script (used when I want to completely stop the server).

It can give idea to others people, or to add new functionalities to the script, or as an example since it’s not an obvious way to use it.

[gekigek99]

Yep that a very cool idea!

It was actually why at the beginning I wanted to implement this script in docker so that it could just freeze and unfreeze the server to avoid the 20 seconds starting time.

I will rename the issue because it might become a major improvement in the future

p.s. have you tried setting the startup time to 0? it should work too

[Br31zh]

Yeah, but I didn’t success to make working the Go version, and by… laziness, I stay on the Python version until binary releases are made. And with 0 starting time, it’s making some exceptions in the console. But since it’s an edge case on an unmaintained version I didn’t try to debug it (neither making issue on it). But if I have time, and if you want (and can’t reproduce), I can do it.

PS : I don’t like Docker for some reasons, so if I can at least use my actual workaround with a standalone Go version, I’ll be happy (I don’t know if you can do it natively, especially on Windows. But if you can it’s even better. I like the fact that actually my server run in it’s own SystemD service and your program is just a proxy that I can set up transparently without changing other things than the server port).

[gekigek99]

Unfortunately I'm spending a lot of time to update the go version and I really can't help you much with the python (also because now it's deprecated)...

If you find a way to freeze unfreeze on Windows and you find a more "scripty" way to automate this process (even in python because then I can translate it) it would really save me some time in the future ;)

Anyway I'll keep this issue open until this feature will be implemented. Maybe I'll set an option to choose if the user wants to freeze/unfreeze or or just stop/start the server.

[Br31zh]

On Windows it seems to be possible to do the same (suspend and resume processes), but I have no idea on how to do that without a GUI, and if it give the same result as on Linux (and I don’t have Windows to test this).

To do that on Linux, you just have to know the PID of the java process. If you launch it directly you can know it, I don’t know how to get it when using screen. In my case I have a minecraft-server service, so I get it with systemctl show minecraft-server.service --property MainPID --value (and 0 means that it’s not started). Maybe you can use a pidfile.

And then, you just have to send STOP (23) signal to this PID to suspend, and CONT (25) signal to resume. Maybe this can help. I’m not good in Python, and don’t know anything in Go.

[Br31zh]

If it can help, there is my actual setup (in Bash): https://git.breizh.pm/minecraft-server-manager/

The service that launch your program: https://git.breizh.pm/minecraft-server-manager/tree/minecraft-wrapper.service

The part of the script that is doing the suspend/resume: https://git.breizh.pm/minecraft-server-manager/tree/minecraft-server-manager.sh#n45

And in config.json I have this:

"startMinecraftServerLin": "/home/minecraft/manager resume",
"stopMinecraftServerLin": "/home/minecraft/manager suspend",

[gekigek99]

I don’t know how to get it when using screen.

I'm not sure but i think there is some way of retrieving the pid of the process "screen" and then freeze the screen process (that, as a subprocess, is running java ecc). Maybe, with some magic, you might even be able to select the exact screen process (in the case of multiple running at the same time).

The part of the script that is doing the suspend/resume: https://git.breizh.pm/Breizh/minecraft-server-manager/src/branch/master/minecraft-server-manager.sh#L43

wow that's a pretty interesting setup... ;)

thanks for sharing these files... one day I'll try to implement this feature also for the official program

[f8ith]

screen does provide the information. screen -ls | awk '/\.nameofscreen\t/ {print strtonum($1)}' should do the trick.

[Br31zh]

Hello,

I’ve updated my script on https://git.breizh.pm/minecraft-server-manager/ to implement my TODOs points (the wait function was broken on midnight log rotation, and I find that rcon is not an elegant solution). But it relies on SystemD even more than before (which is bad for portability…). The main part (freeze and thaw the server) is still managed by a simple kill command like said in the first message of this issue, with some additional functionalities.

The last problem I have is to implement a wait time (and warning message) before stopping the server if some players are online (because SystemD is stopping your program (that work like a proxy) before the server, so all players are disconnected instantly ^^’).

And I finally switched to the Go version BTW. Setting startup time to 0 is working with this version.

PS: for Windows I found this utility: https://docs.microsoft.com/en-us/sysinternals/downloads/pssuspend that seems to be a SIGSTOP/SIGCONT equivalent, don’t tried since I don’t have Windows.

[Br31zh]

I get an idea, do you think it’s possible to add a (optional) way to set a different command on normal stop (when all players leave) and forced stop (when your program is stopped)?

That way I can say to my manager to send a warning and not save the game (since it will be saved by the stop command) instead of saving and suspending the server. Maybe in another issue, but I think it will be linked to this issue (because if you implement freeze/thaw mechanism, you’ll have to add a way to start and stop the server too in addition).

[gekigek99]

The last problem I have is to implement a wait time (and warning message) before stopping the server if some players are online

the message would be in the msh console or in-game? because in the future i would like to implement the possibility for msh to send message in game chat but it will be tricky because of encryption...

And I finally switched to the Go version BTW.

very nice!

PS: for Windows I found this utility: https://docs.microsoft.com/en-us/sysinternals/downloads/pssuspend that seems to be a SIGSTOP/SIGCONT equivalent, don’t tried since I don’t have Windows.

thanks I looked at it and it would do exactly what is needed, in any case I would like to avoid the usage of an external utility and maybe make use of internal system call (if it's not too complicated)... still it might be a very cool temporary solution to debug and test things

[gekigek99]

I get an idea, do you think it’s possible to add a (optional) way to set a different command on normal stop (when all players leave) and forced stop (when your program is stopped)?

sure it is and it would be also pretty easy:

• in function stopEmptyMinecraftServer(forceExec bool) there is the forceExec parameter, if set to true while calling the function, it will forcefully shutdown the server.
• in the block that execute the stopCommand, you can put an if forceExec: and write a log to console or what you prefer to do in case of a forceful shutdown

[gekigek99]

right now I'm in the uni exam period so I have no time but in the next weeks I'll start to work again on the project (I will first finish some other minor issues and then I'll try to solve the freeze dilemma ;) )

[Br31zh]

PS: for Windows I found this utility: https://docs.microsoft.com/en-us/sysinternals/downloads/pssuspend that seems to be a SIGSTOP/SIGCONT equivalent, don’t tried since I don’t have Windows.

thanks I looked at it and it would do exactly what is needed, in any case I would like to avoid the usage of an external utility and maybe make use of internal system call (if it's not too complicated)... still it might be a very cool temporary solution to debug and test things

It seems to be possible without utility, but it’s difficult to found good documentation about it… I found this: https://stackoverflow.com/a/11010508 https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-suspendthread https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-resumethread

But I don’t know how nor if you can use it more or less directly in Golang.

[gekigek99]

thanks @Br31zh for logging these links!

the more time it passes the more I'm interested in this idea of freezing the server... XD

[Br31zh]

I didn’t try but I think last versions will break my setup. I’ll try to adapt it during next weeks (well, probably not before April…). I don’t want to lose the hibernation ability.

[gekigek99]

I know, I'm sorry to hear that maybe you can play a little by starting and stopping the server using the screen command

Still, I think the last updates are necessary to make the freeze/unfreeze feature a reality, since now it will be much easyer interacting with the process

[hungl6844]

I'm going to see what I can do, but it will take me a while to learn docker. I think stop works on Linux and macOS, so I'll use that for those two, and then docker can be used for Windows... I hate to add a large dependency for my projects, but hopefully this will dramatically reduce the initial cpu spike...

[gekigek99]

Unfortunately in this period I cannot really help you since I'm in a "uni exam storm"... XD

What dependency are you planning to use? Are you sure there is nothing on windows to do the job?

[gekigek99]

Just be careful with managing the case where the server is a stopped process and a server shutdown happens

(you need to unstop the process, /save-all and stop the minecraft server gracefully)

[hungl6844]

Unfortunately in this period I cannot really help you since I'm in a "uni exam storm"... XD

Lol, that's fine, I wouldn't ask anyone to help me during school...

What dependency are you planning to use?

Docker, assuming you need docker installed to create a docker container(also assuming that's our use case, I still need to learn docker...)

Are you sure there is nothing on windows to do the job?

We could possibly use Windows scripts that do Linux things, and although I can't remember the name of the toolset, it would be useful... whatever the case, I'm going to try docker first, manual installs of scripts shouldn't be required for a project (although I could have the script download the required files in the script/server directory...)

[gekigek99]

Docker, assuming you need docker installed to create a docker container(also assuming that's our use case, I still need to learn docker...)

oh ok i thought you were planning on using some super duper cool golang libraries ahahah

---

These might be some nice links to start the research: https://github.com/gekigek99/minecraft-server-hibernation/issues/51#issuecomment-767730776

And some undocumented windows stuff: https://stackoverflow.com/a/61371356 https://cpp.hotexamples.com/examples/-/-/NtSuspendProcess/cpp-ntsuspendprocess-function-examples.html (this in particular it's quite easy to grasp what's happening)

(NtSuspendProcess, could be a really nice thing apparently but we are on our own to implement it... i've already did some work with importing windows dll... but still I think it's sort of unexplored territory...)

---

I don't really like the option of using external tools... i mean docker is ok but i dont want msh to rely too much on external stuff...

do you know that @lubocode is developing a docker branch? maybe it could be useful for you

[hungl6844]

Off of the link you sent me, I have found a tool. The only problem is that the eula may forbid a script downloading the tool... It's a Microsoft tool, which saves me from risking decompiling it to make sure it's safe(decompiling is against eula...), so could someone review it to check if I can? I read it, and I think it should be fine, but I would like someone else to verify.

https://pastebin.ubuntu.com/p/NyNwZP2qxJ/

do you know that @lubocode is developing a docker branch? maybe it could be useful for you

I did not know, but I will, with the information I now have, not be using docker, maintaining both a Go fork and a docker fork will be hard for me, but I do wish @lubocode luck with his fork.

[hungl6844]

I am having trouble figuring out how to automate fiding the server pid... jps lists it as new, and only javaw.exe shows up in any other tools I use...

[gekigek99]

Off of the link you sent me, I have found a tool.

You mean SuspendThread function (processthreadsapi.h)? You mean taht you would autodownload this file at msh runtime?

could someone review it to check if I can? I read it, and I think it should be fine, but I would like someone else to verify.

I'm really not into this legal stuff and i doubt someone will be able to answer you here... maybe ask this question on stackexchange, the legal section...

tomorrow if i can, i'll implement an example with this: https://pkg.go.dev/github.com/JamesHovious/w32#NtSuspendProcess

I did not know, but I will, with the information I now have, not be using docker, maintaining both a Go fork and a docker fork will be hard for me, but I do wish @lubocode luck with his fork.

Yea I meant that if you want to have a docker example you can take from him

[gekigek99]

I am having trouble figuring out how to automate fiding the server pid... jps lists it as new, and only javaw.exe shows up in any other tools I use...

You mean you want to implement it in msh golang code?

you can just use this: servctrl.ServTerminal.cmd.Process.Pid

[hungl6844]

servctrl.ServTerminal.cmd.Process.Pid

Sorry, I've never used Go before... are there arguments for that statement that define what process? Do you execute the statement in a class? I've looked it up, but I can't find anything...

EDIT: Well, I was an idiot, I looked through your code, and the answer is obvious. But wouldn't that give me the PID of the application, not the server itself?

[gekigek99]

wouldn't that give me the PID of the application, not the server itself?

It's giving you the pid of the terminal server process (not 100% sure if it is equal to the java server process)

[gekigek99]

This is an example of suspending and resuming a process in windows.
The syscall.StartProcess is probably too low level for me to be managed with confidence... and plus i can't find a way to redirect the process stdin, stdout and stderr to a function but only to the terminal.

package main

import (
    "bufio"
    "fmt"
    "os"
    "os/exec"
    "syscall"
    "time"
)

var (
    modntdll = syscall.NewLazyDLL("ntdll.dll")

    procNtSuspendProcess = modntdll.NewProc("NtSuspendProcess")
    procNtResumeProcess  = modntdll.NewProc("NtResumeProcess")
)

func main() {
    // simulate what's happening in msh
    procAttr := &syscall.ProcAttr{
        Dir:   ".",
        Files: []uintptr{uintptr(syscall.Stdin), uintptr(syscall.Stdout), uintptr(syscall.Stderr)},
        Sys: &syscall.SysProcAttr{
            CreationFlags: syscall.CREATE_NEW_PROCESS_GROUP,
        },
    }

    _, handle, err := syscall.StartProcess("C:\\WINDOWS\\system32\\cmd.exe", []string{"/C", "counter.bat"}, procAttr)
    if err != nil {
        fmt.Println("StartProcess:", err)
    }

    time.Sleep(time.Second)

    fmt.Println("stop")
    _, _, err = procNtSuspendProcess.Call(uintptr(handle))
    if err != nil {
        fmt.Println("procNtSuspendProcess:", err)
    }

    time.Sleep(time.Duration(2) * time.Second)

    fmt.Println("start")
    _, _, err = procNtResumeProcess.Call(uintptr(handle))
    if err != nil {
        fmt.Println("procNtResumeProcess", err)
    }

    // time.Sleep(time.Second)

    fmt.Println("exiting")
    os.Exit(1)
}

this is the bat file called in the script:

echo off

set /a n = 0

:START
echo %n%
set /a "n=%n%+1"
if %n%==1000 (EXIT 0)
GOTO START

[lubocode]

I did not know, but I will, with the information I now have, not be using docker, maintaining both a Go fork and a docker fork will be hard for me, but I do wish @lubocode luck with his fork.

Well, it has been sitting there for some time now, since I have been caught up in uni work and I mean, as long as it works for my purposes 🤷‍♂️ I will probably update it at some point though. Not sure when^^

If you can't manage to do it another way and have to default to a Docker variant, then feel free to ask how my setup works.

[gekigek99]

Thanks @lubocode for the update, I'm researching a way to suspend and unsuspend processes on windows right now (on linux I've been told it's easier)...

And I managed to find a way to do such a thing but now I'm struggling to retrieve the whole child tree of the minecraft server process that I need to suspend (I'm on a good path thought)

I will probably be able to avoid completely the need to rely on docker. So don't worry and feel free to develop the docker rep whenever you have time (Even thought I hope you'll catch up with the updates because it'a really nice project of yours and makes msh much more easy to setup)

(I mean... Come on, if docker and debuggers can suspend processes why can't I... We are on the same piece of hardware right?)

Update: i just noticed you were answering to @SorrowEater ... Anyway... here are my updates

[lubocode]

(Even thought I hope you'll catch up with the updates because it'a really nice project of yours and makes msh much more easy to setup)

Currently writing my Bachelor thesis. We'll see after that :)

(I mean... Come on, if docker and debuggers can suspend processes why can't I... We are on the same piece of hardware right?)

Well, Docker on Windows also uses the Linux Subsystem^^ There is a Windows Docker version, but only very few people actually use that.

Update: i just noticed you were answering to @SorrowEater ... Anyway... here are my updates

🤷‍♂️ Good luck with your tasks anyway^^ It might be worth investigating, if child processes go to sleep when the parent is suspended. Also, my MC installation on Windows seems to have switched to OpenJDK, I don't know since when that happened, but since you probably get process IDs and not program names, that should hopefully not matter. One good thing that came out from the switch though, is that I now know, that I can switch to OpenJDK 16 from 8, which I have been using in the Docker, since Microsoft are now officially using 16 for the MC desktop installation 😁

java.exe --version
openjdk 16.0.1 2021-04-20
OpenJDK Runtime Environment Microsoft-22300 (build 16.0.1+9)
OpenJDK 64-Bit Server VM Microsoft-22300 (build 16.0.1+9, mixed mode)

[gekigek99]

I managed to retrieve the process tree and now it should be quite trivial to make a function to suspend every single process of the specified tree (it will anyway take some time).

you can find my solution in this gist (if you can test it on linux/macos i would be grateful)

---

if child processes go to sleep when the parent is suspended.

I tested it on windows but it does not work: only the java parent process is suspended and not the java server process (the server keeps running possibly with bugs and errors)

this is the scheme:

msh
   |__ java parent process (of which I know the pid, it's the 'servctrl.ServTerm.cmd' variable)
            |__ java server process (in a separated process group, I don't know the pid)

java launches the server as a new process group so every signal sent to the java parent process is not sent to the java server process:
if I send the kill signal to the java parent process it makes the java server process an orphan process to which a new pid is given (effectively keeping the server running))

This might suggest that the behavior observed (only the java parent process being suspended) is due to the java server process being assigned to a new process group, rendering it not sensible to the suspend command.

This leads to my solution, which is to retrieve the whole process tree and suspend every single process one by one.

[gekigek99]

Also, my MC installation on Windows seems to have switched to OpenJDK, I don't know since when that happened, but since you probably get process IDs and not program names, that should hopefully not matter.

Yeah......... I lost a goooood 20 min trying to figure out why minecraft was not working after the update eheh...

[arch-user-france1]

Is it possible to send a ctrl-z or add a plugin that makes that?

Ctrl-Z works for 1.17 - I am not 100% sure Or could you launch MC throgh a process and when this process is suspended MC also stops?

[Br31zh]

Ctrl-Z send the SIGSTP signal, and do almost the same thing than SIGSTOP, which is what I’ve already proposed in the first post of this issue. The main problem is that you can’t do it on Windows (well, not in a simple way), from what I understood.

[gekigek99]

@debian-user-france1

Is it possible to send a ctrl-z or add a plugin that makes that?

I'm working on that... in windows it is pretty tricky to handle these kind of stuff but I'm slowly getting there (I'm a little back with the schedule because of certain bugs that were discovered)

Ctrl-Z works for 1.17 - I am not 100% sure

you are on linux right?

Or could you launch MC throgh a process and when this process is suspended MC also stops?

this is exactly what I am trying to do but handling suspension of processes trees, on 3 different OS and with different server statuses it's not a joke (at least for me)

[gekigek99]

@Br31zh

Ctrl-Z send the SIGSTP signal, and do almost the same thing than SIGSTOP

Yeah right now I'm focused on solving the windows processes trees suspension/continue... once understood, I'll implement this feature in msh - windows...

Once I see that it is working I'll set it up also for linux/mac which should be easy at that point.

This is probably the greatest update the msh has ever received so be patient... also because I'm doing it in my free time (which is never enough eheh ;) )

[MarkBremer]

I was trying to imply it until I realized I didn't understand anything about go. So I wrote a bash script that reads me rcon the player count and freezes the process. Unfortunately msh doesn't handle it at all and crashes. I then found a plugin for bukkit/spigot/paper that throttles the TPS extremely when no one is playing. The CPU idle usage went from ~25% to ~0%-3%. https://www.spigotmc.org/resources/hibernate.4441/

Of course the function would be nice for Forge servers.

[gekigek99]

i was aware of this plugin... the problem is that i need to implement it "outside" the minecraft server and not rely on plugins/other projects

this gist is the beginning of the research: https://gist.github.com/gekigek99/94f3629e929d514ca6eed55e111ae442

[gekigek99]

Ok i did my research and come up with a working solution: https://gist.github.com/gekigek99/dd18e671fdc6c3aab4b6b2ebfce8da0f

This is obviously just for windows but i think that on Linux it should be easier.

For now it's great that there is a solution, might not be definitive but it's a first step.

[hungl6844]

I was thinking I could make a paper-like patcher, where you download the executable and get a selection of popular jars as well as a custom url or something, but then I realized that's way too for out of my ability... I'll try of course, but it might be difficult, and your solution would probably work better either way (especially with paper, the paper jar being suspended might not end the actual minecraft process)

[gekigek99]

@SorrowEater it's one of the multiple possibilities...

I encourage you to experiment with solutions since it can really help you learn a lot.

The gist i did yesterday is infact not really necessary since everyone uses mah on Linux, on which should be a bit easier suspend a process. Still i learned a lot from researching into this subject.

The next step is to implement it into msh and make the suspend function work in msh on windows. Then we'll implement it also for Linux and iOS.

If you wish to contribute it would be super cool, maybe you can learn also some golang (when I started msh i did not even know the existence of this great language)

[gekigek99]

minecraft server suspension (for windows) implemented in 131a830837572cc8e781d26eef9f92c60d91f1e4

[gekigek99]

minecraft server suspension (for linux) implemented in 1c21a5ee5c473f61c19d17d2a3e9c19ccd9a0dd5

[gekigek99]

minecraft server suspension (for darwin) implemented in 5dead21a255d81093f8f162cd969e6dfebaa7de1

[gekigek99]

now it's just a matter of testing it out a bit more

[gekigek99]

@Br31zh did you try this new feature? does it behave as expected?

[Br31zh]

I haven't tested it, because I'm using a different system completely rebuilt from scratch since the version 1.18 of Minecraft. Maybe one day, because my current system is a bit shaky, but for now I don't really have the motivation, sorry ^^.

[gekigek99]

ok thanks anyway

[arch-user-france1]

I won't test it either because of the Microsoft account requirement.