Large-ish backup preventing mark2 from properly restarting a spigot server?

[corydissinger]

I'm not quite sure if this is due to some way I have customized my mark2 installation, or if simply the very large compression (all of my world data compressed is 1.2GB) on my current hardware (8GB RAM, AMD Dual Core 3.2 GHz, 2 TB HDD) takes too long for mark2's taste (which is why I changed the crash-timeout and ping-timeout settings, but they have not prevented this issue.)

Every time my server restarts from my cron setup in scripts.txt, the error described below happens.

Here is my scripts.txt

---

 */5  *    *    *    *    ~save 1m
 0    */6  *    *    *    ~restart 10m;1m
 @serverstopped           ~backup
 @serverstarted           $mv ./backups/temp.tar.gz ./backups/archives/`date +%Y-%m-%d-%H`.tar.gz

---

My mark2.properties custom settings

---

java.cli.X.ms=2G
java.cli.X.mx=4G
mark2.jar-path=spigot-1.6.2-R0.1-SNAPSHOT.jar
plugin.save.warn-message=
plugin.save.message=
plugin.log.enabled=true
plugin.log.gzip=false
plugin.log.path=backups/logs/server-{timestamp}-{status}.log
plugin.backup.path=backups/temp.tar.gz
plugin.monitor.crash-timeout=30
plugin.monitor.ping-timeout=30

---

I can't copy/paste from my mark2 shell for some reason, but here is the order of events:

---

| Stopping server
| Savings players
# server.log written to /my/log/directory
# map backup starting...
# map backup saved to backups/temp.tar.gz
# An event handler threw an exception:
#   Callback: <bound method Process.server_stopped of <process.Process object at 0x2065f10>>
#   Event:
#     class_name: ServerStopped
#   Traceback (most recent call last):
#     File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", lin 293, in addCallbacks
#        self._runCallbacks()
#         .. some more stack traces ...
#     File "/usr/mark2/mk2/events/__init__.py", line 218, in _next_event
#       r = maybeDeferred(callback, event)
#  --- <exception caught here> ---
#     File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer,py", line 137, in maybeDeferred
#        result = f(*args, **kw)
#     File "/usr/mark2/mk2/services/process.py", line 123, in server_stopped
#       self.failsafe.cancel()
#     File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base,py", line 88, in cancel
#        raise error.AlreadyCalled
# twisted.internet.error.AlreadyCalled: Tried to cancel an already-called event

---

[rmb938]

Getting the same error.

[corydissinger]

How big are your backups? This error did not occur until the .tar.gz size was > 1 GB (for me, at least)

[rmb938]

Mine are very close to 1GB about 980MB

[crubinsson]

I am also getting this error. My backups are ~800mb...

[corydissinger]

I have a hunch, possibly a quick temporary fix (since I'm not really in the mood to debug python code atm haha)

1. Make a shell script that takes the worldfiles and backs them up into a tar.gz (mimicking ~backup)
2. Call it from the scripts.txt like so $bash mybackup.sh

If this works, we then know that the python/Twisted code executed from ~backup needs refactoring.

Hopefully I have the time to do this over the weekend.

[crubinsson]

I will do this when I get back tonight. Was planning to make a script for this anyways...

[Brottweiler]

I just now started to get this problem. The world is 1.6G and I have ~restart-empty run every day at the hour 6, and after that I have ~backup running when the server is stopped. It fails to restart with this error and forces me to run killall python so I can then start the server back up again.

[crubinsson]

It is a bummer for us with large worlds relative to the backup performance of the server. I can't have ~backup on my daily restart either. And i can confirm that doing my own backup script at the same time as restart also fails. Somehow the execution times out and fails the twistd service...

---- Christoffer Tibell notifications@github.com wrote ----

I just now started to get this problem. The world is 1.6G and I have ´~restart-empty´ run every day at the hour 6, and after that I have ´~backuprunning when the server is stopped. It fails to restart with this error and forces me to runkillall python` so I can then start the server back up again.

— Reply to this email directly or view it on GitHub.