unexpected EOF error

[W-Jie]

Hi, tidwall. Someday my server crash unexpectedly, and I try to restart tile38 server. When loading AOF file, then get error and exit. So, I cleaned the AOF file and restart, but all the data are loss.

# /usr/local/bin/tile38-server -d /tile38-8989/data -p 8989

   _______ _______
  |       |       |
  |____   |   _   |   Tile38 1.9.0 (3b6eaa3) 64 bit (amd64/linux)
  |       |       |   Port: 8989, PID: 21594
  |____   |   _   |
  |       |       |   tile38.com
  |_______|_______|

2017/10/16 16:10:16 [INFO] Server started, Tile38 version 1.9.0, git 3b6eaa3
2017/10/16 16:10:35 [INFO] AOF loaded 413338 commands: 18.85s, 21931/s, 13 MB/s
2017/10/16 16:10:35 [FATA] unexpected EOF

Before cleaning, I find that the appendonly.aof file size is 249MB, is the aof file too large or damaged? And, why the queue.db file size is 0 ? What is it used for?

# ll -h data/
total 249M
-rw------- 1 adm adm 249M Sep 25 08:20 appendonly.aof
-rw------- 1 adm adm   87 Aug 31 17:29 config
-rw-rw-r-- 1 adm adm    0 Aug 31 17:29 queue.db

Thank you!

[tidwall]

Hi W-Jie

Do you happen to have a log file from when server crashed? It's possible that there's detailed information about the cause of the crash.

Before cleaning, I find that the appendonly.aof file size is 249MB, is the aof file too large or damaged?

An unexpected EOF error means that the AOF was truncated for some reason. 249MB isn't too large.

And, why the queue.db file size is 0 ? What is it used for?

The queue.db file is used for geofence notifications from the SETHOOK command.

[huangpeizhi]

Hi @tidwall

The queue.db file is not empty, and whether the message is triggered is not taken away？

[tidwall]

I'm closing this issue for now because it's a few months old, but feel free to reopen if you are still running into problems.

Thanks!

[hmarcelodn]

I am experiencing the same issue running this in Kubenetes:

admins-MacBook-Pro:scripts admin$ kubectl logs tile38-df59bd788-zbln2

   _______ _______
  |       |       |
  |____   |   _   |   Tile38 1.12.2 (d5af91c) 64 bit (amd64/linux)
  |       |       |   Port: 9851, PID: 1
  |____   |   _   |
  |       |       |   tile38.com, patreon.com/tidwall
  |_______|_______|

2018/09/25 12:52:31 [INFO] Server started, Tile38 version 1.12.2, git d5af91c
2018/09/25 12:52:47 [INFO] AOF loaded 1299952 commands: 15.31s, 84932/s, 31 MB/s
2018/09/25 12:52:47 [FATA] unexpected EOF

Any ideas? It is restarting more than 100 times so far...

[hmarcelodn]

@tidwall my file is 470MB , could it be an issue ?

[mootezbessifi]

@hmarcelodn any findings for your issue? Currently I am running with the same

[tidwall]

About eight months ago someone pinged me and said they ran into an issue with their AOF being corrupted. They too were using Kubernetes and in the cloud. After some investigating apparently the block store environment padded a bunch zeros to the tail end of the AOF file. It was unclear how that happened but after wiping out the zeros the AOF loaded.

Could this possibly be the same issue?

[mootezbessifi]

I am running on top of Kubernetes and using cinder openstack as backend storage. I did a tail for the aof and did not find the bunch zeros.

[jwillmer]

I get the same error running in a docker container once in a while. Tile38 is using it's own volume so nothing should interfere with it but I still get the problem.

last lines of appendonly.aof

image: tile38/tile38:1.19.5

[tidwall]

@jwillmer Did you use cat -v data/appendonly.aof to show the null bytes? Also can you determine the physical size of the file, such as with ls -l data/appendonly.aof?

[jwillmer]

@tidwall I did not. I used a normal editor to get rid of the last line and all works again. Next time it happens I will do that and post it. I also updated to the latest version today so that it is easier to investigate.

[tidwall]

I pushed an update that addresses this issue (93e3a06).

[jwillmer]

Nice, I will update to this version as soon as the new docker image is available in the hub 👍