Crash by: signal SIGBUS: bus error

[Daimon04]

The program crashed due to memory access violation. Stacktrace:

fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7fb9261ca2a3 pc=0x9fb6d9]

goroutine 861220198 [running]:
runtime.throw({0xc1d906?, 0x8?})
        /usr/local/go/src/runtime/panic.go:992 +0x71 fp=0xc0002cd738 sp=0xc0002cd708 pc=0x433f51
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:815 +0x125 fp=0xc0002cd788 sp=0xc0002cd738 pc=0x449385
github.com/oschwald/maxminddb-golang.nodeReader28.readLeft(...)
        /home/daimon/go/src/dfn/vendor/github.com/oschwald/maxminddb-golang/node.go:29
github.com/oschwald/maxminddb-golang.(*nodeReader28).readLeft(0x40d687?, 0x30?)
        <autogenerated>:1 +0x39 fp=0xc0002cd7a8 sp=0xc0002cd788 pc=0x9fb6d9
github.com/oschwald/maxminddb-golang.(*Reader).traverseTree(0xc000228b40, {0xc0002dbd9c, 0x4, 0x120?}, 0xa?, 0x20)
        /home/daimon/go/src/dfn/vendor/github.com/oschwald/maxminddb-golang/reader.go:286 +0xc3 fp=0xc0002cd7d8 sp=0xc0002cd7a8 pc=0x9f9083
github.com/oschwald/maxminddb-golang.(*Reader).lookupPointer(0xc000228b40, {0xc0002dbd90?, 0xbe0e00?, 0x7fb95132ca01?})
        /home/daimon/go/src/dfn/vendor/github.com/oschwald/maxminddb-golang/reader.go:264 +0x17a fp=0xc0002cd858 sp=0xc0002cd7d8 pc=0x9f8e7a
github.com/oschwald/maxminddb-golang.(*Reader).Lookup(0xc000228b40, {0xc0002dbd90?, 0xc0011ff7f0?, 0xc0002cd918?}, {0xaf2a00, 0xc0006dc5a0})
        /home/daimon/go/src/dfn/vendor/github.com/oschwald/maxminddb-golang/reader.go:137 +0x45 fp=0xc0002cd888 sp=0xc0002cd858 pc=0x9f8445
github.com/oschwald/geoip2-golang.(*Reader).City(0xc000964840, {0xc0002dbd90, 0x10, 0x10})
        /home/daimon/go/src/dfn/vendor/github.com/oschwald/geoip2-golang/reader.go:336 +0x127 fp=0xc0002cd950 sp=0xc0002cd888 pc=0x9fc1e7
github.com/digifront/DFN_backend/services/gateway/handler.getGeo({0xd33010?, 0xc000519bc0?}, {0xc0002dbd80, 0xc})
        /home/daimon/go/src/dfn/services/gateway/handler/handlers.go:520 +0x165 fp=0xc0002cda18 sp=0xc0002cd950 pc=0xaa4e85
...Other code

My code:

func getGeo(ctx context.Context, ip string) (*types.Geo, error) {

    netIP := net.ParseIP(ip)
    var geo types.Geo
    Geo.RLock()
    city, err := Geo.Db.City(netIP)
    Geo.RUnlock()
    if err != nil {
        ...
        return &geo, err
    }
    ...
    return &geo, nil
}

I have about 500 RPS, this crash happens 1-2 times a month on the production server. The DB file is updated 1 time per week. I tried to specifically reproduce this problem, but I did not succeed.

[oschwald]

Does this happen when the database is being updated or shortly thereafter? If so, how is the database being updated? If you are not replacing the file atomically, issues like this could occur.

[Daimon04]

Yes, you are right. This happens when the database is updated. To update we use the program https://github.com/maxmind/geoipupdate

[oschwald]

geoipupdate should replace the file atomically if the OS and file system support it. What OS and file system are you using?

[Daimon04]

OS: Ubuntu 20.04 File system: Ext4

[oschwald]

Interesting. We have been using this on similar systems with similar loads and updated by geoipupdate. We have never experienced a SIGBUS.

To confirm, this is a local file system, right? Network drives potentially could introduce issues with a mmap'd file.

Are you certain nothing but geoipupdate is modifying the files while this is running? What version of geoipupdate are you using?

I suppose a workaround that is not particularly satisfying is to load the database into memory rather than using a memory-mapped file. You can do this by using geoip2.FromBytes instead of geoip2.Open.

[Daimon04]

sorry for the long answer. geoipupdate version 4.6.0

"To confirm, this is a local file system, right" - yes, is a local file system

Your recommendation to use geoip2.FromBytes helped me, the error disappeared. Thank you very much.

[oschwald]

Given that I am unable to reproduce this and it sounds like a mmap problem that is not specific to this library, I am going to close this issue.