City *Reader unexpected fault address issue

[piperck]

Hi @oschwald When I am in a situation with high concurrency and slightly insufficient CPU resources, occasionally an unexpected fault address issue occurs, leading to a panic. This kind of memory-level error makes it impossible for recover to catch it.

Below is the stack information I found, which ultimately points to the readLeft method. Can you help take a look or is this a known issue?

Thank you very much.

[oschwald]

There isn't much to go on in that stack trace. I am guessing this is some sort of mmap-related issue particular to your system. You don't mention what OS you are using, but if mmap is not reliable or is error prone on your system, I would suggest loading the database into memory instead. There are several ways to do this, e.g., using the FromBytes function to create the reader or setting the appropriate build tags.

[piperck]

@oschwald Thanks for quick response.🤣

Of course. Load data into memory is what I want...Maybe I'll be more prefer that way. I use your lib geoip2-golang and I found geoip2-golang has function FromBytes

Just a question... To use this do I open the file myself using os.open to read the whole []bytes or do we have another way?

[piperck]

I will test it for a week... And provide some information about what os I use before

uname -srm
Linux 5.4.0-1072-gke x86_64
cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"

OK I use this and it worked for now. I will test it and see if there are any similar problems at a later date.

func openGeoIP2MMDB(filePath string) (*geoip2.Reader, error) {
    mapFile, err := os.Open(filePath)
    if err != nil {
        _ = mapFile.Close()
        return nil, err
    }

    data, err := io.ReadAll(mapFile)
    if err != nil {
        _ = mapFile.Close()
        return nil, err
    }
    return geoip2.FromBytes(data)
}

[oschwald]

For what it is worth, I've used this library to serve many billions of requests in high concurrency situations on Linux and have never received a panic like the one you describe. I would guess there is something else going on here.

[piperck]

By the way... this is not a panic. It's Fatal... And recover can't handle it... My system will crash due to this.🤣

[oschwald]

Are you modifying the database (e.g., updating it or otherwise writing to it) when this happens? Is it possible that the database has been closed immediately before this happens?

I think you could recover if you enabled runtime.SetPanicOnFault, but it would be better to address the root cause.

[piperck]

No I'm prettey sure I don't close or update db and just use it. Let me paste some snippets and show how to use.

When server start, I will init the maxmind client And when server down or recive signal I will close it.

func FinishingWork() {
    maxmind.GeoIP2City.Close()
    maxmind.GeoIP2ConnectionType.Close()
    maxmind.GeoIP2Domain.Close()
    maxmind.GeoIP2Isp.Close()
    segmentioKafka.KafkaProducer.Close()
    //segmentioKafka.KafkaConsumer.Close()
    log.Info().Caller().Msg("safely exit Bye! ")
}

func Initialize() {
    maxmind.GeoIP2City, _ = maxmind.GenerateGeoClient("city")
    maxmind.GeoIP2ConnectionType, _ = maxmind.GenerateGeoClient("connectionType")
    maxmind.GeoIP2Domain, _ = maxmind.GenerateGeoClient("domain")
    maxmind.GeoIP2Isp, _ = maxmind.GenerateGeoClient("isp")
}

func main() {
    //go func() {
    //  http.ListenAndServe("localhost:6060", nil)
    //}()
    //runtime.GOMAXPROCS(3)
    Initialize()
    zerolog.SetGlobalLevel(zerolog.InfoLevel)
    if os.Getenv("env") == "dev" {
        zerolog.SetGlobalLevel(zerolog.DebugLevel)
    }
    signal.Notify(toolkits.SigChan, syscall.SIGINT, syscall.SIGTERM)

var GeoIP2City *geoip2.Reader
var GeoIP2ConnectionType *geoip2.Reader
var GeoIP2Domain *geoip2.Reader
var GeoIP2Isp *geoip2.Reader

func getCurrentAbPathByCaller() string {
    var abPath string
    _, filename, _, ok := runtime.Caller(0)
    if ok {
        abPath = path.Dir(filename)
    }
    return abPath
}

func GenerateGeoClient(uType string) (db *geoip2.Reader, err error) {
    absPW := getCurrentAbPathByCaller()
    switch uType {
    case "city":
        db, err = geoip2.Open(fmt.Sprintf("%s/GeoIP2-City.mmdb", absPW))
    case "connectionType":
        db, err = geoip2.Open(fmt.Sprintf("%s/GeoIP2-Connection-Type.mmdb", absPW))
    case "domain":
        db, err = geoip2.Open(fmt.Sprintf("%s/GeoIP2-Domain.mmdb", absPW))
    case "isp":
        db, err = geoip2.Open(fmt.Sprintf("%s/GeoIP2-ISP.mmdb", absPW))
    default:
        return nil, nil
    }

    return db, err
}

Just use it like this

    ip := net.ParseIP(enriched.UserIpaddress)
    city, err := maxmind.GeoIP2City.City(ip) // this db search may consider about mutex to avoid high peek performance issue.
    if err != nil {
        log.Warn().Caller().Str("ip", enriched.UserIpaddress).Str("geo extract info GeoIP2City error", err.Error()).Send()
    }
    isp, err := maxmind.GeoIP2Isp.ISP(ip)
    if err != nil {
        log.Warn().Caller().Str("ip", enriched.UserIpaddress).Str("geo extract info GeoIP2Isp error", err.Error()).Send()
    }
    domain, err := maxmind.GeoIP2Domain.Domain(ip)
    if err != nil {
        log.Warn().Caller().Str("ip", enriched.UserIpaddress).Str("geo extract info GeoIP2Domain error", err.Error()).Send()
    }
    ct, err := maxmind.GeoIP2ConnectionType.ConnectionType(ip)
    if err != nil {
        log.Warn().Caller().Str("ip", enriched.UserIpaddress).Str("geo extract info GeoIP2ConnectionType error", err.Error()).Send()
    }

Nothing else is done except to use reader directly.

This service running on the fastHttp based framework Fiber. Any suggestion or idea due to these information?

BTW I change it to readFromBytes from db to memory. It's been working normally for 11 hours. If it can still goes well, It may cased by mmap.

Thanks.

[piperck]

Hi @oschwald I use readFromBytes 3days. And do not occur panic anymore. But I found sometimes one day will raise 1 or 2 this warning

geo extract info GeoIP2City error: cannot call Lookup on a closed database, ip: 103.176.136.109, level: warn geo extract info GeoIP2ConnectionType error: the MaxMind DB file's search tree is corrupt, ip: 31.53.153.24, level: warn

I went through the code, and I didn't see anything except that at the end I would manually close reader and I would do that.

Do you have any clues?

Thanks.

[oschwald]

I suspect this is related to the panic you were seeing. It seems likely that something is closing your reader prematurely. It is hard to know the reason without seeing your whole application.

[piperck]

OK so this is why use mmap will fatal.... is that right ?

Because I use readFromBytes so it just print warm and reopen it, not fatal.

[piperck]

OK, I think this is caused by mmap. Running well except use more memory when I use readFromBytes instead of maxmind.open. I will close this issue, It may caused by operation system or enrivoment not compatible just mentioned upon.

Just comment my env for who may meet this in future.

Linux 5.4.0-1072-gke x86_64
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"

Running in K8S GCP.