search

aerospike / aerospike-client-go

Aerospike Client Go
Apache License 2.0
433 stars 199 forks source link

Client v7 occasionally crushes with concurrent map read and map write #446

Open Gaudeamus opened 2 months ago

Gaudeamus commented 2 months ago

Hello, seems like across different client versions the same bug is persistent Current report is for client version v7.6.0

fatal error: concurrent map read and map write

goroutine 1314290423 [running]:
github.com/aerospike/aerospike-client-go/v7.PartitionForRead(0xc01b4086c0?, 0xc00693d860, 0xc3d4cebec0)
        /src/dsp/vendor/github.com/aerospike/aerospike-client-go/v7/partition.go:74 +0x95
github.com/aerospike/aerospike-client-go/v7.newReadCommand(0xc58bb36008, 0xc00693d860, 0xc3d4cebec0, {0xc3d8445860?, 0x5, 0x5}, 0x40?)
        /src/dsp/vendor/github.com/aerospike/aerospike-client-go/v7/read_command.go:54 +0x92
github.com/aerospike/aerospike-client-go/v7.(*Client).Get(0xc58bbb1f20, 0xc3d8445860?, 0xc3d4cebec0, {0xc3d8445860, 0x5, 0x5})
        /src/dsp/vendor/github.com/aerospike/aerospike-client-go/v7/client.go:475 +0x131
gitlab.mgid.com/services/dsp/aerospike.(*Client).Get(0x1739518?, 0xc?, 0xc2c649ea00?, {0xc3d8445860?, 0x1426740?, 0xc6c676c660?})
        /src/dsp/aerospike/client.go:126 +0x91
gitlab.mgid.com/services/dsp/aerospike.(*Session).GetDataByClientUserId(0xc08ee9ec30, {0xc2c649ea00, 0x6}, {0xc32db8c018?, 0x1?})
....

Note: same issue was reported for v6,v5 https://github.com/aerospike/aerospike-client-go/issues/399

khaf commented 2 months ago

Thanks for your report. I'm working on it.

ravisastryk commented 1 week ago

I understand everyone has been busy, but I made an attempt to fix this issue. Please take a moment to review it when you can so that everyone can benefit.

Here is my diff for your review https://github.com/aerospike/aerospike-client-go/pull/454

cc @khaf @Gaudeamus

khaf commented 3 days ago

Hello, seems like across different client versions the same bug is persistent Current report is for client version v7.6.0

fatal error: concurrent map read and map write

goroutine 1314290423 [running]:
github.com/aerospike/aerospike-client-go/v7.PartitionForRead(0xc01b4086c0?, 0xc00693d860, 0xc3d4cebec0)
        /src/dsp/vendor/github.com/aerospike/aerospike-client-go/v7/partition.go:74 +0x95
github.com/aerospike/aerospike-client-go/v7.newReadCommand(0xc58bb36008, 0xc00693d860, 0xc3d4cebec0, {0xc3d8445860?, 0x5, 0x5}, 0x40?)
        /src/dsp/vendor/github.com/aerospike/aerospike-client-go/v7/read_command.go:54 +0x92
github.com/aerospike/aerospike-client-go/v7.(*Client).Get(0xc58bbb1f20, 0xc3d8445860?, 0xc3d4cebec0, {0xc3d8445860, 0x5, 0x5})
        /src/dsp/vendor/github.com/aerospike/aerospike-client-go/v7/client.go:475 +0x131
gitlab.mgid.com/services/dsp/aerospike.(*Client).Get(0x1739518?, 0xc?, 0xc2c649ea00?, {0xc3d8445860?, 0x1426740?, 0xc6c676c660?})
        /src/dsp/aerospike/client.go:126 +0x91
gitlab.mgid.com/services/dsp/aerospike.(*Session).GetDataByClientUserId(0xc08ee9ec30, {0xc2c649ea00, 0x6}, {0xc32db8c018?, 0x1?})
....

Note: same issue was reported for v6,v5 #399

@Gaudeamus Could you please paste the whole panic report? Usually Go reports race conditions with where both the last read and write happened. Without it lots of guess work is needed to decide how the race is happening.

In your other report, I speculated that you may - unnecessarily - be creating a new Client object per each request. Is that the case?

Gaudeamus commented 2 days ago

Hello, khaf Here are most recent logs with aerospike related routines Actually yes, we have watching routine that periodically reconnects to nodes available currently. This is a makeshift solution because of poor connection 

fatal error: concurrent map read and map write

goroutine 270604971 [running]:
github.com/aerospike/aerospike-client-go/v7.PartitionForRead(0xc670a41ef0?, 0xc06b982300, 0xc4ebd71140)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/partition.go:74 +0xa7
github.com/aerospike/aerospike-client-go/v7.newReadCommand(0xc29f119b88, 0xc06b982300, 0xc4ebd71140, {0xc4ec925810?, 0x5, 0x5}, 0x40?)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/read_command.go:54 +0x92
github.com/aerospike/aerospike-client-go/v7.(*Client).Get(0xc29e1477a0, 0xc4ec925810?, 0xc4ebd71140, {0xc4ec925810, 0x5, 0x5})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/client.go:490 +0x131
[project]/[path]/aerospike.(*Client).Get(0x198978c?, 0xc?, 0xc33fb42010?, {0xc4ec925810?, 0x160b040?, 0xc4ec965710?})
    /src/[project]/aerospike/client.go:126 +0x91
[project]/[path]/aerospike.(*Session).GetDataByClientUserId(0xc08efd12c0, {0xc33fb42010, 0x6}, {0xc412fb5bf0?, 0x0?})
    /src/[project]/aerospike/data.personalized.go:66 +0x138
[project]/[path]/[path]/db.aerospikeService.GetClientData({{0x7f8ac81fbf80?, 0xc08efd12c0?}, 0x0?}, {0xc33fb42010?, 0x10?}, {0xc412fb5bf0?, 0x14c7e78?})
    /src/[project]/[path]/db/aerospike.go:76 +0xd6

goroutine 3957 [runnable]:
github.com/aerospike/aerospike-client-go/v7.partitionMap.cleanup(...)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/partitions.go:101
github.com/aerospike/aerospike-client-go/v7.(*Cluster).Close(0xc29f119b88)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/cluster.go:857 +0x13b
github.com/aerospike/aerospike-client-go/v7.(*Client).Close(...)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/client.go:253
[project]/[path]/aerospike.(*Session).connect.func1()
    /src/[project]/aerospike/aerospike.go:183 +0x9d
[project]/[path]/aerospike.(*Session).connect(0xc08efd12c0, {0xc4ec426b28, 0x3, 0x3})
    /src/[project]/aerospike/aerospike.go:197 +0x102
[project]/[path]/aerospike.(*Session).watch.func1()
    /src/[project]/aerospike/aerospike.go:259 +0x225
created by [project]/[path]/aerospike.(*Session).watch in goroutine 1
    /src/[project]/aerospike/aerospike.go:229 +0x13c

goroutine 274511766 [sleep]:
time.Sleep(0xf4240)
    /usr/local/go/src/runtime/time.go:285 +0xf2
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).executeAt(0xc4ec997ce0, {0x1c6fe78, 0xc4ec997ce0}, 0xc06b982300, {0x0?, 0xc29f119b88?, 0x290bde0?}, 0xc4ecdb5380?)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2592 +0x23d
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).execute(0xc4ec997ce0, {0x1c6fe78, 0xc4ec997ce0})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2556 +0x65
github.com/aerospike/aerospike-client-go/v7.(*readCommand).Execute(...)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/read_command.go:266
github.com/aerospike/aerospike-client-go/v7.(*Client).Get(0xc29e1477a0, 0xc4ec9eb450?, 0xc4ec9a7740, {0xc4ec9eb450, 0x5, 0x5})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/client.go:495 +0x1fc
[project]/[path]/aerospike.(*Client).Get(0x198978c?, 0xc?, 0xc33f99ce60?, {0xc4ec9eb450?, 0x160b040?, 0xc4ec9d7920?})
    /src/[project]/aerospike/client.go:126 +0x91
[project]/[path]/aerospike.(*Session).GetDataByClientUserId(0xc08efd12c0, {0xc33f99ce60, 0x6}, {0xc4ecda8080?, 0x0?})
    /src/[project]/aerospike/data.personalized.go:66 +0x138
[project]/[path]/[path]/db.aerospikeService.GetClientData({{0x7f8ac81fbf80?, 0xc08efd12c0?}, 0x0?}, {0xc33f99ce60?, 0xc00ca00708?}, {0xc4ecda8080?, 0x0?})
    /src/[project]/[path]/db/aerospike.go:76 +0xd6
[project]/[path]/[path]/service.(*Context).parseRequest(0xc1017b8008)
    /src/[project]/[path]/service/handle_parse.go:109 +0xe65

goroutine 273497931 [sleep]:
time.Sleep(0xf4240)
    /usr/local/go/src/runtime/time.go:285 +0xf2
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).executeAt(0xc4ecd3c630, {0x1c6fe78, 0xc4ecd3c630}, 0xc06b982300, {0x0?, 0xc29f119b88?, 0x290bde0?}, 0xc4ecd0ea00?)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2592 +0x23d
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).execute(0xc4ecd3c630, {0x1c6fe78, 0xc4ecd3c630})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2556 +0x65
github.com/aerospike/aerospike-client-go/v7.(*readCommand).Execute(...)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/read_command.go:266
github.com/aerospike/aerospike-client-go/v7.(*Client).Get(0xc29e1477a0, 0xc4ecd421e0?, 0xc4eaf01b00, {0xc4ecd421e0, 0x5, 0x5})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/client.go:495 +0x1fc
[project]/[path]/aerospike.(*Client).Get(0x198978c?, 0xc?, 0xc33f9aeef0?, {0xc4ecd421e0?, 0x160b040?, 0xc4ecd34590?})
    /src/[project]/aerospike/client.go:126 +0x91
[project]/[path]/aerospike.(*Session).GetDataByClientUserId(0xc08efd12c0, {0xc33f9aeef0, 0x6}, {0xc4eaf89638?, 0x1?})
    /src/[project]/aerospike/data.personalized.go:66 +0x138
[project]/[path]/[path]/db.aerospikeService.GetClientData({{0x7f8ac81fbf80?, 0xc08efd12c0?}, 0x0?}, {0xc33f9aeef0?, 0x10?}, {0xc4eaf89638?, 0x14c7e78?})
    /src/[project]/[path]/db/aerospike.go:76 +0xd6
[project]/[path]/[path]/service.(*Context).parseRequest(0xc6561ba008)
    /src/[project]/[path]/service/handle_request.go:242 +0x1d85

goroutine 273497757 [sleep]:
time.Sleep(0xf4240)
    /usr/local/go/src/runtime/time.go:285 +0xf2
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).executeAt(0xc4ecc68bb0, {0x1c6fe78, 0xc4ecc68bb0}, 0xc06b982300, {0x0?, 0xc29f119b88?, 0x290bde0?}, 0xc4ed411f00?)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2592 +0x23d
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).execute(0xc4ecc68bb0, {0x1c6fe78, 0xc4ecc68bb0})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2556 +0x65
github.com/aerospike/aerospike-client-go/v7.(*readCommand).Execute(...)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/read_command.go:266
github.com/aerospike/aerospike-client-go/v7.(*Client).Get(0xc29e1477a0, 0xc4ecc25950?, 0xc4ecc01500, {0xc4ecc25950, 0x5, 0x5})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/client.go:495 +0x1fc
[project]/[path]/aerospike.(*Client).Get(0x198978c?, 0xc?, 0xc33f86eda6?, {0xc4ecc25950?, 0x160b040?, 0xc4ecc3b320?})
    /src/[project]/aerospike/client.go:126 +0x91
[project]/[path]/aerospike.(*Session).GetDataByClientUserId(0xc08efd12c0, {0xc33f86eda6, 0x6}, {0xc0b4285b00?, 0x0?})
    /src/[project]/aerospike/data.personalized.go:66 +0x138
[project]/[path]/[path]/db.aerospikeService.GetClientData({{0x7f8ac81fbf80?, 0xc08efd12c0?}, 0x0?}, {0xc33f86eda6?, 0x10?}, {0xc0b4285b00?, 0x14c7e78?})
    /src/[project]/[path]/db/aerospike.go:76 +0xd6
[project]/[path]/[path]/service.(*Context).parseRequest(0xc62d95ac08)
    /src/[project]/[path]/service/handle_request.go:242 +0x1d85

goroutine 273497418 [sleep]:
time.Sleep(0xf4240)
    /usr/local/go/src/runtime/time.go:285 +0xf2
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).executeAt(0xc4ec24f290, {0x1c6fe78, 0xc4ec24f290}, 0xc06b982300, {0x0?, 0xc29f119b88?, 0x290bde0?}, 0xc4ed31bf00?)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2592 +0x23d
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).execute(0xc4ec24f290, {0x1c6fe78, 0xc4ec24f290})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2556 +0x65
github.com/aerospike/aerospike-client-go/v7.(*readCommand).Execute(...)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/read_command.go:266
github.com/aerospike/aerospike-client-go/v7.(*Client).Get(0xc29e1477a0, 0xc4ed318810?, 0xc4ebb07bc0, {0xc4ed318810, 0x1, 0x1})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/client.go:495 +0x1fc
[project]/[path]/aerospike.(*Client).Get(0x197e652?, 0x3?, 0xc33f9f22f4?, {0xc4ed318810?, 0x160b040?, 0xc4ed3187f0?})
    /src/[project]/aerospike/client.go:126 +0x91
[project]/[path]/aerospike.(*Session).GetSinceLastVisitByClientUserId(0xc08efd12c0, {0xc33f9f22f4, 0x6}, {0xc376b98e88?, 0x0?})
    /src/[project]/aerospike/data.personalized.go:140 +0xad
[project]/[path]/[path]/service.(*Context).parseRequest(0xc63a433308)
    /src/[project]/[path]/service/handle_request.go:268 +0x1fee

goroutine 259377035 [sleep]:
time.Sleep(0xf4240)
    /usr/local/go/src/runtime/time.go:285 +0xf2
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).executeAt(0xc4e6b611e0, {0x1c6fe78, 0xc4e6b611e0}, 0xc06b982300, {0x0?, 0xc29f119b88?, 0x290bde0?}, 0xc4ecf51000?)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2592 +0x23d
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).execute(0xc4e6b611e0, {0x1c6fe78, 0xc4e6b611e0})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2556 +0x65
github.com/aerospike/aerospike-client-go/v7.(*readCommand).Execute(...)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/read_command.go:266
github.com/aerospike/aerospike-client-go/v7.(*Client).Get(0xc29e1477a0, 0xc4e6b774f0?, 0xc4ecf72240, {0xc4e6b774f0, 0x5, 0x5})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/client.go:495 +0x1fc
[project]/[path]/aerospike.(*Client).Get(0x198978c?, 0xc?, 0xc33f9c3fb0?, {0xc4e6b774f0?, 0x160b040?, 0xc4ed68a260?})
    /src/[project]/aerospike/client.go:126 +0x91
[project]/[path]/aerospike.(*Session).GetDataByClientUserId(0xc08efd12c0, {0xc33f9c3fb0, 0x6}, {0xc69cabe288?, 0x0?})
    /src/[project]/aerospike/data.personalized.go:66 +0x138
[project]/[path]/[path]/db.aerospikeService.GetClientData({{0x7f8ac81fbf80?, 0xc08efd12c0?}, 0x0?}, {0xc33f9c3fb0?, 0x10?}, {0xc69cabe288?, 0x14c7e78?})
    /src/[project]/[path]/db/aerospike.go:76 +0xd6
[project]/[path]/[path]/service.(*Context).parseRequest(0xc64b8e1908)
    /src/[project]/[path]/service/handle_request.go:242 +0x1d85

goroutine 273495450 [runnable]:
runtime.CallersFrames(...)
    /usr/local/go/src/runtime/symtab.go:80
runtime.Caller(0xc)
    /usr/local/go/src/runtime/extern.go:307 +0x75
github.com/aerospike/aerospike-client-go/v7.stackTrace({0xc09d290350?, 0xc09d2902b0?})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/error.go:511 +0xa9
github.com/aerospike/aerospike-client-go/v7.newError(0x14, {0xc09d290350?, 0x198978c?, 0x198978c?})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/error.go:127 +0x70
github.com/aerospike/aerospike-client-go/v7.newInvalidNamespaceError(...)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/partition.go:324
github.com/aerospike/aerospike-client-go/v7.PartitionForRead(0xc67115b440?, 0xc06b982300, 0xc4ecc015c0)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/partition.go:77 +0x21a
github.com/aerospike/aerospike-client-go/v7.newReadCommand(0xc29f119b88, 0xc06b982300, 0xc4ecc015c0, {0xc4ed45c2d0?, 0x5, 0x5}, 0x40?)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/read_command.go:54 +0x92
github.com/aerospike/aerospike-client-go/v7.(*Client).Get(0xc29e1477a0, 0xc4ed45c2d0?, 0xc4ecc015c0, {0xc4ed45c2d0, 0x5, 0x5})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/client.go:490 +0x131
[project]/[path]/aerospike.(*Client).Get(0x198978c?, 0xc?, 0xc33ea56430?, {0xc4ed45c2d0?, 0x160b040?, 0xc4ecc3be60?})
    /src/[project]/aerospike/client.go:126 +0x91
[project]/[path]/aerospike.(*Session).GetDataByClientUserId(0xc08efd12c0, {0xc33ea56430, 0x6}, {0xc0b4285fb0?, 0x0?})
    /src/[project]/aerospike/data.personalized.go:66 +0x138
[project]/[path]/[path]/db.aerospikeService.GetClientData({{0x7f8ac81fbf80?, 0xc08efd12c0?}, 0x0?}, {0xc33ea56430?, 0x10?}, {0xc0b4285fb0?, 0x14c7e78?})
    /src/[project]/[path]/db/aerospike.go:76 +0xd6
[project]/[path]/[path]/service.(*Context).parseRequest(0xc3a7b8f908)
    /src/[project]/[path]/service/handle_request.go:242 +0x1d85

goroutine 273498080 [sleep]:
time.Sleep(0xf4240)
    /usr/local/go/src/runtime/time.go:285 +0xf2
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).executeAt(0xc4ed5104d0, {0x1c6fe78, 0xc4ed5104d0}, 0xc06b982300, {0x0?, 0xc29f119b88?, 0x290bde0?}, 0xc4ed525e80?)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2592 +0x23d
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).execute(0xc4ed5104d0, {0x1c6fe78, 0xc4ed5104d0})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2556 +0x65
github.com/aerospike/aerospike-client-go/v7.(*readCommand).Execute(...)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/read_command.go:266
github.com/aerospike/aerospike-client-go/v7.(*Client).Get(0xc29e1477a0, 0xc4ed53a150?, 0xc4ed5380c0, {0xc4ed53a150, 0x1, 0x1})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/client.go:495 +0x1fc
[project]/[path]/aerospike.(*Client).Get(0x197e652?, 0x3?, 0xc33f995f64?, {0xc4ed53a150?, 0x160b040?, 0xc4ed53a130?})
    /src/[project]/aerospike/client.go:126 +0x91
[project]/[path]/aerospike.(*Session).GetSinceLastVisitByClientUserId(0xc08efd12c0, {0xc33f995f64, 0x6}, {0xc7273dd488?, 0x0?})
    /src/[project]/aerospike/data.personalized.go:140 +0xad
[project]/[path]/[path]/service.(*Context).parseRequest(0xc097286c08)
    /src/[project]/[path]/service/handle_request.go:268 +0x1fee

goroutine 273499039 [sleep]:
time.Sleep(0xf4240)
    /usr/local/go/src/runtime/time.go:285 +0xf2
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).executeAt(0xc4ec24f4a0, {0x1c6fe78, 0xc4ec24f4a0}, 0xc06b982300, {0x0?, 0xc29f119b88?, 0x290bde0?}, 0xc4ed352080?)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2592 +0x23d
github.com/aerospike/aerospike-client-go/v7.(*baseCommand).execute(0xc4ec24f4a0, {0x1c6fe78, 0xc4ec24f4a0})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/command.go:2556 +0x65
github.com/aerospike/aerospike-client-go/v7.(*readCommand).Execute(...)
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/read_command.go:266
github.com/aerospike/aerospike-client-go/v7.(*Client).Get(0xc29e1477a0, 0xc4ec2539f0?, 0xc4ebb07c80, {0xc4ec2539f0, 0x5, 0x5})
    /src/[project]/vendor/github.com/aerospike/aerospike-client-go/v7/client.go:495 +0x1fc
[project]/[path]/aerospike.(*Client).Get(0x198978c?, 0xc?, 0xc33f9f25b0?, {0xc4ec2539f0?, 0x160b040?, 0xc4ed3189b0?})
    /src/[project]/aerospike/client.go:126 +0x91
[project]/[path]/aerospike.(*Session).GetDataByClientUserId(0xc08efd12c0, {0xc33f9f25b0, 0x6}, {0xc33f9f2550?, 0x5?})
    /src/[project]/aerospike/data.personalized.go:66 +0x138
[project]/[path]/[path]/db.aerospikeService.GetClientData({{0x7f8ac81fbf80?, 0xc08efd12c0?}, 0x0?}, {0xc33f9f25b0?, 0x1?}, {0xc33f9f2550?, 0xc5fa3c0540?})
    /src/[project]/[path]/db/aerospike.go:76 +0xd6
[project]/[path]/[path]/service.(*Context).parseRequest(0xc6788bac08)
    /src/[project]/[path]/service/handle_request.go:242 +0x1d85