BatchGetObjects giving false for several objects randomly

[kartavya-ramnani]

So, I do a batchGetObjects for 128 keys with the default BatchPolicy.

most of the time I get 128 items in result but there are times ( 1 in 15) where I donot get several of those items and it happens randomly. And when I check for that key using GetObject, I get that item. I have 3 aerospike hosts. I am not able to figure out the issue. Please help.

Can it be a problem in the BatchPolicy or in the problem of the way hosts are set up ?

[khaf]

The client will return partial results if one or more of the nodes encounter a problem and the results are not fully returned. You should get an error in those situations along with the partial results.

[kartavya-ramnani]

Yes, But before this, I had aerospike-client-go 2.7.0 and I have never faced this issue. Now, that I have migrated to 3.0.0 and we started testing, this issue has started coming up for BatchGet when I am getting partial results even though the value exists and I can get them when I am doing normal get.

Can this be the reason why ? Note: There has been significant changes to clustering code. We recommend extensive testing before using in production.

[khaf]

There has been significant change to batch code in concert with the clustering, but all the tests (including long running ones) are passing. Where are you testing your code? Local or cloud? Is your cluster stable?

[kartavya-ramnani]

I downgraded from v3.0.0 to v2.12.0 and things started working fine. We use aerospike server version 4.5 (current is 5.0, i am guessing?), so maybe that is why the issue has been happening ?

Can we have v2.12.1 which has the fix for this : https://github.com/aerospike/aerospike-client-go/issues/301 as this fix got introduced in v3.0.1 ?

[khaf]

Unfortunately tagging v2.12 is not possible with out current setup. Can you share more info about your current setup so that I can fix the problem in the library?

[kartavya-ramnani]

We are using aerospike server version 4.5 on our data centre. I did not observe this issue running on my local machine but when I deployed the code to beta environment which has 3 aerospike hosts, this issue started happening where batchGetObjects started returning partial results for 1 in 15 requests. The weird part was the partial results were different everytime, sometimes some keys were missing, sometimes others.

The getObject was running perfectly and returning value when I used to search for the missing keys in the partial results from BatchGetObjects.

Please let me know what more information I can share. I was looking forward to upgrade my code to v3.0.0.

[khaf]

Are you also getting an error with the partial results? How many Keys are you passing? Do you think you can post a gist that fairly represents your use case?

[kartavya-ramnani]

1. Not getting an error, only getting the bool slice with false values for the keys which were not found.

2. I have 3 hosts, I pass 128 keys and a slice of interface{} (toCast) (pointer to an empty custom struct) and call BatchGetObjects.

myStructList := make([]*MyStruct, len(keysList))
toCast := make([]interface{}, len(keysList))
for i := range toCast {
    myStruct := &MyStruct{}
    toCast[i] = myStruct
    myStructList[i] = myStruct
}

And call BatchGetObjects :

isFoundList, err := as.BatchGetObjects(nil, keysList, toCast) // getting nil err

for isFound, record := range isFoundList {
    if record == false {
               // received partial result, do something about it. 
    }
}

[khaf]

OK, it looks like this may be related to the https://github.com/aerospike/aerospike-client-go/issues/305 Can you check and confirm?

[khaf]

This should be fixed in v3.0.3.