search

aerospike / aerospike-client-java

Aerospike Java Client Library
Other
236 stars 212 forks source link

Error Code 4: Parameter error when using round-robin reverse-proxy #63

Closed markterm closed 8 years ago

markterm commented 8 years ago

We have an Aerospike installation with a round-robin reverse-proxy in front of the nodes (this is in practice a Kubernetes service).

This sometimes works, but for the same request often returns: com.aerospike.client.AerospikeException: Error Code 4: Parameter error at com.aerospike.client.command.MultiCommand.parseGroup(MultiCommand.java:96) at com.aerospike.client.command.MultiCommand.parseResult(MultiCommand.java:71) at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:57) at com.aerospike.client.query.QueryExecutor$QueryThread.run(QueryExecutor.java:137)

If I reconfigure it so that the Kubernetes service only points to a single node then it works fine.

I am using version 3.2.4 of the Java client and 3.9.1 of the server. This is very similar to this issue: https://github.com/aerospike/aerospike-client-go/issues/97

wchu-citrusleaf commented 8 years ago

For all Parameter Errors, there will be a WARN line in the Aerospike Server giving more detail. This will help troubleshoot the cause.

markterm commented 8 years ago

Here you go:

Sep 08 2016 10:07:06 GMT: INFO (drv_ssd): (drv_ssd.c:2072) device /dev/sdb: used 53382991616, contig-free 456118M (3648950 wblocks), swb-free 3, w-q 0 w-tot 466307 (0.0/s), defrag-q 0 defrag-tot 19082 (0.0/s) defrag-w-tot 1866 (0.0/s)

Sep 08 2016 10:07:13 GMT: INFO (scan): (scan.c:607) starting basic scan job 6203919797479063517 {default:validated-profile} priority 2, sample-pct 100

Sep 08 2016 10:07:13 GMT: INFO (scan): (scan.c:607) starting basic scan job 6203919797479063517 {default:validated-profile} priority 2, sample-pct 100

Sep 08 2016 10:07:13 GMT: WARNING (scan): (scan.c:611) basic scan job 6203919797479063517 failed to start (4)

Sep 08 2016 10:07:14 GMT: WARNING (scan): (scan.c:345) send error - fd 142 sz 153706 rv 140406

(or screenshot Hipster-stylee: https://snag.gy/6W1Rhi.jpg)

wchu-citrusleaf commented 8 years ago

It looks like somehow you have exactly the same scan job submitted as the same trid (6203919797479063517) consecutively. Thus the second instance returns with PARAM_ERROR (4)

markterm commented 8 years ago

Ah, the call is from Aerospark, I'll have a look into what's going on at that side.