EntityMemcache$Bucket.class getNextToStore NEGATIVE hot key / high qps against memcache

[GoogleCodeExporter]

What steps will reproduce the problem?
1.  Annotate entity classes with Cache=600
2.  Use global cache in builder for gets/puts: ex.: Objectify ofy = 
factory().begin().cache( true ).consistency( Consistency.STRONG ).deadline( 
10.0 );
3.  Get an entity that doesn't exist in memcache ex.: AppVersion result = 
ofy.load().type( AppVersion.class ).id( appVersion ).get();

What is the expected output? What do you see instead?
Expect low QPS against memcache and no hot key issues.  Instead google 
engineers report that, while under load, we are generating extremely high QPS 
for a single key with value: 'NEGATIVE'. The problem is that we can't figure 
out what the key is, i.e.: 04shWKZIz0T1peslrYrCd/ELFoc=

What version of the product are you using? On what operating system? 4.0b1/GAE

Please provide any additional information below.

Original issue reported on code.google.com by rxre...@gmail.com on 30 Apr 2013 at 12:32

Attachments:

• [Screen Shot 2013-04-29 at 6.32.00 PM.png](https://storage.googleapis.com/google-code-attachments/objectify-appengine/issue-157/comment-0/Screen Shot 2013-04-29 at 6.32.00 PM.png)

[GoogleCodeExporter]

Objectify caches negative results, the value will be 'NEGATIVE'. As with 
everything in the Objectify cache, the key is whatever memcache does when you 
use the native datastore Key object.

If you are seeing lots of requests for a single memcache key, it's because your 
code is making lots of requests for that key. 'NEGATIVE' indicates that there 
is nothing in the datastore; without the caching, you would be making fetches 
to the datastore.

Original comment by lhori...@gmail.com on 30 Apr 2013 at 3:05

[GoogleCodeExporter]

Thanks for the quick response.  We do have a few entities that could go 
considered as "global" objects which we have moved in to sharded memcache keys, 
however this appears to be one we missed.  

This may not be an objectify problem, but would you by chance have any 
suggestions on how we could decode the key in to something usable?  Or, hook in 
to either the Objectify API, or underlying memcache API and get an idea of 
which calls are hammering a single key?

Thanks for your time and help. 

Original comment by rxre...@gmail.com on 30 Apr 2013 at 3:18

[GoogleCodeExporter]

Unfortunately that key is actually a sha1 hash of the java-serialized key, 
which is what GAE's memcache service does for random object keys. This is not a 
particularly good strategy so thank you for calling that to my attention.

If you checkout and build master, Objectify will stringify the keys before 
putting them in memcache. With this you will be able to determine exactly what 
key is being stored.

For further discussion, please post to the objectify google group.

Original comment by lhori...@gmail.com on 30 Apr 2013 at 7:26

• Changed state: Invalid