Shard counter will sometimes return a string value instead of int

[GoogleCodeExporter]

What steps will reproduce the problem?
1. When the get_count() method is called on the shard counter, it will 
occasionally return a string value instead of an int value. The string value is 
always "1" and does not correspond to the value of the counter in general.
2. I have not been able to determine when this error will occur. It appears to 
only happen while deployed and not locally.

What is the expected output? What do you see instead?

When the get_count() method is called on the shard counter, it will 
occasionally return a string value instead of an int value. The string value is 
always "1" and does not correspond to the value of the counter in general.

What version of the product are you using? On what operating system?

App Engine 1.5.2 using Python

Please provide any additional information below.

There is a str() cast in the base code that I removed in my version. Why is 
there a cast to a string?

Original issue reported on code.google.com by scott.le...@gmail.com on 21 Jul 2011 at 10:47

[GoogleCodeExporter]

This is my version of the shardcounter.py of these samples. I added a 
BackupGeneralCounter that is a non-sharded counter to help track when the shard 
counter failed. In on the line of the original:

memcache.add(name, str(total), 60)

I removed the str() cast because the first time that the function would be 
called it would return an int. The second time it would return a string. 
Perhaps this is tied to the problem?

Original comment by scott.le...@gmail.com on 25 Jul 2011 at 9:48

Attachments:

• shardcounter.py

[GoogleCodeExporter]

More information about the problem:

My code is different from the code on the site because I removed the str() cast 
from that line. I initially did this because the first call to get_count would 
return an integer and the second would be a string (because the value was 
stored as a string in the memcache). This simple case has been fixed, but 
occasionally the value will be returned as a string even though it is never (as 
far as I can tell) stored as a string and the value for the string does not 
match the values that get_count() has been returning. (Which prevents us from 
simply casting the return value to an int and using that instead.)

Thank you for the help!

Original comment by scott.le...@gmail.com on 28 Jul 2011 at 12:30

[GoogleCodeExporter]

Can you reproduce the problem consistently? Does it occur only on appspot.com? 
Or also in the devappserver?

Original comment by fredsa@google.com on 28 Jul 2011 at 5:41

[GoogleCodeExporter]

The problem occurs about 1 out of 10 or 20 times I run the application. I do 
not have repro steps, otherwise I would happily list them. I've only seen the 
problem on appspot.com.

Original comment by scott.le...@gmail.com on 28 Jul 2011 at 9:20

[GoogleCodeExporter]

There are a couple of things going on here:

1. Unfortunately, it looks like we introduced a bug into the sharded counter 
article (http://code.google.com/appengine/articles/sharding_counters.html) 
around March of this year.

   We incorrectly modified the last line of the increment(name) method to:
      memcache.incr(name, initial_value=0)

   We should've left it the original code, without the initial_value which was:
      memcache.incr(name)

2. In Python, if you pass an initial_value to the memcache.incr() method and 
the key does not exist, then two things happen:
   a) The value stored in memcache is a str
   b) The memcache.incr() return value is long
   This is documented in issue http://code.google.com/p/googleappengine/issues/detail?id=2012

   Note, however, that this behavior was fixed a while back in the Java runtime. The stored value and returned value are both java.lang.Long, which is what you'd expect.

--------------------------

Combined #1 and #2 mean that, once the memcache key expires (the article uses 
60 seconds), and your code calls increment(name), then the memcache value for 
that counter is set to '1' (a str) rather than 1 (a long). So, the bug in the 
sample code is that the memcache counter will sometimes, or often, depending on 
your traffic, reset to '1', which is not the correct count. However, the 
sharded datastore counter does continue to count correctly.

I'll get the article corrected

Original comment by fredsa@google.com on 29 Jul 2011 at 10:28

[GoogleCodeExporter]

Excellent! Thank you for your help!

-Scott-

Original comment by scott.le...@gmail.com on 30 Jul 2011 at 10:44

[GoogleCodeExporter]

The documentation was update on August 1st 2011.

The samples were updated on August 3rd 2011.

Original comment by fredsa@google.com on 22 Aug 2011 at 9:25

• Changed state: Fixed