Future projection sometimes returns ten items with values of None

[GoogleCodeExporter]

What steps will reproduce the problem?

When using the Testbed framework, create a future projection, e.g.

class Model(ndb.Model): name = ndb.StringProperty(indexed=True) hash = ndb.StringProperty(indexed=True)

elsewhere ... populate Model

COUNT=500

future_kods = Model.query().fetch_async(COUNT, projection=[Model.name, Model.hash] )

models = future_kods.get_result()

for model in models: print "name: % hash: %s" % (name, hash)

What is the expected output? What do you see instead?

Around 95% of the time the above entirely works exactly as expected.

Around 5% of the time (i.e. approximately every one in twenty executions), on a 
model set of around 65 items, approximately 10 of the models will have empty 
results.

Note that I am using unit testing from the command line, so between calls the 
process is stopped and started.

As an example, suppose the model set correlates to the alphabet, like this: The 
'name' is the alphabet (A-Z) and hash is a number 1-26, this is the output 
approximately 95% of the time (and pardon the length, but I think it's 
illuminating):

name: A hash: 1
name: B hash: 2
name: C hash: 3
name: D hash: 4
name: E hash: 5
name: F hash: 6
name: G hash: 7
name: H hash: 8
name: I hash: 9
name: J hash: 10
name: K hash: 11
name: L hash: 12
name: M hash: 13
name: N hash: 14
name: O hash: 15
name: P hash: 16
name: Q hash: 17
name: R hash: 18
name: S hash: 19
name: T hash: 20
name: U hash: 21
name: V hash: 22
name: W hash: 23
name: X hash: 24
name: Y hash: 25
name: Z hash: 26

On the occasion where the issue occurs, one gets something like this:

name: A hash: 1
name: B hash: 2
name: None hash: None
name: D hash: 4
name: None hash: None
name: F hash: 6
name: G hash: 7
name: None hash: None
name: I hash: 9
name: J hash: 10
name: K hash: 11
name: L hash: 12
name: M hash: 13
name: None hash: None
name: O hash: 15
name: P hash: 16
name: None hash: None
name: None hash: None
name: None hash: None
name: T hash: 20
name: U hash: 21
name: V hash: 22
name: None hash: None
name: None hash: None
name: Y hash: 25
name: None hash: None

There are some insights that may be of assistance:
1. The problem randomly occurs - if I repeat the test, it consistently occurs, 
unpredictably;
2. The return values without results appear randomly selected among the dataset;
3. They number of items without results always number ten.*

* At least in my test cases, which have not been the alphabet as above, but 
number around 64 items.

What version of the product are you using? On what operating system?

$ cat VERSION
release: "1.7.5"
timestamp: 1357690550
api_versions: ['1']
supported_api_versions:
  python:
    api_versions: ['1']
  python27:
    api_versions: ['1']
  go:
    api_versions: ['go1']

Please provide any additional information below.

I am happy to provide more information on request.

Original issue reported on code.google.com by brianmh...@gmail.com on 25 Feb 2013 at 8:10

[GoogleCodeExporter]

I have some more information on this issue, which may be illuminating.

I can reproduce this issue with:

     models = Model.query().iter(
            projection=[Model.name, Model.hash]
            )

     # or
     models = Model.query().iter()

     # or
     models = Model.query()  

In other words, the issue does not seem to be specific to projections or 
asynchronous behaviour.

It may be that the problem is one of consistency in the HRD.

Perhaps I could test this if there were a way to modify the consistency policy 
(--datastore_consistency_policy) in Testbed unit tests?

It isn't clear how to pass a consistency option (eg 
`TimeBasedHRConsistencyPolicy` or `PseudoRandomHRConsistencyPolicy` from 
`google.appengine.datastore.datastore_stub_util`) to Testbed's 
`init_datastore_v3_stub()`. A little assistance would be greatly appreciated!

I will continue to muck around and see if I can determine the underlying 
problem.

Original comment by brianmh...@gmail.com on 27 Feb 2013 at 5:35

[GoogleCodeExporter]

I also had an issue with NDB projections, but it was while running the 
dev_appserver (did not try with testteb).

A simple query:
User.query(User.login >= req).fetch(5, projection=[User.login])

correctly found the users but their "login" property was None.

Original comment by ale...@freshplanet.com on 27 Mar 2013 at 4:06

[GoogleCodeExporter]

I eventually made a fairly elaborate set of test cases and concluded that the 
issue that was exhibiting came was derived from the consistency policy (which 
is an excellent endorsement of the choice to have the consistency policy follow 
that at deployment).

The dev appserver was working as designed, though it would still be good to be 
able to programatically change the consistency policy when unit testsing.

Original comment by brianmh...@gmail.com on 27 Mar 2013 at 4:18

[GoogleCodeExporter]

Since it has been distilled down to a problem with consistency, the title of 
this issue might better read "add the option to change the db consistency 
policy in unit tests"

Original comment by brianmh...@gmail.com on 9 Apr 2013 at 2:04