.belongs() on GAE limited to 30 entries

[alfonsodg]

From gummywub...@gmail.com on January 19, 2011 20:37:11

What steps will reproduce the problem? 1. Use any table with >30 entries

2. Perform a db(table.id.belongs(desired_keys)).select() where desired_key is a list of >30 entries What is the expected output? What do you see instead? The current result is an exception: File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 1643, in init (MAX_ALLOWABLE_QUERIES, len(bound_queries))) BadArgumentError: Cannot satisfy query -- too many subqueries (max: 30, got 57). Probable cause: too many IN/!= filters in query. What version of the product are you using? On what operating system? 1.89.5 Please provide any additional information below. GAE documentation ( http://code.google.com/appengine/docs/python/datastore/queries.html#Restrictions_on_Queries ) "A single query containing != or IN operators is limited to 30 sub-queries."

Apparently (from some cursory research), the db.get() and related methods do not have this limitation. http://code.google.com/appengine/docs/python/datastore/functions.html Thanks, :R

Original issue: http://code.google.com/p/web2py/issues/detail?id=160

[alfonsodg]

From massimo....@gmail.com on May 04, 2011 19:49:16

I do not think there is anything we can do about this. Please correct me if I am wrong.

Status: Invalid

[alfonsodg]

From gummywub...@gmail.com on May 21, 2011 13:56:40

Well, what I've done is wrap the calls that use belongs. If it's >n, then I break it into constituent parts and re-assemble. Since my wrapper is a bit hacky, I lose the Rows() type return value and just have a simple list. If something were done like this internal to DAL on GAE, it would be much more transparent.

This is really much like handling list: types on non-GAE. Logic in the DAL layer implements what the native DB doesn't.

Can I and others work around it? Sure. I think it would make your product better.

[alfonsodg]

From pallav.n...@gmail.com on January 06, 2012 13:12:18

I'm also interested in updates on this issue. Yay/nay?

[alfonsodg]

From in4tu...@gmail.com on July 19, 2012 20:01:51

Here is a hack that I currently use:

    MAX_NUM_BELONGS_QUERY_GAE_BATCH_SIZE = 30
    def gae_belongs_only_query(db, table, field, belongs_to_list, selections, batch_size=MAX_NUM_BELONGS_QUERY_GAE_BATCH_SIZE):
        """ Restricted implementation to include only a single query that has a "belongs".
            The select part (selections) is unrestricted.
            Returns a list of Rows.
        """
        results = []
        assert batch_size <= MAX_NUM_BELONGS_QUERY_GAE_BATCH_SIZE
        for i in range(0, len(belongs_to_list), batch_size):
            vs = belongs_to_list[i: i + batch_size]
            rs = db(table[field].belongs(vs)).select(selections)
            results += rs
        return results

        results = gae_belongs_only_query(db, db.mytable, 'id', myids, db.mytable.ALL)