add support for partial indexes

[timgraham]

refs #104

[timgraham]

After e9711203ac720f34ce14ecb53489828887c4ef64, simple conditions no longer work, but we may need a separate path for generating this MQL anyway.

======================================================================
FAIL: test_boolean_restriction_partial (indexes.tests.PartialIndexTests.test_boolean_restriction_partial)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/code/django/tests/indexes/tests.py", line 453, in test_boolean_restriction_partial
    self.assertEqual(
AssertionError: "{'published': {'$eq': True}}" != "{'parent__field__0': {'$eq': True}}"
- {'published': {'$eq': True}}
+ {'parent__field__0': {'$eq': True}}

[WaVEV]

After e971120, simple conditions no longer work, but we may need a separate path for generating this MQL anyway.

======================================================================
FAIL: test_boolean_restriction_partial (indexes.tests.PartialIndexTests.test_boolean_restriction_partial)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/code/django/tests/indexes/tests.py", line 453, in test_boolean_restriction_partial
    self.assertEqual(
AssertionError: "{'published': {'$eq': True}}" != "{'parent__field__0': {'$eq': True}}"
- {'published': {'$eq': True}}
+ {'parent__field__0': {'$eq': True}}

Fixed, the columns are built without "alias" so the Col.as_mql fails.

[timgraham]

Can the following be supported? If not, we can edit the test with a condition that is supported.

======================================================================
ERROR: test_multiple_conditions (indexes.tests.PartialIndexTests.test_multiple_conditions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/code/django/django/test/testcases.py", line 1457, in skip_wrapper
    return test_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django/tests/indexes/tests.py", line 486, in test_multiple_conditions
    sql = str(index._get_condition_mql(Article, schema_editor=editor))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django-mongodb/django_mongodb/indexes.py", line 10, in _get_condition_mql
    return where.as_mql_idx(compiler, schema_editor.connection)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django-mongodb/django_mongodb/query.py", line 316, in where_node_idx
    mql = child.as_mql_idx(compiler, connection)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django-mongodb/django_mongodb/lookups.py", line 26, in builtin_lookup_idx
    return connection.mongo_operators_idx[self.lookup_name](lhs_mql, value)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
KeyError: 'contains'

Also, please add tests/indexes_ in this repo so we can make sure we have complete test coverage for the new index MQL generation.

[timgraham]

Maybe it makes sense to put the as_mql_idx methods in indexes.py?

[WaVEV]

Found a bug in where.as_mql (should rise EmptyResultSet), and also I have to research in which way the test from constraints will be adapted, it isn't straightforward

[timgraham]

Oh, I didn't notice the constraints problems you mentioned previously. I added a Django fork commit to place isnull usage in UniqueConstraint.condition tests, but other problems like this remain which I think it what you're referring to:

======================================================================
ERROR: test_validate_expression_condition (constraints.tests.UniqueConstraintTests.test_validate_expression_condition)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/code/django/django/test/testcases.py", line 1457, in skip_wrapper
    return test_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django/tests/constraints/tests.py", line 1014, in test_validate_expression_condition
    constraint.validate(UniqueConstraintProduct, non_unique_product)
  File "/home/tim/code/django/django/db/models/constraints.py", line 461, in validate
    if (self.condition & Exists(queryset.filter(self.condition))).check(
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django/django/db/models/query_utils.py", line 140, in check
    return compiler.execute_sql(SINGLE) is not None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django-mongodb/django_mongodb/compiler.py", line 246, in execute_sql
    query = self.build_query(
            ^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django-mongodb/django_mongodb/compiler.py", line 379, in build_query
    query.project_fields = self.get_project_fields(columns, ordering_fields)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django-mongodb/django_mongodb/compiler.py", line 563, in get_project_fields
    fields.update(fields.pop(self.collection_name, {}))
                             ^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django/django/utils/functional.py", line 47, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django-mongodb/django_mongodb/compiler.py", line 438, in collection_name
    base_table = next(
                 ^^^^^
StopIteration

The SQL:

SELECT 1 AS "_check"
WHERE COALESCE((EXISTS
                  (SELECT 1 AS "a"
                   FROM "constraints_uniqueconstraintconditionproduct" U0
                   WHERE (U0."name" = 'p1'
                          AND U0."color" IS NULL)
                   LIMIT 1)), 1);

I think it fails because in MongoDB, we have no notion of executing a query without a collection.

I wonder if we could make it work without the outer query? But it looks tricky because I don't see a place we could hook into to customize the query: https://github.com/django/django/blob/042b381e2e37c0c37b8a8f6cc9947f1a2ebfa0dd/django/db/models/query_utils.py#L117-L151

[WaVEV]

Oh, I didn't notice the constraints problems you mentioned previously. I added a Django fork commit to place isnulll usage in UniqueConstraint.condition tests, but other problems like this remain which I think it what you're referring to:

======================================================================
ERROR: test_validate_expression_condition (constraints.tests.UniqueConstraintTests.test_validate_expression_condition)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/code/django/django/test/testcases.py", line 1457, in skip_wrapper
    return test_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django/tests/constraints/tests.py", line 1014, in test_validate_expression_condition
    constraint.validate(UniqueConstraintProduct, non_unique_product)
  File "/home/tim/code/django/django/db/models/constraints.py", line 461, in validate
    if (self.condition & Exists(queryset.filter(self.condition))).check(
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django/django/db/models/query_utils.py", line 140, in check
    return compiler.execute_sql(SINGLE) is not None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django-mongodb/django_mongodb/compiler.py", line 246, in execute_sql
    query = self.build_query(
            ^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django-mongodb/django_mongodb/compiler.py", line 379, in build_query
    query.project_fields = self.get_project_fields(columns, ordering_fields)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django-mongodb/django_mongodb/compiler.py", line 563, in get_project_fields
    fields.update(fields.pop(self.collection_name, {}))
                             ^^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django/django/utils/functional.py", line 47, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/home/tim/code/django-mongodb/django_mongodb/compiler.py", line 438, in collection_name
    base_table = next(
                 ^^^^^
StopIteration

The SQL:

SELECT 1 AS "_check"
WHERE COALESCE((EXISTS
                  (SELECT 1 AS "a"
                   FROM "constraints_uniqueconstraintconditionproduct" U0
                   WHERE (U0."name" = 'p1'
                          AND U0."color" IS NULL)
                   LIMIT 1)), 1);

I think it fails because in MongoDB, we have no notion of executing a query without a collection.

I wonder if we could make it work without the outer query? But it looks tricky because I don't see a place we could hook into to customize the query: https://github.com/django/django/blob/042b381e2e37c0c37b8a8f6cc9947f1a2ebfa0dd/django/db/models/query_utils.py#L117-L151

I added like a dummy collection to handle this and created the pr fixing the remaining unit tests. https://github.com/mongodb-forks/django/pull/11