search

doableware / djongo

Django and MongoDB database connector
https://www.djongomapper.com
GNU Affero General Public License v3.0
1.86k stars 351 forks source link

obj.exists() - SQLDecode Error, can't work with DRF. #659

Open dinesh-hills opened 1 year ago

dinesh-hills commented 1 year ago

One line description of the issue

.exists method blows up django app, and this happens a lot when DRF validator execute def qs_exists() to check for unique constraints.

Python script

```python >>> Users.objects.filter(pk=1).exists() ``` #### Traceback Traceback (most recent call last): File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/djongo/sql2mongo/query.py", line 808, in __iter__ yield from iter(self._query) File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/djongo/sql2mongo/query.py", line 167, in __iter__ yield self._align_results(doc) File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/djongo/sql2mongo/query.py", line 269, in _align_results if selected.table == self.left_table: File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/djongo/sql2mongo/sql_tokens.py", line 133, in table return alias2token[name].table File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/djongo/sql2mongo/sql_tokens.py", line 133, in table return alias2token[name].table File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/djongo/sql2mongo/sql_tokens.py", line 133, in table return alias2token[name].table [Previous line repeated 957 more times] File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/djongo/sql2mongo/sql_tokens.py", line 130, in table name = self.given_table File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/djongo/sql2mongo/sql_tokens.py", line 141, in given_table name = self._token.get_real_name() File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/sqlparse/sql.py", line 361, in get_real_name return self._get_first_name(dot_idx) File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/sqlparse/sql.py", line 386, in _get_first_name return token.get_name() File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/sqlparse/sql.py", line 355, in get_name return self.get_alias() or self.get_real_name() File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/sqlparse/sql.py", line 344, in get_alias _, ws = self.token_next_by(t=T.Whitespace) File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/sqlparse/sql.py", line 244, in token_next_by return self._token_matching(funcs, idx, end) File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/sqlparse/sql.py", line 223, in _token_matching if func(token): File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/sqlparse/sql.py", line 242, in funcs = lambda tk: imt(tk, i, m, t) File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/sqlparse/utils.py", line 100, in imt elif types and any(token.ttype in ttype for ttype in types): File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/sqlparse/utils.py", line 100, in elif types and any(token.ttype in ttype for ttype in types): File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/sqlparse/tokens.py", line 19, in __contains__ return item is not None and (self is item or item[:len(self)] == self) RecursionError: maximum recursion depth exceeded in comparison The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/djongo/cursor.py", line 76, in fetchone return self.result.next() File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/djongo/sql2mongo/query.py", line 797, in __next__ result = next(self._result_generator) File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/djongo/sql2mongo/query.py", line 830, in __iter__ raise exe from e djongo.exceptions.SQLDecodeError: Keyword: FAILED SQL: SELECT %(0)s AS "a" FROM "org_organization" WHERE "org_organization"."id" = %(1)s LIMIT 1 Params: (1, 1) Version: 1.3.6 Sub SQL: None FAILED SQL: None Params: None Version: None The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/django/db/utils.py", line 98, in inner return func(*args, **kwargs) File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/djongo/cursor.py", line 81, in fetchone raise db_exe from e djongo.database.DatabaseError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 1, in File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/django/db/models/query.py", line 1226, in exists return self.query.has_results(using=self.db) File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/django/db/models/sql/query.py", line 592, in has_results return compiler.has_results() File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1366, in has_results return bool(self.execute_sql(SINGLE)) File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1409, in execute_sql val = cursor.fetchone() File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/django/db/utils.py", line 97, in inner with self: File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/django/db/utils.py", line 98, in inner return func(*args, **kwargs) File "/home/dinesh/.local/share/virtualenvs/ProjectHQ-CRbS596i/lib/python3.10/site-packages/djongo/cursor.py", line 81, in fetchone raise db_exe from e django.db.utils.DatabaseError
f754699 commented 1 year ago

我也是这个问题

boxxello commented 1 year ago

I'm having the same issue in filtering by string fields with nested subqueries using the ORM + exists() (instead of pipeline+mongo_aggregate() which obviously works fine), if needed I can also provide my stacktrace.

tvandraren commented 1 year ago

Having a somewhat similar issue trying to create a model from the admin site

[12/May/2023 22:43:44] "GET /admin/gramnerk/traduccion/add/ HTTP/1.1" 200 13927
Internal Server Error: /admin/gramnerk/traduccion/add/
Traceback (most recent call last):
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/djongo/sql2mongo/query.py", line 808, in __iter__
    yield from iter(self._query)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/djongo/sql2mongo/query.py", line 167, in __iter__
    yield self._align_results(doc)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/djongo/sql2mongo/query.py", line 269, in _align_results
    if selected.table == self.left_table:
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/djongo/sql2mongo/sql_tokens.py", line 133, in table
    return alias2token[name].table
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/djongo/sql2mongo/sql_tokens.py", line 133, in table
    return alias2token[name].table
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/djongo/sql2mongo/sql_tokens.py", line 133, in table
    return alias2token[name].table
  [Previous line repeated 914 more times]
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/djongo/sql2mongo/sql_tokens.py", line 130, in table
    name = self.given_table
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/djongo/sql2mongo/sql_tokens.py", line 141, in given_table
    name = self._token.get_real_name()
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/sqlparse/sql.py", line 361, in get_real_name
    return self._get_first_name(dot_idx)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/sqlparse/sql.py", line 386, in _get_first_name
    return token.get_name()
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/sqlparse/sql.py", line 355, in get_name
    return self.get_alias() or self.get_real_name()
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/sqlparse/sql.py", line 344, in get_alias
    _, ws = self.token_next_by(t=T.Whitespace)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/sqlparse/sql.py", line 244, in token_next_by
    return self._token_matching(funcs, idx, end)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/sqlparse/sql.py", line 223, in _token_matching
    if func(token):
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/sqlparse/sql.py", line 242, in <lambda>
    funcs = lambda tk: imt(tk, i, m, t)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/sqlparse/utils.py", line 100, in imt
    elif types and any(token.ttype in ttype for ttype in types):
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/sqlparse/utils.py", line 100, in <genexpr>
    elif types and any(token.ttype in ttype for ttype in types):
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/sqlparse/tokens.py", line 19, in __contains__
    return item is not None and (self is item or item[:len(self)] == self)
RecursionError: maximum recursion depth exceeded in comparison

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/djongo/cursor.py", line 76, in fetchone
    return self.result.next()
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/djongo/sql2mongo/query.py", line 797, in __next__
    result = next(self._result_generator)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/djongo/sql2mongo/query.py", line 830, in __iter__
    raise exe from e
djongo.exceptions.SQLDecodeError: 

        Keyword: FAILED SQL: SELECT %(0)s AS "a" FROM "gramnerk_lengua" WHERE "gramnerk_lengua"."id" = %(1)s LIMIT 1
Params: (1, 4)
Version: 1.3.6
        Sub SQL: None
        FAILED SQL: None
        Params: None
        Version: None

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/db/utils.py", line 98, in inner
    return func(*args, **kwargs)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/djongo/cursor.py", line 81, in fetchone
    raise db_exe from e
djongo.database.DatabaseError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/contrib/admin/options.py", line 686, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 134, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/views/decorators/cache.py", line 62, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/contrib/admin/sites.py", line 242, in inner
    return view(request, *args, **kwargs)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1891, in add_view
    return self.changeform_view(request, None, form_url, extra_context)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 46, in _wrapper
    return bound_method(*args, **kwargs)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/utils/decorators.py", line 134, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1751, in changeform_view
    return self._changeform_view(request, object_id, form_url, extra_context)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1797, in _changeform_view
    form_validated = form.is_valid()
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/forms/forms.py", line 205, in is_valid
    return self.is_bound and not self.errors
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/forms/forms.py", line 200, in errors
    self.full_clean()
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/forms/forms.py", line 439, in full_clean
    self._post_clean()
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/forms/models.py", line 492, in _post_clean
    self.instance.full_clean(exclude=exclude, validate_unique=False)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/db/models/base.py", line 1448, in full_clean
    self.clean_fields(exclude=exclude)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/db/models/base.py", line 1500, in clean_fields
    setattr(self, f.attname, f.clean(raw_value, self))
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/db/models/fields/__init__.py", line 755, in clean
    self.validate(value, model_instance)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/db/models/fields/related.py", line 1088, in validate
    if not qs.exists():
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1226, in exists
    return self.query.has_results(using=self.db)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/db/models/sql/query.py", line 592, in has_results
    return compiler.has_results()
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1366, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1409, in execute_sql
    val = cursor.fetchone()
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/db/utils.py", line 97, in inner
    with self:
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/django/db/utils.py", line 98, in inner
    return func(*args, **kwargs)
  File "/home/vandraren/repos/khunlyngramnerk/venv/lib/python3.10/site-packages/djongo/cursor.py", line 81, in fetchone
    raise db_exe from e
django.db.utils.DatabaseError
[12/May/2023 22:43:50] "POST /admin/gramnerk/traduccion/add/ HTTP/1.1" 500 3844122

with

djongo==1.3.6
pymongo[srv]>=3.12,<3.13
Django>=4.1,<4.2
KonRatt commented 10 months ago

I guess !547 might solve this issue