Open marian-code opened 3 months ago
OK so here's an update I went into admin console and in supermarket category I added category-deli
key after that I tried importing Food again. I got different error:
{'selected_version': 'sk', 'selected_datatypes': {'food': {'name': 'food', 'selected': True}, 'unit': {'name': 'unit', 'selected': False}, 'category': {'name': 'category', 'selected': False}, 'property': {'name': 'property', 'selected': False}, 'store': {'name': 'store', 'selected': False}, 'conversion': {'name': 'conversion', 'selected': False}}, 'update_existing': True, 'use_metric': True}
Internal Server Error: /api-import-open-data/
Traceback (most recent call last):
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
return super().execute(query, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.IntegrityError: UNIQUE constraint failed: cookbook_food.space_id, cookbook_food.name
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/recipes/venv/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
return view_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/views/generic/base.py", line 104, in view
return self.dispatch(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/opt/recipes/venv/lib/python3.12/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/opt/recipes/venv/lib/python3.12/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/cookbook/views/api.py", line 1601, in post
response_obj['food'] = data_importer.import_food().to_dict()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/cookbook/helper/open_data_importer.py", line 374, in import_food
Food.load_bulk(create_list, None)
File "/opt/recipes/venv/lib/python3.12/site-packages/treebeard/models.py", line 115, in load_bulk
node_obj = cls.add_root(**node_data)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/cookbook/models.py", line 165, in add_root
return super().add_root(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/treebeard/mp_tree.py", line 623, in add_root
return MP_AddRootHandler(cls, **kwargs).process()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/treebeard/mp_tree.py", line 341, in process
newobj.save()
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/base.py", line 814, in save
self.save_base(
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/base.py", line 877, in save_base
updated = self._save_table(
^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/base.py", line 1020, in _save_table
results = self._do_insert(
^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django_prometheus/models.py", line 43, in _do_insert
::ffff:172.23.0.3 - - [24/Mar/2024:14:32:16 +0100] "POST /api-import-open-data/ HTTP/1.0" 500 244461 "https://recipes.rynik.duckdns.org/space-manage/1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
return super()._do_insert(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/base.py", line 1061, in _do_insert
return manager._insert(
^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/query.py", line 1805, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
cursor.execute(sql, params)
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 102, in execute
return super().execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
return super().execute(query, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.IntegrityError: UNIQUE constraint failed: cookbook_food.space_id, cookbook_food.name
ERROR:django.request:Internal Server Error: /api-import-open-data/
Traceback (most recent call last):
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
return super().execute(query, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.IntegrityError: UNIQUE constraint failed: cookbook_food.space_id, cookbook_food.name
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/recipes/venv/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
return view_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/views/generic/base.py", line 104, in view
return self.dispatch(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/opt/recipes/venv/lib/python3.12/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/opt/recipes/venv/lib/python3.12/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/cookbook/views/api.py", line 1601, in post
response_obj['food'] = data_importer.import_food().to_dict()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/cookbook/helper/open_data_importer.py", line 374, in import_food
Food.load_bulk(create_list, None)
File "/opt/recipes/venv/lib/python3.12/site-packages/treebeard/models.py", line 115, in load_bulk
node_obj = cls.add_root(**node_data)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/cookbook/models.py", line 165, in add_root
return super().add_root(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/treebeard/mp_tree.py", line 623, in add_root
return MP_AddRootHandler(cls, **kwargs).process()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/treebeard/mp_tree.py", line 341, in process
newobj.save()
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/base.py", line 814, in save
self.save_base(
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/base.py", line 877, in save_base
updated = self._save_table(
^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/base.py", line 1020, in _save_table
results = self._do_insert(
^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django_prometheus/models.py", line 43, in _do_insert
return super()._do_insert(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/base.py", line 1061, in _do_insert
return manager._insert(
^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/query.py", line 1805, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
cursor.execute(sql, params)
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 102, in execute
return super().execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/recipes/venv/lib/python3.12/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
return super().execute(query, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.IntegrityError: UNIQUE constraint failed: cookbook_food.space_id, cookbook_food.name
After that I unchecked `update existing data and the import passed OK for Food.
The I tried importing Conversion whis now passed OK with only three errors:
{'selected_version': 'sk', 'selected_datatypes': {'food': {'name': 'food', 'selected': False}, 'unit': {'name': 'unit', 'selected': False}, 'category': {'name': 'category', 'selected': False}, 'property': {'name': 'property', 'selected': False}, 'store': {'name': 'store', 'selected': False}, 'conversion': {'name': 'conversion', 'selected': True}}, 'update_existing': True, 'use_metric': True}
Traceback (most recent call last):
File "/opt/recipes/cookbook/helper/open_data_importer.py", line 456, in import_conversion
food_id=self.slug_id_cache['food'][self.data[datatype][k]['food']],
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'food-banana'
food-banana is not in self.slug_id_cache["food"]
Traceback (most recent call last):
File "/opt/recipes/cookbook/helper/open_data_importer.py", line 456, in import_conversion
food_id=self.slug_id_cache['food'][self.data[datatype][k]['food']],
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
food-cilantro-raw is not in self.slug_id_cache["food"]
KeyError: 'food-cilantro-raw'
Traceback (most recent call last):
File "/opt/recipes/cookbook/helper/open_data_importer.py", line 456, in import_conversion
food_id=self.slug_id_cache['food'][self.data[datatype][k]['food']],
food-canned-coconut-milk is not in self.slug_id_cache["food"]
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'food-canned-coconut-milk'
I think this issue might be related to incomplete translations. This should be handled by the build script in the open data repo itself but maybe I missed something. Will need to look into this in more detail
Thanks for the response. If you need more details I can provide them. Or if you point in the right direction I can do some investigation too.
the problem is the conversion import. The list of conversion should only include conversions where the foods and units are in the language dataset. This is done by the build script https://github.com/TandoorRecipes/open-tandoor-data/blob/main/build.py
What I think is happening is that some foods are not translated for SK, thus they are left out of the dataset. The conversions shoudl detect that the foods or units are missing and not include the conversion but it seems that this is not the case.
That could be it, Deli key is the only one with missing value in the sk.json
Tandoor Version
1.5.15
Setup
Docker / Docker-Compose
Reverse Proxy
Nginx Proxy Manager (NPM)
Other
No response
Bug description
I'm trying to import tandor open data (sk) version but I consistently get these errors:
I tried importing (en) data, that works OK. I even started completely blank instance but the errors are the same.
I did the Open data translation for SK and I know I left some keys that do not have a coresponding version in Slovak language blank. Could this be the issue?
Relevant logs
Food import errors
Category import errors