Cannot delete workflow

[homework36]

everything is working according to docker output but ...

It's possible that it's due to the tiny server size. But there might be something else going on. We are still not able to have a larger vGPU instance as of today (4/6/24). At the meantime, I launched another small vGPU with Ubuntu 20.04 to test compatibility with another potential OS version.

Error message inside rodan-main:

rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    | ERROR Internal Server Error: /api/project/b7f62369-53ef-4f8c-a294-70826ed30a82/
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    | Traceback (most recent call last):
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     return self.cursor.execute(sql, params)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    | psycopg2.errors.InternalError_: unexpected chunk number 2 (expected 1) for toast value 25525 in pg_toast_2619
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    | The above exception was the direct cause of the following exception:
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    | Traceback (most recent call last):
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35, in inner
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     response = get_response(request)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 128, in _get_response
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     response = self.process_exception_by_middleware(e, request)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 126, in _get_response
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     return view_func(*args, **kwargs)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/views/generic/base.py", line 69, in view
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     return self.dispatch(request, *args, **kwargs)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 505, in dispatch
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     response = self.handle_exception(exc)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 465, in handle_exception
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     self.raise_uncaught_exception(exc)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     raise exc
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 502, in dispatch
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     response = handler(request, *args, **kwargs)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/rest_framework/generics.py", line 282, in get
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     return self.retrieve(request, *args, **kwargs)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/rest_framework/mixins.py", line 54, in retrieve
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     instance = self.get_object()
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/rest_framework/generics.py", line 99, in get_object
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     self.check_object_permissions(self.request, obj)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 343, in check_object_permissions
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     if not permission.has_object_permission(request, self, obj):
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/rest_framework/permissions.py", line 283, in has_object_permission
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     if not user.has_perms(perms, obj):
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/contrib/auth/models.py", line 274, in has_perms
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     return all(self.has_perm(perm, obj) for perm in perm_list)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/contrib/auth/models.py", line 274, in <genexpr>
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     return all(self.has_perm(perm, obj) for perm in perm_list)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/contrib/auth/models.py", line 267, in has_perm
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     return _user_has_perm(self, perm, obj)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/contrib/auth/models.py", line 181, in _user_has_perm
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     if backend.has_perm(user, perm, obj):
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/guardian/backends.py", line 97, in has_perm
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     return check.has_perm(perm, obj)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/guardian/core.py", line 78, in has_perm
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     return perm in self.get_perms(obj)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/guardian/core.py", line 161, in get_perms
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     perms = list(set(chain(user_perms, group_perms)))
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 272, in __iter__
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     self._fetch_all()
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1182, in _fetch_all
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     self._result_cache = list(self._iterable_class(self))
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 178, in __iter__
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     for row in compiler.results_iter(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size):
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1019, in results_iter
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch, chunk_size=chunk_size)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1068, in execute_sql
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     cursor.execute(sql, params)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 68, in execute
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     return executor(sql, params, many, context)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     return self.cursor.execute(sql, params)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     raise dj_exc_value.with_traceback(traceback) from exc_value
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    |     return self.cursor.execute(sql, params)
rodan_rodan-main.1.86e2mtm6elo8@prod-rodan-swarm-debian    | django.db.utils.InternalError: unexpected chunk number 2 (expected 1) for toast value 25525 in pg_toast_2619

[homework36]

related upstream postgre error message:

ERROR:  unexpected chunk number 2 (expected 1) for toast value 25525 in pg_toast_2619
rodan_postgres.1.orf57rr3nq8u@prod-rodan-swarm-debian    | STATEMENT:  SELECT "auth_permission"."codename" FROM "auth_permission" INNER JOIN "django_content_type" ON ("auth_permission"."content_type_id" = "django_content_type"."id") INNER JOIN "guardian_groupobjectpermission" ON ("auth_permission"."id" = "guardian_groupobjectpermission"."permission_id") INNER JOIN "auth_group" ON ("guardian_groupobjectpermission"."group_id" = "auth_group"."id") INNER JOIN "auth_user_groups" ON ("auth_group"."id" = "auth_user_groups"."group_id") WHERE ("auth_permission"."content_type_id" = 16 AND "guardian_groupobjectpermission"."content_type_id" = 16 AND "auth_user_groups"."user_id" = 3 AND "guardian_groupobjectpermission"."object_pk" = 'b7f62369-53ef-4f8c-a294-70826ed30a82') ORDER BY "django_content_type"."app_label" ASC, "django_content_type"."model" ASC, "auth_permission"."codename" ASC

In addition, when postgre is having issues, login popup window will also appear (nothing to do with interactive classifier). The postgre database has been corrupted due to unexpected container exit (in #1161). Trying to recover the database now. In the worst case, everything will be removed for a new install.

[kyrieb-ekat]

I have gotten a similar error when I remove jobs from a workflow rapidly, or connect a bunch of ports to one another very quickly. I got this one adding an input port to classifying.

[homework36]

This instance has some mod issues from the beginning. We will move to another new instance for rodan2 soon.

[homework36]

on staging as well

[yinanazhou]

I've pushed the changes to staging. Changes should be up tomorrow on staging.

[homework36]

cleared for prod as well