Custom Postgres Enum on a pre-existing database causes unhandled Error

[General4077]

Description

Connecting mathesar to a pre-existing postgres database with custom enumerations causes an unspecified error when loading tables. Running the Production config with the DEBUG flag on (I know this isn't realistic but I was poking around) an AssertionError is raised when serializing the offending column.

Internal Server Error: /api/db/v0/tables/
Traceback (most recent call last):
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/rest_framework/views.py", line 466, in handle_exception
    response = exception_handler(exc, context)
  File "/etc/mathesar/mathesar/exception_handlers.py", line 63, in mathesar_exception_handler
    raise exc
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/rest_framework/mixins.py", line 43, in list
    return self.get_paginated_response(serializer.data)
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/rest_framework/serializers.py", line 745, in data
    ret = super().data
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/rest_framework/serializers.py", line 246, in data
    self._data = self.to_representation(self.instance)
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/rest_framework/serializers.py", line 663, in to_representation
    return [
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/rest_framework/serializers.py", line 664, in <listcomp>
    self.child.to_representation(item) for item in iterable
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/rest_framework/serializers.py", line 515, in to_representation
    ret[field.field_name] = field.to_representation(attribute)
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/rest_framework/serializers.py", line 663, in to_representation
    return [
  File "/etc/mathesar/mathesar-venv/lib/python3.10/site-packages/rest_framework/serializers.py", line 664, in <listcomp>
    self.child.to_representation(item) for item in iterable
  File "/etc/mathesar/mathesar/api/serializers/columns.py", line 100, in to_representation
    assert db_type is not None
AssertionError

Expected behavior

Tables should load in with the Enumerations or a detailed error message should be raised.

To Reproduce

Connect mathesar to a postgres database with custom enumerations

from sqlalchemy import Integer, create_engine
from sqlalchemy.dialects.postgresql import ENUM
from sqlalchemy.orm import DeclarativeBase, mapped_column

Engine = create_engine(
    "postgresql+psycopg2://<user>:<pass>@<host>/<db>", echo=True
)

class Base(DeclarativeBase):
    pass

ACTIVE_STATUS_ENUM = ENUM("ACTIVE", "INACTIVE", name="active_status_enum")

class Example(Base):
    __tablename__ = "example"

    id = mapped_column(Integer, primary_key=True)
    status = mapped_column(ACTIVE_STATUS_ENUM, nullable=False, default="ACTIVE")

Base.metadata.create_all(Engine)

Environment

• OS: Host: Ubuntu 22.04; Client: Windows 10
• Browser: Brave
• Browser Version: (1.64.109 Chromium: 123.0.6312.58 (Official Build) (64-bit)
• Mathesar Version: 0.1.5

Workaround (Very ugly bandaid)

I'm sure this larger than just ENUMs and probably extends to all custom types in postgres but for my use case I just patched db/columns/base.py:MathesarColumn.db_type property to default to a text type if no valid type was found.

    @property
    def db_type(self):
        """
        Get this column's database type enum.
        """
        self._assert_that_engine_is_present()
        res = get_db_type_enum_from_class(self.type.__class__)
        if res:
            return res
        import sqlalchemy
        from db.types.base import PostgresType
        return PostgresType('text')

This mostly works but doesn't understand the enum and provides no usable feedback for invalid inputs. Updating a row: Inserting a row:

Additional context

I looked through several issues looking to see if this was already discussed and I'm not sure if it ties in with the higher level discussion in issue 403

This sounds like an issue the maintainers would want to address but I work with python and have passing familiarity with django if there's a direction for a more permanent solution I'd be happy to put some time towards that.

[kgodey]

Thanks for reporting this, @General4077, we'll take a look soon and determine next steps.

[mathemancer]

Hey @General4077 ! At the moment, we're not prioritizing adding significant support for types. With that said, your solution could be merged if you:

• Add a test, and
• Submit a PR.

For context, we're pushing hard to get our beta release out right now, and trying to avoid significant feature changes in the back end while we do that.

Improved support for PostgreSQL types is a high priority after the beta release.

[kgodey]

I've moved this to the "beta" milestone, but we'll be happy to accept a PR from you anytime @General4077.

[General4077]

@kgodey @mathemancer I'd be happy to try and get something in on this

[kgodey]

Thank you @General4077!