tethysplatform / tethys

The Tethys Platform main Django website project repository.
http://tethysplatform.org/
BSD 2-Clause "Simplified" License
92 stars 49 forks source link

Change ArrayField columns to JSONField #889

Closed sdc50 closed 1 year ago

sdc50 commented 1 year ago

Currently there are two columns in the tethys models that use the Postgresql-specific field type ArrayField:

Both of these fields only store string values, so they could easily be stored as a JSONField using a list. The JSONField is much more widely supported and would enable Tethys to use other database backends (other than Postgres).

The only reason for needing to use an ArrayField would be if you wanted to have an array of some type that is supported by Postgres, but not supported by JSON (e.g. datetime).

swainn commented 1 year ago

Hey @sdc50 I just pulled latest after merging #890 and I'm getting this error when I try to migrate:

Traceback (most recent call last):
  File "/home/agwa/tethys/tethys_portal/manage.py", line 20, in <module>
    execute_from_command_line(sys.argv)
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/core/management/__init__.py", line 395, in execute
    django.setup()
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/apps.py", line 27, in ready
    self.module.autodiscover()
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/__init__.py", line 24, in autodiscover
    autodiscover_modules('admin', register_to=site)
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/utils/module_loading.py", line 47, in autodiscover_modules
    import_module('%s.%s' % (app_config.name, module_to_search))
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/agwa/tethys/tethys_apps/admin.py", line 443, in <module>
    register_custom_group()
  File "/home/agwa/tethys/tethys_apps/admin.py", line 425, in register_custom_group
    class CustomGroup(GroupAdmin):
  File "/home/agwa/tethys/tethys_apps/admin.py", line 426, in CustomGroup
    form = make_gop_app_access_form()
  File "/home/agwa/tethys/tethys_apps/admin.py", line 377, in make_gop_app_access_form
    for app_qs in all_apps:
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/db/models/query.py", line 280, in __iter__
    self._fetch_all()
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/db/models/query.py", line 68, in __iter__
    for row in compiler.results_iter(results):
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
    value = converter(value, expression, connection)
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/db/models/fields/json.py", line 83, in from_db_value
    return json.loads(value, cls=self.decoder)
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/json/__init__.py", line 339, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not list
sdc50 commented 1 year ago

@swainn do you know witch migration it was trying to run? Was this error the the tethys_compute one?

swainn commented 1 year ago

I don't think it gets that far. It's failing on execute_from_command_line in the manage.py

swainn commented 1 year ago

Hey @sdc50 I just pulled latest after merging #890 and I'm getting this error when I try to migrate:

Traceback (most recent call last):
  File "/home/agwa/tethys/tethys_portal/manage.py", line 20, in <module>
    execute_from_command_line(sys.argv)
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/core/management/__init__.py", line 395, in execute
    django.setup()
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/apps.py", line 27, in ready
    self.module.autodiscover()
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/__init__.py", line 24, in autodiscover
    autodiscover_modules('admin', register_to=site)
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/utils/module_loading.py", line 47, in autodiscover_modules
    import_module('%s.%s' % (app_config.name, module_to_search))
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/agwa/tethys/tethys_apps/admin.py", line 443, in <module>
    register_custom_group()
  File "/home/agwa/tethys/tethys_apps/admin.py", line 425, in register_custom_group
    class CustomGroup(GroupAdmin):
  File "/home/agwa/tethys/tethys_apps/admin.py", line 426, in CustomGroup
    form = make_gop_app_access_form()
  File "/home/agwa/tethys/tethys_apps/admin.py", line 377, in make_gop_app_access_form
    for app_qs in all_apps:
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/db/models/query.py", line 280, in __iter__
    self._fetch_all()
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/db/models/query.py", line 68, in __iter__
    for row in compiler.results_iter(results):
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
    value = converter(value, expression, connection)
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/site-packages/django/db/models/fields/json.py", line 83, in from_db_value
    return json.loads(value, cls=self.decoder)
  File "/home/agwa/miniconda3/envs/tethys/lib/python3.10/json/__init__.py", line 339, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not list

Fixed in #891