django-daiquiri / daiquiri

A framework for the publication of scientific databases
https://escience.aip.de/daiquiri
Apache License 2.0
26 stars 8 forks source link

BUG: the VOTable unicodeChar is not supported for upload #151

Closed agy-why closed 1 year ago

agy-why commented 1 year ago

The number of supported VOTable datatype for upload is quite limited and failed silently:

ERROR daiquiri on_failure: Traceback (most recent call last):
  File "/home/dq/.local/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/dq/.local/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/dq/source/daiquiri/query/tasks.py", line 195, in run_database_ingest_task
    columns = ingest_table(job.schema_name, job.table_name, file_path)
  File "/home/dq/source/daiquiri/query/utils.py", line 302, in ingest_table
    adapter.create_table(schema_name, table_name, columns)
  File "/home/dq/source/daiquiri/core/adapter/database/base.py", line 208, in create_table
    'columns': ', '.join([
  File "/home/dq/source/daiquiri/core/adapter/database/base.py", line 211, in <listcomp>
    'column_type': self.COLUMNTYPES[column['datatype']]
KeyError: 'unicodeChar'

The supported column types are:

https://github.com/django-daiquiri/daiquiri/blob/4e63b5cd7c89cef4f88071e959818494efefaf70/daiquiri/core/adapter/database/postgres.py#L75

    COLUMNTYPES = {
        'char': 'text',
        'boolean': 'boolean',
        'short': 'smallint',
        'int': 'integer',
        'long': 'bigint',
        'float': 'real',
        'double': 'double precision'
    }
agy-why commented 1 year ago

We need to solve two aspects:

  1. add new supported COLUMNTYPES
  2. prevent silent failing and catch the error into the Job Model for debugging
agy-why commented 1 year ago

A current work around is to set the datatype to char instead of unicodeChar in the VOTable to be uploaded.