search

frictionlessdata / frictionless-py

Data management framework for Python that provides functionality to describe, extract, validate, and transform tabular data
https://framework.frictionlessdata.io
MIT License
710 stars 148 forks source link

Warnings when writing resource to duckdb #1601

Open fjuniorr opened 1 year ago

fjuniorr commented 1 year ago

When running:

from frictionless import Resource, formats

resource = Resource(
    {
        "name": "reprex",
        "data": [
            ["name", "continent"],
            ["germany", "europe"],
            ["france", "europe"],
            ["spain", "europe"],
        ],
        "schema": {
            "fields": [
                {"name": "name", "type": "string"},
                {"name": "continent", "type": "string"},
            ]
        },
    }
)

conn_string = 'duckdb:///database.db'
resource.write(conn_string, control = formats.SqlControl(table=resource.name))

Although the table get's created in duckdb:

D select * from reprex;
┌─────────┬───────────┐
│  name   │ continent │
│ varchar │  varchar  │
├─────────┼───────────┤
│ germany │ europe    │
│ france  │ europe    │
│ spain   │ europe    │
└─────────┴───────────┘

I get the following warnings:

/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'character_octet_length'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'datetime_precision'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'interval_type'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'interval_precision'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'character_set_catalog'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'character_set_schema'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'character_set_name'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'collation_catalog'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'collation_schema'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'collation_name'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'domain_catalog'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'domain_schema'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'domain_name'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'udt_catalog'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'udt_schema'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'udt_name'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'scope_catalog'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'scope_schema'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'scope_name'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'maximum_cardinality'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'dtd_identifier'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'is_self_referencing'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'is_identity'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'identity_generation'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'identity_start'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'identity_increment'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'identity_maximum'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'identity_minimum'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'identity_cycle'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'is_generated'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'generation_expression'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'is_updatable'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'list' of column 'constraint_column_indexes'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'list' of column 'constraint_column_names'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'list' of column 'labels'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'default_character_set_catalog'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'default_character_set_schema'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'default_character_set_name'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'self_referencing_column_name'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'reference_generation'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'user_defined_type_catalog'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'user_defined_type_schema'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:359: SAWarning: Did not recognize type 'null' of column 'user_defined_type_name'
  columns = self._get_columns_info(rows, domains, enums, schema)  # type: ignore[attr-defined]
/site-packages/duckdb_engine/__init__.py:162: DuckDBEngineWarning: duckdb-engine doesn't yet support reflection on indices
roll commented 11 months ago

@fjuniorr I think currently, sqlachemy dialect for DuckDB is not fully finished so I didn't manage to add full test coverage because of issues like https://github.com/Mause/duckdb_engine/issues/289