bug: Cannot Access INFORMATION_SCHEMA Tables in Apache Druid

[ongdisheng]

What happened?

When using Ibis with Apache Druid as backend, I’m unable to access the following tables from INFORMATIONSCHEMA: COLUMNS, ROUTINES, SCHEMATA, and TABLES_. Only user-created tables (e.g., wikiticker-2015-09-12-sampled) are accessible, as shown in the log output.

What version of ibis are you using?

9.5.0

What backend(s) are you using, if any?

Druid

Relevant log output

import ibis
con = ibis.druid.connect()

print('Table list:', con.list_tables())
print(con.table('wikiticker-2015-09-12-sampled'))
print(con.table('COLUMNS'))

Table list: ['COLUMNS', 'ROUTINES', 'SCHEMATA', 'TABLES', 'segments', 'server_segments', 'servers', 'supervisors', 'tasks', 'wikiticker-2015-09-12-sampled']
DatabaseTable: wikiticker-2015-09-12-sampled
  __time         timestamp
  channel        string
  cityName       string
  comment        string
  countryIsoCode string
  countryName    string
  isAnonymous    string
  isMinor        string
  isNew          string
  isRobot        string
  isUnpatrolled  string
  metroCode      int64
  namespace      string
  page           string
  regionIsoCode  string
  regionName     string
  user           string
  delta          int64
  added          int64
  deleted        int64
Traceback (most recent call last):
  File "c:\Users\DI SHENG\Downloads\ibis\ibis_druid.py", line 6, in <module>
    print(con.table('COLUMNS'))
          ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\site-packages\ibis\backends\sql\__init__.py", line 137, in table
    table_schema = self.get_schema(name, catalog=catalog, database=database)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\site-packages\ibis\backends\druid\__init__.py", line 157, in get_schema
    return self._get_schema_using_query(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\site-packages\ibis\backends\druid\__init__.py", line 134, in _get_schema_using_query
    with self._safe_raw_sql(f"EXPLAIN PLAN FOR {query}") as result:
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\site-packages\ibis\backends\druid\__init__.py", line 130, in _safe_raw_sql
    cur.execute(query, *args, **kwargs)
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\site-packages\pydruid\db\api.py", line 72, in g
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\site-packages\pydruid\db\api.py", line 271, in execute
    first_row = next(results)
                ^^^^^^^^^^^^^
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\site-packages\pydruid\db\api.py", line 386, in _stream_query
    raise exceptions.ProgrammingError(msg)
pydruid.db.exceptions.ProgrammingError: druidException (INVALID_INPUT): Object 'COLUMNS' not found (line [1], column [32])

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[cpcloud]

Can you try passing in database="INFORMATION_SCHEMA" to con.table when accessing one of those tables?

[ongdisheng]

@cpcloud Thank you for the suggestion! I tried passing database="INFORMATION_SCHEMA" to con.table. Unfortunately, I'm still unable to access the tables. However, the error message has changed. Here’s the new error message:

Traceback (most recent call last):
  File "c:\Users\DI SHENG\Downloads\ibis\ibis_druid.py", line 6, in <module>
    print(con.table('COLUMNS', database="INFORMATION_SCHEMA"))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\site-packages\ibis\backends\sql\__init__.py", line 137, in table
    table_schema = self.get_schema(name, catalog=catalog, database=database)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\site-packages\ibis\backends\druid\__init__.py", line 157, in get_schema
    return self._get_schema_using_query(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\site-packages\ibis\backends\druid\__init__.py", line 137, in _get_schema_using_query
    (plan,) = json.loads(row)
              ^^^^^^^^^^^^^^^
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\DI SHENG\AppData\Local\Programs\Python\Python311\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)