pydruid not compatible with druid 27

[cpcloud]

👋🏻 Hi!

Over in the ibis project, we're using pydruid to power our druid backend.

It looks like druid 27 is out, and our bot that updates docker image versions opened a PR that is failing in the pydruid connector.

It seems that some fields have been removed or changed in the REST protocol.

Thanks for all your work on this project!

[abhishekagarwal87]

Hi - It seems like a field called "errorClass" was removed from the error response. This wasn't intended. Can you give the response from Druid API that your test is getting?

[cpcloud]

This is still broken in Druid 28 for the same reason (Ibis CI run: https://github.com/ibis-project/ibis/actions/runs/6871306008/job/18688017127?pr=7551).

I've put up a PR to address this issue here: https://github.com/druid-io/pydruid/pull/308

Here's the traceback from one of our failing tests:

/nix/store/hxwgw8y74c2hy3ap7m2kwa093a8r2cr8-python3-3.10.13-env/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1385: in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
/nix/store/hxwgw8y74c2hy3ap7m2kwa093a8r2cr8-python3-3.10.13-env/lib/python3.10/site-packages/sqlalchemy/sql/compiler.py:476: in _execute_on_connection
    return connection._execute_compiled(
/nix/store/hxwgw8y74c2hy3ap7m2kwa093a8r2cr8-python3-3.10.13-env/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1630: in _execute_compiled
    ret = self._execute_context(
/nix/store/hxwgw8y74c2hy3ap7m2kwa093a8r2cr8-python3-3.10.13-env/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1953: in _execute_context
    self._handle_dbapi_exception(
/nix/store/hxwgw8y74c2hy3ap7m2kwa093a8r2cr8-python3-3.10.13-env/lib/python3.10/site-packages/sqlalchemy/engine/base.py:2138: in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
/nix/store/hxwgw8y74c2hy3ap7m2kwa093a8r2cr8-python3-3.10.13-env/lib/python3.10/site-packages/sqlalchemy/util/compat.py:211: in raise_
    raise exception
/nix/store/hxwgw8y74c2hy3ap7m2kwa093a8r2cr8-python3-3.10.13-env/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1910: in _execute_context
    self.dialect.do_execute(
/nix/store/hxwgw8y74c2hy3ap7m2kwa093a8r2cr8-python3-3.10.13-env/lib/python3.10/site-packages/sqlalchemy/engine/default.py:736: in do_execute
    cursor.execute(statement, parameters)
/nix/store/hxwgw8y74c2hy3ap7m2kwa093a8r2cr8-python3-3.10.13-env/lib/python3.10/site-packages/pydruid/db/api.py:62: in g
    return f(self, *args, **kwargs)
/nix/store/hxwgw8y74c2hy3ap7m2kwa093a8r2cr8-python3-3.10.13-env/lib/python3.10/site-packages/pydruid/db/api.py:256: in execute
    first_row = next(results)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pydruid.db.api.Cursor object at 0x7f55cf650970>
query = 'SELECT sum(CAST("INFORMATION_SCHEMA"."TABLES"."TABLE_NAME" = \'_ibis_test_table_7jiw7mdxhfhdtdg7eio5xjknhu\' AS INTEGER)) AS "sum_1" \nFROM "INFORMATION_SCHEMA"."TABLES"'

    def _stream_query(self, query):
        """
        Stream rows from a query.

        This method will yield rows as the data is returned in chunks from the
        server.
        """
        self.description = None

        headers = {"Content-Type": "application/json"}

        payload = {"query": query, "context": self.context, "header": self.header}

        auth = (
            requests.auth.HTTPBasicAuth(self.user, self.password) if self.user else None
        )
        r = requests.post(
            self.url,
            stream=True,
            headers=headers,
            json=payload,
            auth=auth,
            verify=self.ssl_verify_cert,
            cert=self.ssl_client_cert,
            proxies=self.proxies,
        )
        if r.encoding is None:
            r.encoding = "utf-8"
        # raise any error messages
        if r.status_code != 200:
            try:
                payload = r.json()
            except Exception:
                payload = {
                    "error": "Unknown error",
                    "errorClass": "Unknown",
                    "errorMessage": r.text,
                }
>           msg = "{error} ({errorClass}): {errorMessage}".format(**payload)
E           KeyError: 'errorClass'

/nix/store/hxwgw8y74c2hy3ap7m2kwa093a8r2cr8-python3-3.10.13-env/lib/python3.10/site-packages/pydruid/db/api.py:362: KeyError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>
> /nix/store/hxwgw8y74c2hy3ap7m2kwa093a8r2cr8-python3-3.10.13-env/lib/python3.10/site-packages/pydruid/db/api.py(362)_stream_query()
-> msg = "{error} ({errorClass}): {errorMessage}".format(**payload)

Here's the value of payload:

(Pdb) payload
{'error': 'druidException', 'errorCode': 'invalidInput', 'persona': 'USER', 'category': 'INVALID_INPUT', 'errorMessage': 'Cast function cannot convert value of type BOOLEAN to type INTEGER (line [1], column [12])', 'context': {'sourceType': 'sql', 'line': '1', 'column': '12', 'endLine': '1', 'endColumn': '118'}}

Hopefully it's clear that errorClass is missing.