StopIteration exception when running Hive queries on Python3

[elukey]

Is the issue already present in https://github.com/cloudera/hue/issues or discussed in the forum https://discourse.gethue.com?

Describe the bug:

Sometimes right after clicking the "play" button in the hive panel, I see:

And on the logs:

[10/Sep/2020 15:58:40 +0000] decorators   ERROR    Error running execute
Traceback (most recent call last):
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 298, in fields
    return [HiveServerTColumnValue2(field).val.pop(0) for field in self.cols]
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 298, in <listcomp>
    return [HiveServerTColumnValue2(field).val.pop(0) for field in self.cols]
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 393, in rows
    yield row.fields()
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 300, in fields
    raise StopIteration
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/decorators.py", line 114, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/api.py", line 225, in execute
    response = _execute_notebook(request, notebook, snippet)
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/api.py", line 153, in _execute_notebook
    response['handle'] = interpreter.execute(notebook, snippet)
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/connectors/hiveserver2.py", line 98, in decorator
    return func(*args, **kwargs)
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/connectors/hiveserver2.py", line 301, in execute
    result = db.use(query.database, session=_session)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/dbms.py", line 895, in use
    return self.client.use(query, session=session)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 1302, in use
    data = self._client.execute_query(query, session=session)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 938, in execute_query
    return self.execute_query_statement(statement=query.query['query'], max_rows=max_rows, configuration=configuration, session=session)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 945, in execute_query_statement
    results, schema, operation_handle, session = self.execute_statement(statement=statement, max_rows=max_rows, configuration=configuration, orientation=orientation, session=session)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 982, in execute_statement
    (res, session) = self.call(self._client.ExecuteStatement, req, session=session)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 718, in call
    return self.call_return_result_and_session(fn, req, status, session=session)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 755, in call_return_result_and_session
    session = self.open_session(self.user)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 709, in open_session
    configuration = self.get_configuration(session=session)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 1163, in get_configuration
    properties = [(row[0].split('=')[0], row[0].split('=')[1]) for row in results.rows() if '=' in row[0]]
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 1163, in <listcomp>
    properties = [(row[0].split('=')[0], row[0].split('=')[1]) for row in results.rows() if '=' in row[0]]
RuntimeError: generator raised StopIteration

Steps to reproduce it?

I happens sometimes when running hive queries in the Hue's hive panel. Not sure what triggers it yet.

Hue version or source? (e.g. open source 4.5, CDH 5.16, CDP 1.0...). System info (e.g. OS, Browser...).

4.7.1

[elukey]

After some tests with thrift_version I was able to avoid the StopInteration exceptions, I think those are raised due to the super old Hive version that we run (CDH 5.16 1.1). I see in the code that there are conditionals also for thrift_version >= 7, but if I use 7 I still see the errors. We'll upgrade to Hive 2.2.3 soon-ish (hopefully), but I cannot test with higher hive versions atm :(

The StopIteration exceptions should be avoided in py3 for generators, so in theory it should be a matter of swapping them with something else (like return).

[romainr]

Closing if understood correctly

[elukey]

@romainr the remaining issue seems to be that StopIteration is raised in various part of the hive code, and it triggered it pops up to the UI (making the user very confused). In theory StopIteration should not be used in py3 generators, but I am not familiar with the code to make a judgement about how to fix it :(

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.