verdict-project / verdict

Interactive-Speed Analytics: 200x Faster, 200x Fewer Cluster Resources, Approximate Query Processing
http://verdictdb.org
Apache License 2.0
248 stars 66 forks source link

Handle manual query interrupt in pyverdict #350

Closed ebatsell closed 5 years ago

ebatsell commented 5 years ago

When manually killing a pyverdict process with ctrl-c (when running a python script or in the python shell), verdictdb would previously continue running the query processes. I've tested for bypass statements and scramble tables in mysql and postgres and this solution seems to work, though note that this is hard to test consistently because one needs to start large queries in order for the problem to be noticeable.

dongyoungy commented 5 years ago

I have tried this on Impala and does not seem to work.

I locally imported your version of pyverdict:

>>> from pyverdict import VerdictContext as vc
>>> conn = vc.new_impala_context('cp-2.cloudera...', 21..., 'tpch...')

and ran a long query and ctrl+c did not kill the query (I could see the query running on Impala query monitor even after seeing KeyboardInterrupt):

>>> rs = conn.sql("select cntrycode, count(*) as numcust, sum(c_acctbal) as totacctbal from ( select substring(c_phone,1,2) as cntrycode, c_acctbal from tpch100g_parquet.customer where substring(c_phone,1,2) in ('13', '31', '23', '29', '30', '18', '17') and c_acctbal > ( select avg(c_acctbal) from tpch100g_parquet.customer where c_acctbal > 0.00 and substring(c_phone,1,2) in ('13', '31', '23', '29', '30', '18', '17')) and not exists ( select * from tpch100g_parquet.orders_scramble where o_custkey = c_custkey)) as custsale group by cntrycode order by cntrycode;")
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dyoon/Work/verdictdb-evan/pyverdict/pyverdict/verdictcontext.py", line 226, in sql
    return self.sql_raw_result(query).to_df()
AttributeError: 'NoneType' object has no attribute 'to_df'
>>>
KeyboardInterrupt
>>>
KeyboardInterrupt
>>>

The query only terminated when I closed the shell itself with ctrl+d.

pyongjoo commented 5 years ago

@ebatsell Please re-push this commit(s) after checking out this repository. By doing so, CircleCI should be triggered automatically.