search

maggienj / ActiveData

Provide high speed filtering and aggregation over data
Mozilla Public License 2.0
0 stars 0 forks source link

Fix - tests.test_jx.test_deep_ops.TestDeepOps.test_id_select #45

Closed maggienj closed 7 years ago

maggienj commented 7 years ago 
2017-07-06 14:56:07 - Timer end  : Add 4.0 documents to testing_000_k20170706_145606 (took 0.124 seconds)
2017-07-06 14:56:07 - 
http://localhost:9200/testing_000_k20170706_145606/_flush:
    {"forced": false, "wait_if_ongoing": true}
2017-07-06 14:56:07 - PUT http://localhost:9200/testing_000_k20170706_145606/_flush
2017-07-06 14:56:08 - response: {"_shards":{"total":3,"successful":3,"failed":0}}
Error
Traceback (most recent call last):
  File "C:\Python27\lib\unittest\case.py", line 329, in run
    testMethod()
  File "C:\Users\user\PycharmProjects\ActiveData\tests\test_jx\test_deep_ops.py", line 694, in test_id_select
    self.utils.execute_es_tests(test)
  File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 150, in execute_es_tests
    self.send_queries(subtest)
  File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 226, in send_queries
    Log.error("Failed test {{name|quote}}", {"name": subtest.name}, e)
  File "C:\Users\user\PycharmProjects\ActiveData\mo_logs\__init__.py", line 392, in error
    raise e
Except: ERROR: Failed test "test_id_select"
    File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 226, in send_queries
    File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 150, in execute_es_tests
    File "C:\Users\user\PycharmProjects\ActiveData\tests\test_jx\test_deep_ops.py", line 694, in test_id_select
    File "C:\Python27\lib\unittest\case.py", line 329, in run
    File "C:\Python27\lib\unittest\case.py", line 393, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\runner.py", line 151, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pycharm\teamcity\unittestpy.py", line 207, in run
    File "C:\Python27\lib\unittest\main.py", line 232, in runTests
    File "C:\Python27\lib\unittest\main.py", line 95, in __init__
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pycharm\_jb_unittest_runner.py", line 29, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>
caused by
    ERROR: Failed request
    File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 377, in error
    File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 215, in send_queries
    File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 150, in execute_es_tests
    File "C:\Users\user\PycharmProjects\ActiveData\tests\test_jx\test_deep_ops.py", line 694, in test_id_select
    File "C:\Python27\lib\unittest\case.py", line 329, in run
    File "C:\Python27\lib\unittest\case.py", line 393, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\runner.py", line 151, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pycharm\teamcity\unittestpy.py", line 207, in run
    File "C:\Python27\lib\unittest\main.py", line 232, in runTests
    File "C:\Python27\lib\unittest\main.py", line 95, in __init__
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pycharm\_jb_unittest_runner.py", line 29, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>
caused by
    ERROR: problem
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\jx_usingES.py", line 170, in query
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\jx.py", line 71, in run
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\actions\jx.py", line 62, in jx_query
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\__init__.py", line 54, in output
    File "C:\Python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    File "C:\Python27\lib\site-packages\flask\app.py", line 1994, in __call__
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 197, in execute
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 209, in run_wsgi
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 267, in handle_one_request
    File "C:\Python27\lib\BaseHTTPServer.py", line 340, in handle
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 232, in handle
    File "C:\Python27\lib\SocketServer.py", line 652, in __init__
    File "C:\Python27\lib\SocketServer.py", line 331, in finish_request
    File "C:\Python27\lib\SocketServer.py", line 596, in process_request_thread
    File "C:\Python27\lib\threading.py", line 754, in run
    File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
    File "C:\Python27\lib\threading.py", line 774, in __bootstrap
caused by
    ERROR: Error with FromES
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\es09\util.py", line 51, in post
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\es14\deep.py", line 233, in es_deepop
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\jx_usingES.py", line 155, in query
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\jx.py", line 71, in run
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\actions\jx.py", line 62, in jx_query
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\__init__.py", line 54, in output
    File "C:\Python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    File "C:\Python27\lib\site-packages\flask\app.py", line 1994, in __call__
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 197, in execute
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 209, in run_wsgi
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 267, in handle_one_request
    File "C:\Python27\lib\BaseHTTPServer.py", line 340, in handle
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 232, in handle
    File "C:\Python27\lib\SocketServer.py", line 652, in __init__
    File "C:\Python27\lib\SocketServer.py", line 331, in finish_request
    File "C:\Python27\lib\SocketServer.py", line 596, in process_request_thread
    File "C:\Python27\lib\threading.py", line 754, in run
    File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
    File "C:\Python27\lib\threading.py", line 774, in __bootstrap
caused by
    ERROR: Problem with search (path=/testing_000_k/test_result/_search):
    {
        "filter": {"and": [
            {"match_all": {}},
            {"nested": {
                "filter": {"match_all": {}},
                "inner_hits": {"size": 100000},
                "path": "a._a"
            }}
        ]},
        "from": 0,
        "size": 10,
        "stored_fields": []
    }
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 1107, in search
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\es09\util.py", line 40, in post
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\es14\deep.py", line 233, in es_deepop
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\jx_usingES.py", line 155, in query
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\jx.py", line 71, in run
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\actions\jx.py", line 62, in jx_query
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\__init__.py", line 54, in output
    File "C:\Python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    File "C:\Python27\lib\site-packages\flask\app.py", line 1994, in __call__
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 197, in execute
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 209, in run_wsgi
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 267, in handle_one_request
    File "C:\Python27\lib\BaseHTTPServer.py", line 340, in handle
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 232, in handle
    File "C:\Python27\lib\SocketServer.py", line 652, in __init__
    File "C:\Python27\lib\SocketServer.py", line 331, in finish_request
    File "C:\Python27\lib\SocketServer.py", line 596, in process_request_thread
    File "C:\Python27\lib\threading.py", line 754, in run
    File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
    File "C:\Python27\lib\threading.py", line 774, in __bootstrap
caused by
    ERROR: Problem with call to http://localhost:9200/testing_000_k/test_result/_search
{"filter": {"and": [{"match_all": {}}, {"nested": {"filter": {"match_all": {}}, "path": "a._a", "inner_hits": {"size": 100000}}}]}, "stored_fields": [], "from": 0, "size": 10}
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 782, in post
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 1100, in search
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\es09\util.py", line 40, in post
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\es14\deep.py", line 233, in es_deepop
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\jx_usingES.py", line 155, in query
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\jx.py", line 71, in run
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\actions\jx.py", line 62, in jx_query
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\__init__.py", line 54, in output
    File "C:\Python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    File "C:\Python27\lib\site-packages\flask\app.py", line 1994, in __call__
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 197, in execute
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 209, in run_wsgi
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 267, in handle_one_request
    File "C:\Python27\lib\BaseHTTPServer.py", line 340, in handle
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 232, in handle
    File "C:\Python27\lib\SocketServer.py", line 652, in __init__
    File "C:\Python27\lib\SocketServer.py", line 331, in finish_request
    File "C:\Python27\lib\SocketServer.py", line 596, in process_request_thread
    File "C:\Python27\lib\threading.py", line 754, in run
    File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
    File "C:\Python27\lib\threading.py", line 774, in __bootstrap
caused by
    ERROR: Bad Request: {"error":{"root_cause":[{"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [filter].","line":1,"col":12}],"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [filter].","line":1,"col":12},"status":400}
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 760, in post
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 1100, in search
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\es09\util.py", line 40, in post
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\es14\deep.py", line 233, in es_deepop
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\jx_usingES.py", line 155, in query
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\queries\jx.py", line 71, in run
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\actions\jx.py", line 62, in jx_query
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\__init__.py", line 54, in output
    File "C:\Python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    File "C:\Python27\lib\site-packages\flask\app.py", line 1994, in __call__
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 197, in execute
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 209, in run_wsgi
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 267, in handle_one_request
    File "C:\Python27\lib\BaseHTTPServer.py", line 340, in handle
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 232, in handle
    File "C:\Python27\lib\SocketServer.py", line 652, in __init__
    File "C:\Python27\lib\SocketServer.py", line 331, in finish_request
    File "C:\Python27\lib\SocketServer.py", line 596, in process_request_thread
    File "C:\Python27\lib\threading.py", line 754, in run
    File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
    File "C:\Python27\lib\threading.py", line 774, in __bootstrap

2017-07-06 14:56:09 - Deleting index testing_000_k20170706_145606
2017-07-06 14:56:09 - GET http://localhost:9200/_aliases
2017-07-06 14:56:09 - response: {"testdata20170703_235336":{"aliases":{}},"saved_queries201...<snip>...:{}}},"active_data_requests20170524_113002":{"aliases":{}}}
2017-07-06 14:56:09 - 
http://localhost:9200/_aliases:
    {"actions": [{"remove": {"index": "testing_000_k20170706_145606", "alias": "testing_000_k"}}]}
2017-07-06 14:56:09 - PUT http://localhost:9200/_aliases
2017-07-06 14:56:09 - response: {"acknowledged":true}
2017-07-06 14:56:10 - delete response {"acknowledged": true} 

Ran 1 test in 3.873s

FAILED (errors=1)

Process finished with exit code 1
maggienj commented 7 years ago

changed a few instances of "filter" to "query": {"bool": { "filter

it does show the query with updated changes in the console..... but, for some reason it still errs ....


kabalidaa (pid 6356) - 2017-07-10 16:06:29 - Main Thread - "deep.py:232" (es_deepop) - Timer start: call to ES
kabalidaa (pid 6356) - 2017-07-10 16:06:29 - Main Thread - "elasticsearch.py:1102" (search) - 
Query /testing_000_w/test_result/_search
    {
        "from": 0,
        "query": {"bool": {"must": [
            {"match_all": {}},
            {"nested": {
                "inner_hits": {"size": 100000},
                "path": "a._a",
                "query": {"bool": {"filter": {"match_all": {}}}}
            }}
        ]}},
        "size": 10,
        "stored_fields": []
    }
kabalidaa (pid 6356) - 2017-07-10 16:06:29 - get more - "elasticsearch.py:1102" (search) - 
Query /testing_000_w/test_result/_search
    {
        "fields": [],
        "filter": {"and": [
            {"match_all": {}},
            {"not": {"nested": {
                "path": "a._a",
                "query": {"bool": {"filter": {"match_all": {}}}}
            }}}
        ]}
    }
kabalidaa (pid 6356) - 2017-07-10 16:06:29 - Main Thread - "deep.py:233" (es_deepop) - Timer end  : call to ES (took 0.046 seconds)
kabalidaa (pid 6356) - 2017-07-10 16:06:29 - Main Thread - "jx.py:65" (jx_query) - Timer end  : translate (took 0.166 seconds)
kabalidaa (pid 6356) - 2017-07-10 16:06:29 - Main Thread - "jx.py:81" (jx_query) - Timer end  : total duration (took 6.006 seconds)
kabalidaa (pid 6356) - 2017-07-10 16:06:29 - Main Thread - "__init__.py:32" (send_error) - WARNING: Could not process
{"meta": {"testing": true}, "from": "testing_000_w.a._a", "select": ["_id"], "format": "cube"}
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\actions\__init__.py", line 32, in send_error
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\actions\jx.py", line 100, in jx_query
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\__init__.py", line 54, in output
    File "C:\Python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    File "C:\Python27\lib\site-packages\flask\app.py", line 1994, in __call__
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 197, in execute
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 209, in run_wsgi
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 267, in handle_one_request
    File "C:\Python27\lib\BaseHTTPServer.py", line 340, in handle
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 232, in handle
    File "C:\Python27\lib\SocketServer.py", line 652, in __init__
    File "C:\Python27\lib\SocketServer.py", line 331, in finish_request
    File "C:\Python27\lib\SocketServer.py", line 596, in process_request_thread
    File "C:\Python27\lib\threading.py", line 754, in run
    File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
    File "C:\Python27\lib\threading.py", line 774, in __bootstrap
caused by
    ERROR: problem
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\jx_usingES.py", line 170, in query
    File "C:\Users\user\PycharmProjects\ActiveData\jx_python\jx.py", line 71, in run
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\actions\jx.py", line 62, in jx_query
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\__init__.py", line 54, in output
    File "C:\Python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    File "C:\Python27\lib\site-packages\flask\app.py", line 1994, in __call__
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 197, in execute
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 209, in run_wsgi
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 267, in handle_one_request
    File "C:\Python27\lib\BaseHTTPServer.py", line 340, in handle
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 232, in handle
    File "C:\Python27\lib\SocketServer.py", line 652, in __init__
    File "C:\Python27\lib\SocketServer.py", line 331, in finish_request
    File "C:\Python27\lib\SocketServer.py", line 596, in process_request_thread
    File "C:\Python27\lib\threading.py", line 754, in run
    File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
    File "C:\Python27\lib\threading.py", line 774, in __bootstrap
caused by
    ERROR: problem formatting
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es14\deep.py", line 258, in es_deepop
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\jx_usingES.py", line 155, in query
    File "C:\Users\user\PycharmProjects\ActiveData\jx_python\jx.py", line 71, in run
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\actions\jx.py", line 62, in jx_query
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\__init__.py", line 54, in output
    File "C:\Python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    File "C:\Python27\lib\site-packages\flask\app.py", line 1994, in __call__
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 197, in execute
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 209, in run_wsgi
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 267, in handle_one_request
    File "C:\Python27\lib\BaseHTTPServer.py", line 340, in handle
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 232, in handle
    File "C:\Python27\lib\SocketServer.py", line 652, in __init__
    File "C:\Python27\lib\SocketServer.py", line 331, in finish_request
    File "C:\Python27\lib\SocketServer.py", line 596, in process_request_thread
    File "C:\Python27\lib\threading.py", line 754, in run
    File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
    File "C:\Python27\lib\threading.py", line 774, in __bootstrap
caused by
    ERROR: Thread "get more" did not end well
    File "C:\Users\user\PycharmProjects\ActiveData\mo_threads\threads.py", line 288, in join
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es14\deep.py", line 244, in inners
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es14\setop.py", line 246, in format_table
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es14\setop.py", line 278, in format_cube
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es14\deep.py", line 252, in es_deepop
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\jx_usingES.py", line 155, in query
    File "C:\Users\user\PycharmProjects\ActiveData\jx_python\jx.py", line 71, in run
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\actions\jx.py", line 62, in jx_query
    File "C:\Users\user\PycharmProjects\ActiveData\active_data\__init__.py", line 54, in output
    File "C:\Python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    File "C:\Python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    File "C:\Python27\lib\site-packages\flask\app.py", line 1994, in __call__
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 197, in execute
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 209, in run_wsgi
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 267, in handle_one_request
    File "C:\Python27\lib\BaseHTTPServer.py", line 340, in handle
    File "C:\Python27\lib\site-packages\werkzeug\serving.py", line 232, in handle
    File "C:\Python27\lib\SocketServer.py", line 652, in __init__
    File "C:\Python27\lib\SocketServer.py", line 331, in finish_request
    File "C:\Python27\lib\SocketServer.py", line 596, in process_request_thread
    File "C:\Python27\lib\threading.py", line 754, in run
    File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
    File "C:\Python27\lib\threading.py", line 774, in __bootstrap
caused by
    ERROR: Error with FromES
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es09\util.py", line 51, in post
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es14\deep.py", line 227, in get_more
    File "C:\Users\user\PycharmProjects\ActiveData\mo_threads\threads.py", line 213, in _run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\_pydev_bundle\pydev_monkey.py", line 589, in __call__
caused by
    ERROR: Problem with search (path=/testing_000_w/test_result/_search):
    {
        "fields": [],
        "filter": {"and": [
            {"match_all": {}},
            {"not": {"nested": {
                "path": "a._a",
                "query": {"bool": {"filter": {"match_all": {}}}}
            }}}
        ]}
    }
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 1113, in search
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es09\util.py", line 40, in post
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es14\deep.py", line 227, in get_more
    File "C:\Users\user\PycharmProjects\ActiveData\mo_threads\threads.py", line 213, in _run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\_pydev_bundle\pydev_monkey.py", line 589, in __call__
caused by
    ERROR: Problem with call to http://localhost:9200/testing_000_w/test_result/_search
{"filter": {"and": [{"match_all": {}}, {"not": {"nested": {"path": "a._a", "query": {"bool": {"filter": {"match_all": {}}}}}}}]}, "fields": []}
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 785, in post
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 1106, in search
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es09\util.py", line 40, in post
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es14\deep.py", line 227, in get_more
    File "C:\Users\user\PycharmProjects\ActiveData\mo_threads\threads.py", line 213, in _run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\_pydev_bundle\pydev_monkey.py", line 589, in __call__
caused by
    ERROR: Bad Request: {"error":{"root_cause":[{"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [filter].","line":1,"col":12}],"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [filter].","line":1,"col":12},"status":400}
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 763, in post
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 1106, in search
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es09\util.py", line 40, in post
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es14\deep.py", line 227, in get_more
    File "C:\Users\user\PycharmProjects\ActiveData\mo_threads\threads.py", line 213, in _run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\_pydev_bundle\pydev_monkey.py", line 589, in __call__
klahnakoski commented 7 years ago

the reason is shown in the last cause: 

 Unknown key for a START_OBJECT in [filter].

which is refering specifically to this part of the query, which is wrong:

"filter": {"and": [

ESv5+ no longer uses filter it uses query

maggienj commented 7 years ago

yes, all "filter"s have been replaced with "query-bool-filter" already. still it is showing as if it has not been replaced.

maggienj commented 7 years ago

since all were replaced, it is showing this query in the first block.


{
  "from": 0,
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        },
        {
          "nested": {
            "inner_hits": {
              "size": 100000
            },
            "path": "a._a",
            "query": {
              "bool": {
                "filter": {
                  "match_all": {}
                }
              }
            }
          }
        }
      ]
    }
  },
  "size": 10,
  "stored_fields": []
}

but, in the second block it shows a query, as if "filter" has not be "replaced" with "query-bool" and as if "and" has not been replaced with "must".
Still ran through the debugger multiple times and it isn't showing the line where it has not been replaced yet. Also, the stack trace is not showing where it has not been replaced yet....

klahnakoski commented 7 years ago

The deep ops start with a query template, and modify the filters in those templates to achieve the filtering required, and then proceeds to build the rest of query from there. Check the code that is responsible for the original template.

maggienj commented 7 years ago

Initially, I did replace all the "filter" to "query-bool-filter". But, i think now... that wherever "filter" is applied it shouldn't be simply replaced with "query-bool-filter", instead it should be .... something like what is shown in the below template... booleanquery with "must" match_all and filter should follow this pattern.... i guess.... so, trying to make this change... 

   FromES.query = {
        "query": {
            "bool": { "must": {
                "match_all": {}
            }},
            "filter": simplify_esfilter(jx_expression(query.where).to_esfilter())
        }
    }
klahnakoski commented 7 years ago

Please make a branch, push your changes, and make a pull request. that will allow me to understand the context of what you are saying

klahnakoski commented 7 years ago

The template you showed still has a "filter". Replace that with "query"

maggienj commented 7 years ago

a branch has been already made , committed and pushed. branch: issue45-iter2

(iter2 branch is more like a warm-up-branch for deepOps/nested queries conversion as per es5.
This warm-up branch is used to understand what changes are required to make this test pass.

Once a clear picture is got, then, will create a cleaner branch to make changes..... )

Just thought of completing to a certain extent before "pull request". anyway will make a pull request shortly.

maggienj commented 7 years ago

I guess, we may need to use a similar template for "nested queries" (deep ops) in ES5 https://stackoverflow.com/questions/34947195/elasticsearch-nested-object-filter?rq=1 I guess the term "deep ops" in activedata is related-to/uses "nested queries" template in es5.

https://www.elastic.co/guide/en/elasticsearch/reference/5.2/query-dsl-bool-query.html

maggienj commented 7 years ago

when the unittest is run, it just appears to be waiting and not running to test completion.... though es5 and app_dev are running...

kabalidaa (pid 5964) - 2017-07-10 23:18:35 - Main Thread - "__init__.py:115" (test_mode_wait) - wait for column (table=testing_000_o, name=_id) metadata to arrive
kabalidaa (pid 5964) - 2017-07-10 23:18:36 - Main Thread - "__init__.py:115" (test_mode_wait) - wait for column (table=testing_000_o, name=_id) metadata to arrive
kabalidaa (pid 5964) - 2017-07-10 23:18:37 - Main Thread - "__init__.py:115" (test_mode_wait) - wait for column (table=testing_000_o, name=_id) metadata to arrive
kabalidaa (pid 5964) - 2017-07-10 23:18:38 - Main Thread - "__init__.py:115" (test_mode_wait) - wait for column (table=testing_000_o, name=_id) metadata to arrive
kabalidaa (pid 5964) - 2017-07-10 23:18:39 - Main Thread - "__init__.py:115" (test_mode_wait) - wait for column (table=testing_000_o, name=_id) metadata to arrive
kabalidaa (pid 5964) - 2017-07-10 23:18:40 - Main Thread - "__init__.py:115" (test_mode_wait) - wait for column (table=testing_000_o, name=_id) metadata to arrive
kabalidaa (pid 5964) - 2017-07-10 23:18:41 - Main Thread - "__init__.py:115" (test_mode_wait) - wait for column (table=testing_000_o, name=_id) metadata to arrive
maggienj commented 7 years ago

continuation from the previous err msg... and then it throws this err... caused by ERROR: HTTPConnectionPool(host='127.0.0.1', port=5000): Read timed out. (read timeout=600) File "C:\Python27\lib\site-packages\requests\adapters.py", line 499, in send File "C:\Python27\lib\site-packages\requests\sessions.py", line 609, in send File "C:\Python27\lib\site-packages\requests\sessions.py", line 488, in request File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\http.py", line 148, in request File "C

maggienj commented 7 years ago

I think, "filter" cannot be replaced with "query" as per this doc... so, moved the query and filter to the must and filter parameters in few occurrences to test this.

move the query and filter to the must and filter parameters in the bool query:

GET _search
{
  "query": {
    "bool": {
      "must": {
        "match": {
          "text": "quick brown fox"
        }
      },
      "filter": {
        "term": {
          "status": "published"
        }
      }
    }
  }
}

https://www.elastic.co/guide/en/elasticsearch/reference/5.4/query-dsl-filtered-query.html

klahnakoski commented 7 years ago

Thank you. The pull request was necessary for me to point out the problems. I made only a few comments, but the same error was seen multiple times. I suggest you make a new branch off of es5, call it "issue45b", and try again.

maggienj commented 7 years ago

Created a new branch for issue45b.
As soon as app_dev was started , it errs .


Connected to pydev debugger (build 171.4163.6)
2017-07-11 02:48:34 - WARNING: Path not found("expressions") Returning None.
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 331, in _get_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 303, in _get_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 351, in _get_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 359, in _set_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 272, in set_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_logs\constants.py", line 34, in set
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 92, in setup
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 195, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>

2017-07-11 02:48:34 - WARNING: Path not found: pyLibrary.queries.expressions.ALLOW_SCRIPTING
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 276, in set_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_logs\constants.py", line 34, in set
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 92, in setup
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 195, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>
caused by
    ERROR: Path not found Tried to get attribute of None
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 361, in _set_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 272, in set_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_logs\constants.py", line 34, in set
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 92, in setup
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 195, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>

2017-07-11 02:48:34 - WARNING: Path not found("meta") Returning None.
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 331, in _get_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 303, in _get_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 351, in _get_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 357, in _set_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 272, in set_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_logs\constants.py", line 34, in set
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 92, in setup
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 195, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>

2017-07-11 02:48:34 - WARNING: Path not found("meta") Returning None.
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 331, in _get_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 303, in _get_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 351, in _get_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 359, in _set_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 272, in set_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_logs\constants.py", line 34, in set
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 92, in setup
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 195, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>

2017-07-11 02:48:34 - WARNING: Path not found: pyLibrary.queries.meta.ENABLE_META_SCAN
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 276, in set_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_logs\constants.py", line 34, in set
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 92, in setup
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 195, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>
caused by
    ERROR: Path not found Tried to get attribute of None
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 361, in _set_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_dots\__init__.py", line 272, in set_attr
    File "C:\Users\user\PycharmProjects\ActiveData\mo_logs\constants.py", line 34, in set
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 92, in setup
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 195, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>

kabalidaa (pid 3884) - 2017-07-11 02:48:35 - Main Thread - "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py:112" (setup) - WARNING: ActiveData is in debug mode
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 112, in setup
    File "C:/Users/user/PycharmProjects/ActiveData/active_data/app.py", line 195, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>

kabalidaa (pid 3884) - 2017-07-11 02:48:35 - refresh metadata - "meta.py:431" (not_monitor) - 
********************************************************************************
** metadata scan has been disabled
********************************************************************************
kabalidaa (pid 3884) - 2017-07-11 02:48:36 - Main Thread - "elasticsearch.py:120" (__init__) - 
********************************************************************************
** elasticsearch debugging for http://localhost:9200/saved_queries20170524_113002/query is on
********************************************************************************
kabalidaa (pid 3884) - 2017-07-11 02:48:36 - Main Thread - "elasticsearch.py:305" (extend) - Timer start: Add 1.0 documents to saved_queries20170524_113002
kabalidaa (pid 3884) - 2017-07-11 02:48:49 - Main Thread - "elasticsearch.py:357" (extend) - Timer end  : Add 1.0 documents to saved_queries20170524_113002 (took 12.960 seconds)
kabalidaa (pid 3884) - 2017-07-11 02:48:52 - Main Thread - "elasticsearch.py:120" (__init__) - 
********************************************************************************
** elasticsearch debugging for http://localhost:9200/saved_queries20170524_113002/query is on
********************************************************************************
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
maggienj commented 7 years ago

just created a new branch off of es5 and started app_dev. app_dev raises lot of errs... (no other changes made in this new br)

klahnakoski commented 7 years ago

since the pyLibrary.queries files moved, any reference to those files must be updated. To their new location. I thought your config files were in the repo, so they should have been changed. We will check you revision numbers and the problem in the morning.

maggienj commented 7 years ago

not sure which line adds the word "filter" in the second line here... Checked all "filter" usages to replace it to "query"..... but, couldn't find an appropriate line to replace to "query"..... existing "filter" lines are related to "aggs" or "_missing" or "_join".... so , didn't replace in those lines. anyway, not sure, which line is adding this "filter" in this query..... 

ERROR: Problem with search (path=/testing_000_t/test_result/_search):
    {
        "fields": [],
        "filter": {"bool": {
            "must": [{"match_all": {}}],
            "must_not": {"nested": {
                "path": "a._a",
                "query": {"match_all": {}}
            }}
        }}
    }
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 1097, in search
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es09\util.py", line 40, in post
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es14
klahnakoski commented 7 years ago

please include the last cause so I know what ES is complaining about

klahnakoski commented 7 years ago

hint: the "filter" may not always be in quotes. JSON can be built from python dicts, and also Data objects: eg Data(filter=esfilter)

maggienj commented 7 years ago

ES - as per the last cause... 

caused by
    ERROR: Bad Request: {"error":{"root_cause":[{"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [filter].","line":1,"col":12}],"type":"parsing_exception","reason":"Unknown key for a START_OBJECT in [filter].","line":1,"col":12},"status":400}
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 755, in post
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 1090, in search
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es09\util.py", line 40, in post
    File "C:\Users\user\PycharmProjects\ActiveData\jx_elasticsearch\es14\deep.py", line 227, in get_more
    File "C:\Users\user\PycharmProjects\ActiveData\mo_threads\threads.py", line 213, in _run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\_pydev_bundle\pydev_monkey.py", line 589, in __call__
maggienj commented 7 years ago

yay! that hint worked like a charm! no more "filter" errs.

maggienj commented 7 years ago

test passed.

maggienj commented 7 years ago

pull request completed. merged. closing this.