search

maggienj / ActiveData

Provide high speed filtering and aggregation over data
Mozilla Public License 2.0
0 stars 0 forks source link

Fix TestSetOps.test_number #13

Open maggienj opened 7 years ago

maggienj commented 7 years ago

Unit Test Name: tests.test_jx.test_expressions_w_set_ops.TestSetOps.test_number

New Err: caused by ERROR: 400 { "reason": "mapper [v] of different type, current_type [double], merged_type [keyword]", "type": "illegal_argument_exception" } (and 0 others) while loading line id=2BD4CF04A425EAB5CDB20262461B4492A8A0B2DB into index "testing_093_h20170605_132725": {"v": "2"}

Full Err StackTrace is shown here..


2017-06-05 13:27:25 - GET http://localhost:9200/_aliases
2017-06-05 13:27:25 - response: {"saved_queries20170524_113002":{"aliases":{"saved_queries"...<snip>...":{}},"active_data_requests20170524_113002":{"aliases":{}}}
2017-06-05 13:27:25 - 
http://localhost:9200/testing_093_h20170605_132725:
    {"mappings": {"test_result": {"dynamic_templates": [{"default_numbers": {"match_mapping_type": "long", "mapping": {"type": "double", "store": true}, "match": "*"}}, {"default_strings": {"match_mapping_type": "string", "mapping": {"index": "not_analyzed", "type": "string", "store": true}, "match": "*
2017-06-05 13:27:26 - response: {"acknowledged":true,"shards_acknowledged":true}
2017-06-05 13:27:26 - GET http://localhost:9200/_cluster/state
2017-06-05 13:27:26 - response: {"cluster_name":"maggie_dev","version":9740,"state_uuid":"q...<snip>...32152","allocation_id":{"id":"vDr5mgQwQz2yNN12GVD4rw"}}]}}}
2017-06-05 13:27:26 - 
********************************************************************************
** Made new index "testing_093_h20170605_132725"
********************************************************************************
2017-06-05 13:27:26 - GET http://localhost:9200/_aliases
2017-06-05 13:27:26 - response: {"saved_queries20170524_113002":{"aliases":{"saved_queries"...<snip>...":{}},"active_data_requests20170524_113002":{"aliases":{}}}
2017-06-05 13:27:26 - 
********************************************************************************
** elasticsearch debugging for http://localhost:9200/testing_093_h20170605_132725/test_result is on
********************************************************************************
2017-06-05 13:27:26 - GET http://localhost:9200/_aliases
2017-06-05 13:27:26 - response: {"saved_queries20170524_113002":{"aliases":{"saved_queries"...<snip>...":{}},"active_data_requests20170524_113002":{"aliases":{}}}
2017-06-05 13:27:26 - GET http://localhost:9200/_aliases
2017-06-05 13:27:26 - response: {"saved_queries20170524_113002":{"aliases":{"saved_queries"...<snip>...":{}},"active_data_requests20170524_113002":{"aliases":{}}}
2017-06-05 13:27:26 - 
********************************************************************************
** elasticsearch debugging for http://localhost:9200/testing_093_h20170605_132725/test_result is on
********************************************************************************
2017-06-05 13:27:26 - 
http://localhost:9200/_aliases:
    {"actions": [{"add": {"index": "testing_093_h20170605_132725", "alias": "testing_093_h"}}]}
2017-06-05 13:27:26 - PUT http://localhost:9200/_aliases
2017-06-05 13:27:26 - response: {"acknowledged":true}
2017-06-05 13:27:26 - GET http://localhost:9200/_cluster/state
2017-06-05 13:27:26 - response: {"cluster_name":"maggie_dev","version":9741,"state_uuid":"F...<snip>...32152","allocation_id":{"id":"vDr5mgQwQz2yNN12GVD4rw"}}]}}}
2017-06-05 13:27:26 - Timer start: Add 5.0 documents to testing_093_h20170605_132725
2017-06-05 13:27:26 - 
http://localhost:9200/testing_093_h20170605_132725/test_result/_bulk:
    {"index":{"_id": "11D7932DAC8E17EA500A797A5CF75A722818AD6F"}}
    {"v": 1}
    {"index":{"_id": "2BD4CF04A425EAB5CDB20262461B4492A8A0B2DB"}}
    {"v": "2"}
    {"index":{"_id": "E9971DA6842FEFD93D63D9E772F57033D412C979"}}
    {"v": 3}
    {"index":{"_id": "8D2D8BFCA99FF77C211177400D2415AC2D19E87D"}}
    {"v": "4"}
    {"index":{"_
2017-06-05 13:27:26 - PUT http://localhost:9200/testing_093_h20170605_132725/test_result/_bulk
2017-06-05 13:27:26 - response: {"took":59,"errors":true,"items":[{"index":{"_index":"testing_093_h20170605_132725","_type":"test_result","_id":"11D7932DAC8E17EA5
2017-06-05 13:27:26 - Timer end  : Add 5.0 documents to testing_093_h20170605_132725 (took 0.085 seconds)

2017-06-05 13:27:26 - Deleting index testing_093_h20170605_132725
2017-06-05 13:27:26 - GET http://localhost:9200/_aliases
2017-06-05 13:27:26 - response: {"saved_queries20170524_113002":{"aliases":{"saved_queries"...<snip>...":{}},"active_data_requests20170524_113002":{"aliases":{}}}
2017-06-05 13:27:26 - 
http://localhost:9200/_aliases:
    {"actions": [{"remove": {"index": "testing_093_h20170605_132725", "alias": "testing_093_h"}}]}
2017-06-05 13:27:26 - PUT http://localhost:9200/_aliases
2017-06-05 13:27:26 - response: {"acknowledged":true}
2017-06-05 13:27:26 - delete response {"acknowledged": true}

Error
Traceback (most recent call last):
  File "C:\Python27\lib\unittest\case.py", line 329, in run
    testMethod()
  File "C:\Users\user\PycharmProjects\ActiveData\tests\test_jx\test_expressions_w_set_ops.py", line 631, in test_number
    self.utils.execute_es_tests(test)
  File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 149, in execute_es_tests
    self.fill_container(subtest, tjson=tjson)
  File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 187, in fill_container
    Log.error("can not load {{data}} into container", {"data":subtest.data}, e)
  File "C:\Users\user\PycharmProjects\ActiveData\mo_logs\__init__.py", line 392, in error
    raise e
Except: ERROR: can not load [{"v": 1}, {"v": "2"}, {"v": 3}, {"v": "4"}, {}] into container
    File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 187, in fill_container
    File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 149, in execute_es_tests
    File "C:\Users\user\PycharmProjects\ActiveData\tests\test_jx\test_expressions_w_set_ops.py", line 631, in test_number
    File "C:\Python27\lib\unittest\case.py", line 329, in run
    File "C:\Python27\lib\unittest\case.py", line 393, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\runner.py", line 151, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pycharm\teamcity\unittestpy.py", line 207, in run
    File "C:\Python27\lib\unittest\main.py", line 232, in runTests
    File "C:\Python27\lib\unittest\main.py", line 95, in __init__
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pycharm\_jb_unittest_runner.py", line 29, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>
caused by
    ERROR: problem sending to ES
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 362, in extend
    File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 175, in fill_container
    File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 149, in execute_es_tests
    File "C:\Users\user\PycharmProjects\ActiveData\tests\test_jx\test_expressions_w_set_ops.py", line 631, in test_number
    File "C:\Python27\lib\unittest\case.py", line 329, in run
    File "C:\Python27\lib\unittest\case.py", line 393, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\runner.py", line 151, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pycharm\teamcity\unittestpy.py", line 207, in run
    File "C:\Python27\lib\unittest\main.py", line 232, in runTests
    File "C:\Python27\lib\unittest\main.py", line 95, in __init__
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pycharm\_jb_unittest_runner.py", line 29, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>
caused by
    ERROR: Problems with insert
    File "C:\Users\user\PycharmProjects\ActiveData\pyLibrary\env\elasticsearch.py", line 357, in extend
    File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 175, in fill_container
    File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 149, in execute_es_tests
    File "C:\Users\user\PycharmProjects\ActiveData\tests\test_jx\test_expressions_w_set_ops.py", line 631, in test_number
    File "C:\Python27\lib\unittest\case.py", line 329, in run
    File "C:\Python27\lib\unittest\case.py", line 393, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\runner.py", line 151, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pycharm\teamcity\unittestpy.py", line 207, in run
    File "C:\Python27\lib\unittest\main.py", line 232, in runTests
    File "C:\Python27\lib\unittest\main.py", line 95, in __init__
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pycharm\_jb_unittest_runner.py", line 29, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>
caused by
    ERROR: 400 {
    "reason": "mapper [v] of different type, current_type [double], merged_type [keyword]",
    "type": "illegal_argument_exception"
} (and 0 others) while loading line id=2BD4CF04A425EAB5CDB20262461B4492A8A0B2DB into index "testing_093_h20170605_132725":
{"v": "2"}
    File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 175, in fill_container
    File "C:\Users\user\PycharmProjects\ActiveData\tests\__init__.py", line 149, in execute_es_tests
    File "C:\Users\user\PycharmProjects\ActiveData\tests\test_jx\test_expressions_w_set_ops.py", line 631, in test_number
    File "C:\Python27\lib\unittest\case.py", line 329, in run
    File "C:\Python27\lib\unittest\case.py", line 393, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\suite.py", line 108, in run
    File "C:\Python27\lib\unittest\suite.py", line 70, in __call__
    File "C:\Python27\lib\unittest\runner.py", line 151, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pycharm\teamcity\unittestpy.py", line 207, in run
    File "C:\Python27\lib\unittest\main.py", line 232, in runTests
    File "C:\Python27\lib\unittest\main.py", line 95, in __init__
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pycharm\_jb_unittest_runner.py", line 29, in <module>
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1015, in run
    File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.1.1\helpers\pydev\pydevd.py", line 1578, in <module>
maggienj commented 7 years ago

we have added a new datatype "keyword" to a list of available datatypes. It appears that "keyword" datatype is impacting this "double" and/or may be explicit conversion is required somewhere.... not sure..

maggienj commented 7 years ago

ERROR: can not load [{"v": 1}, {"v": "2"}, {"v": 3}, {"v": "4"}, {}] into container ...so it looks like it is happening while it is creating the index in ES. So, may be, check where it is creating the index...

klahnakoski commented 7 years ago

This is the biggest change we must deal with when moving to ESv5+: ES demands each property has only one datatype! This is just like a database column, and we must work around it.

klahnakoski commented 7 years ago

The plan is to rename every single property, appending the datatype. There is a GSOC student working on this problem right now, rohit in #π for the sqlite database!

Here is a description of the renaming plan: https://github.com/mozilla/jx-sqlite#overloading-property-types There is lots in that document that does not pertain to ES, but I hope you get the idea: Alter elasitcsearch.py so that it adds property name suffixes like .$string, .$number, .$keyword. Of course, changing the way a document is stored will force us to change every query we send to the cluster.

The TestSchemaMerging suite is responsible for testing that a diversity of data can be put into ES. You may also notice some @skipIf(global_settings.use == "elasticsearch" lines that are waiting for this change to happen.

Leave this problem until you have nothing else to work on.

klahnakoski commented 7 years ago

elasticsearch.py does have a tjson variable (aka "typed json"), which is used to implement a primitive form of property suffixing, which only works on ES1.7. Feel free to reuse that variable (and the code paths) to implement this more sophisticated form.

klahnakoski commented 7 years ago

Please rename this bug to "Fix TestSetOps.test_number"

maggienj commented 7 years ago

renamed.