search

naturalness / partycrasher

Automated scalable crash bucketing
http://partycrasher.readthedocs.io/
GNU General Public License v3.0
15 stars 3 forks source link

Fix validation bug #79

Closed ghost closed 8 years ago

ghost commented 8 years ago

Validation fails after ~7000 crashes with:

Traceback (most recent call last): File "buckettest.py", line 213, in simulationdata = response.json() File "/home/joshua/partycrasher/venv/local/lib/python2.7/site-packages/requests/models.py", line 808, in json return complexjson.loads(self.text, kwargs) File "/usr/lib/python2.7/json/init**.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded

eddieantonio commented 8 years ago

Errors using Python 2.7 and ElasticSearch 2.3:

Python Traceback
127.0.0.1 - - [08/Jul/2016 14:09:45] "POST /reports HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/flask_cors/extension.py", line 188, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/flask_cors/extension.py", line 188, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/eddieantonio/Projects/PartyCrasher/partycrasher/rest_service.py", line 306, in add_report
    report, url = ingest_one(report, project)
  File "/Users/eddieantonio/Projects/PartyCrasher/partycrasher/rest_service.py", line 927, in ingest_one
    report = crasher.ingest(report, dryrun=dryrun)
  File "/Users/eddieantonio/Projects/PartyCrasher/partycrasher/__init__.py", line 182, in ingest
    return self.bucketer.assign_save_buckets(true_crash)
  File "/Users/eddieantonio/Projects/PartyCrasher/partycrasher/bucketer.py", line 360, in assign_save_buckets
    return super(MLT, self).assign_save_buckets(crash, buckets)
  File "/Users/eddieantonio/Projects/PartyCrasher/partycrasher/bucketer.py", line 127, in assign_save_buckets
    saved_crash = ESCrash(crash, index=self.index)
  File "/Users/eddieantonio/Projects/PartyCrasher/partycrasher/es_crash.py", line 95, in __call__
    refresh=True)
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 248, in create
    return self.index(index, doc_type, body, id=id, params=params, op_type='create')
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 279, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/elasticsearch/transport.py", line 329, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 109, in perform_request
    self._raise_error(response.status, raw_data)
  File "/Users/eddieantonio/.virtualenvs/PartyCrasher/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 108, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, u'mapper_parsing_exception', {u'status': 400, u'error': {u'root_cause': [{u'reason': u"Field name [dmi.bios.date] cannot contain '.'", u'type': u'mapper_parsing_exception'}], u'type': u'mapper_parsing_exception', u'reason': u"Field name [dmi.bios.date] cannot contain '.'"}})
ElasticSearch Logs
MapperParsingException[Field name [dmi.bios.date] cannot contain '.']
        at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:278)
        at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:223)
        at org.elasticsearch.index.mapper.object.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:139)
        at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:140)
        at org.elasticsearch.index.mapper.DocumentMapperParser.parseCompressed(DocumentMapperParser.java:121)
        at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:391)
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$2.execute(MetaDataMappingService.java:386)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:388)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[2016-07-08 14:09:45,930][INFO ][rest.suppressed          ] /crashes/crash/launchpad:0000386302 Params: {refresh=true, index=crashes, op_type=create, id=launchpad:0000386302, type=crash}
MapperParsingException[Field name [dmi.bios.date] cannot contain '.']
        at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:278)
        at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:223)
        at org.elasticsearch.index.mapper.object.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:139)
        at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:140)
        at org.elasticsearch.index.mapper.DocumentMapperParser.parseCompressed(DocumentMapperParser.java:121)
        at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:391)
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$2.execute(MetaDataMappingService.java:386)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:388)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)