search

18F / fbopen

[DEPRECATED] An open API server, data import tools, and sample apps to help small businesses search for opportunities to work with the U.S. government.
Other
101 stars 45 forks source link

FBO Data Loader: "ElasticsearchIllegalArgumentException"... #119

Closed arowla closed 10 years ago

arowla commented 10 years ago

Something's going wrong with our MOD updates, apparently.

Rest of the error:

ElasticsearchIllegalArgumentException[failed to execute script]; nested: UnresolveablePropertyException[unable to resolve token: unable to resolve variable 'description']

I'm wondering if this is just something we missed when we went to using ext, or if it's something changed in the scripting in the newer version(s) of ES.

dwcaraway commented 10 years ago

weird. is that message coming coming from https://github.com/18F/fbopen/blob/7c3af8bc8e3702ca2412781bc864b69f0fbdf23b/api/app.js ?

arowla commented 10 years ago

Nope, it's from https://github.com/18F/fbopen/blob/master/loaders/fbo.gov/fbo-nightly.sh#L68

arowla commented 10 years ago

I was researching this a bit in our data. It turns out to be a very small percentage of records which were experiencing this, so I didn't follow all the way through to find out why those particular records were failing. One guess is that they are MODs for records which didn't previously exist in the index.

I'm going to dump some code I was using for research here (it is incomplete... it depends on a pre-processed file containing a subset of JSON from the Elasticsearch output).. but just in case it is useful:

>>> with open('items', 'r') as items:
...     objs = json.load(items)
...     err_objs = [ x for x in objs if 'update' in x and x['update']['status'] != 200 ]
...     upd_objs = [ x for x in objs if 'update' in x ]
...     print("all records: {}, updates: {}, update errors: {}".format(len(objs), len(upd_objs), len(err_objs)))
...     print("error ids: {}".format(",".join([x['update']['_id'] for x in err_objs])))
...
all records: 926, updates: 336, update errors: 13
error ids: fbo.gov:PRESOL:fa4460-14-t-0021,fbo.gov:PRESOL:fa4659-14-r-0012,fbo.gov:PRESOL:com-f,fbo.gov:PRESOL:lcnls14r0048,fbo.gov:PRESOL:sp0600-13-r-0507,fbo.gov:PRESOL:sja800-14-r-0002,fbo.gov:PRESOL:dtfh71-14-b-00031,fbo.gov:PRESOL:ag-94tz-s-14-0035,fbo.gov:PRESOL:n0025314t0192,fbo.gov:PRESOL:fa5004-14-r-c008,fbo.gov:PRESOL:w912pl-14-r-0001,fbo.gov:PRESOL:w912pl-14-r-0001,fbo.gov:PRESOL:w912pl-14-r-0001

The IDs listed above would be the ones to research more fully.