search

mozilla / taar-lite

A lightweight version of the TAAR service intended for specific deployments with reduced feature visibility.
Mozilla Public License 2.0
2 stars 6 forks source link

Error in production for GUID key lookup #31

Closed crankycoder closed 6 years ago

crankycoder commented 6 years ago

newrelic is showing a crash w.r.t GUID lookups:

URL path is : /taarlite/api/v1/addon_recommendations/%7Bad0d925d-88f8-47f1-85ea-8463569e756e%7D/

Traceback (most recent call last):
File "/usr/local/lib/python3.6/threading.py", line 884, in _bootstrap
File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
File "/usr/local/lib/python3.6/threading.py", line 864, in run
File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 69, in _worker
File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/gthread.py", line 284, in handle
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/gthread.py", line 333, in handle_request
File "/usr/local/lib/python3.6/site-packages/newrelic/api/web_transaction.py", line 1280, in _nr_wsgi_application_wrapper_
File "/usr/local/lib/python3.6/site-packages/newrelic/api/web_transaction.py", line 807, in __init__
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__
File "/usr/local/lib/python3.6/site-packages/newrelic/api/web_transaction.py", line 1169, in _nr_wsgi_application_wrapper_
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
File "/usr/local/lib/python3.6/site-packages/newrelic/hooks/framework_flask.py", line 45, in _nr_wrapper_handler_
File "/app/taar_api/app.py", line 68, in recommendations
File "/usr/local/lib/python3.6/site-packages/taar_lite/recommenders/guid_based_recommender.py", line 199, in recommend
File "/usr/local/lib/python3.6/site-packages/taar_lite/recommenders/guid_based_recommender.py", line 263, in norm_rownorm_sum
crankycoder commented 6 years ago

application_errors_-_taarlite_api__prod__-_new_relic

crankycoder commented 6 years ago

I just fetched the latest S3 files:

In [15]: coinstall = json.load(open('guid_coinstallation.json'))

In [16]: ranking = json.load(open('guid_install_ranking.json'))

In [17]: coinstall_guids = set(coinstall.keys())

In [18]: ranking_guids = set(ranking.keys())

In [19]: coinstall_guids - ranking_guids
Out[19]: set()

If I obtain the previous co-installation JSON model:

In [21]: old_coinstal_guids = set(old_coinstall.keys())

In [22]: old_coinstal_guids - ranking_guids
Out[22]:
{'breeze-minimize-window-button@andreyorst.com',
 'instarss@turmel.info',
 'jid1-q3sH8pJhc4KGUp@jetpack',
 'jid1-zYaXnuzvDbGcEg@jetpack',
 'killit@jetpack',
 'octopatcher_wowmotty@gmail.com',
 'rym2spotify@magdolen.sk',
 'selection-popup@foreverglory.me',
 'tb-clone-tab-single@codefisher.org',
 '{04331ca8-933b-4f14-bc84-806e7227be94}',
 '{24e9b60f-0b13-43ce-aa07-ceea8ab747a8}',
 '{d400d670-e438-49e6-9788-c9ae0d37158b}',
 '{d6fc5642-b9b3-4ba2-b5cd-70116a22f0fa}'}

In [23]:

This error happens when the ETL jobs are updated - so there is something out of sync with self._guid_maps.

self._guid_maps is only updated on https://github.com/mozilla/taar-lite/blob/238385f76cb7dade1c2801f363cd357e4bf45f6e/taar_lite/recommenders/guid_based_recommender.py#L74. I'm putting in a patch so that whenever the JSON loader is refreshed, it triggers a recomputation of the normalization tables.

crankycoder commented 6 years ago

The error in production should only happen on ETL job completion when the longitudinal DAG is run.

cc/ @jasonthomas