search

webcompat / webcompat-metrics-server

Server in charge of delivering different data to the webcompat-metrics-client
Mozilla Public License 2.0
6 stars 5 forks source link

Fixes #107 - Redefines routes for category calling the DB #108

Closed karlcow closed 5 years ago

karlcow commented 5 years ago

@laghee This is currently a draft PR which will redefine the routes to call the db. The issues count is working well online,

but I still need to figure out a couple of things for testing the DB.

karlcow commented 5 years ago

ok I think I have a working solution. I need to tackle the failing tests now.

karlcow commented 5 years ago

so the tests seem to fail on CircleCI and this is half expected. Probably some issues with the config.

locally this is what I get:

→ nose2 -v --pretty-assert
test_index (tests.unit.test_web.WebTestCase)
Test the index page. ... /Users/karl/.virtualenvs/ochazuke/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py:814: UserWarning: Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".
  'Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. '
ok
test_date_range (tests.unit.test_helpers.HelpersTestCase)
Given from_date and to_date, return a list of days. ... ok
test_date_range_invalid (tests.unit.test_helpers.HelpersTestCase)
Given an invalid date, return None for the range. ... ok
test_date_range_same_day (tests.unit.test_helpers.HelpersTestCase)
Given a same day range, return the one day range. ... ok
test_get_timeline_slice (tests.unit.test_helpers.HelpersTestCase)
Given a list of dates, return the appropriate slice of data. ... ok
test_get_timeline_slice_out_of_range (tests.unit.test_helpers.HelpersTestCase)
Empty list if the dates list and the timeline do not match. ... ok
test_is_valid_args (tests.unit.test_helpers.HelpersTestCase)
Return True or False depending on the args. ... ok
test_normalize_date_range (tests.unit.test_helpers.HelpersTestCase)
Test dates normalization. ... ok
test_needsdiagnosis_invalid_param (tests.unit.test_api.APITestCase)
Ignore invalid parameters on /needsdiagnosis-timeline. ... ok
test_needsdiagnosis_invalid_param_values (tests.unit.test_api.APITestCase)
Ignore invalid parameters values on /needsdiagnosis-timeline. ... ok
test_needsdiagnosis_valid_param (tests.unit.test_api.APITestCase)
Valid parameters on /needsdiagnosis-timeline. ... ok
test_needsdiagnosis_without_params (tests.unit.test_api.APITestCase)
/data/needsdiagnosis-timeline without params fail. ... ok
test_triage_stats (tests.unit.test_api.APITestCase)
/data/triage-bugs sends back JSON. ... ok
test_weeklydata (tests.unit.test_api.APITestCase)
Send back on /data/weekly-counts a JSON. ... ok

----------------------------------------------------------------------
Ran 14 tests in 0.059s

OK
karlcow commented 5 years ago

@laghee I will try to square this before Whistler. And we have a lot of talking to do there ;) on how to make it better.

karlcow commented 5 years ago 
2019-06-04T23:14:19.870697+00:00 heroku[web.1]: State changed from crashed to starting
2019-06-04T23:14:23.179753+00:00 heroku[web.1]: Starting process with command `gunicorn ochazuke:app`
2019-06-04T23:14:26.897179+00:00 heroku[web.1]: State changed from starting to up
2019-06-04T23:14:26.449811+00:00 app[web.1]: [2019-06-04 23:14:26 +0000] [4] [INFO] Starting gunicorn 19.9.0
2019-06-04T23:14:26.450646+00:00 app[web.1]: [2019-06-04 23:14:26 +0000] [4] [INFO] Listening at: http://0.0.0.0:55063 (4)
2019-06-04T23:14:26.450771+00:00 app[web.1]: [2019-06-04 23:14:26 +0000] [4] [INFO] Using worker: sync
2019-06-04T23:14:26.457776+00:00 app[web.1]: [2019-06-04 23:14:26 +0000] [8] [INFO] Booting worker with pid: 8
2019-06-04T23:14:26.497276+00:00 app[web.1]: [2019-06-04 23:14:26 +0000] [9] [INFO] Booting worker with pid: 9
2019-06-04T23:14:27.971343+00:00 app[web.1]: Failed to find application object 'app' in 'ochazuke'
2019-06-04T23:14:27.971645+00:00 app[web.1]: [2019-06-04 23:14:27 +0000] [8] [INFO] Worker exiting (pid: 8)
2019-06-04T23:14:27.985706+00:00 app[web.1]: Failed to find application object 'app' in 'ochazuke'
2019-06-04T23:14:27.986121+00:00 app[web.1]: [2019-06-04 23:14:27 +0000] [9] [INFO] Worker exiting (pid: 9)
2019-06-04T23:14:28.174587+00:00 app[web.1]: [2019-06-04 23:14:28 +0000] [4] [INFO] Shutting down: Master
2019-06-04T23:14:28.174686+00:00 app[web.1]: [2019-06-04 23:14:28 +0000] [4] [INFO] Reason: App failed to load.
2019-06-04T23:14:29.140318+00:00 heroku[web.1]: State changed from up to crashed
2019-06-04T23:14:29.116207+00:00 heroku[web.1]: Process exited with status 4
karlcow commented 5 years ago
karlcow commented 5 years ago 
2019-06-05T00:31:08.883306+00:00 heroku[router]: at=info method=GET path="/data/needsdiagnosis-timeline?from=2019-05-01&to=2019-05-31" host=webcompat-metrics-pr-108.herokuapp.com request_id=d5504414-9f07-4ab6-bb2f-dd558974fd07 fwd="106.154.16.70" dyno=web.1 connect=1ms service=24ms status=500 bytes=455 protocol=https
2019-06-05T00:31:08.866222+00:00 app[web.1]: (2019-06-05  00:31:08 +0000) INFO: DATE_RANGE issues_count.timestamp BETWEEN :timestamp_1 AND :timestamp_2
2019-06-05T00:31:08.869816+00:00 app[web.1]: (2019-06-05  00:31:08 +0000) INFO: CATEGORY SELECT issues_count.id AS issues_count_id, issues_count.timestamp AS issues_count_timestamp, issues_count.count AS issues_count_count, issues_count.milestone AS issues_count_milestone
2019-06-05T00:31:08.869821+00:00 app[web.1]: FROM issues_count
2019-06-05T00:31:08.869825+00:00 app[web.1]: WHERE issues_count.milestone = ?
2019-06-05T00:31:08.873717+00:00 app[web.1]: (2019-06-05  00:31:08 +0000) ERROR: Exception on /data/needsdiagnosis-timeline [GET]
2019-06-05T00:31:08.873734+00:00 app[web.1]: Traceback (most recent call last):
2019-06-05T00:31:08.873738+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
2019-06-05T00:31:08.873740+00:00 app[web.1]: cursor, statement, parameters, context
2019-06-05T00:31:08.873742+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
2019-06-05T00:31:08.873745+00:00 app[web.1]: cursor.execute(statement, parameters)
2019-06-05T00:31:08.873748+00:00 app[web.1]: sqlite3.OperationalError: no such table: issues_count
2019-06-05T00:31:08.873751+00:00 app[web.1]: 
2019-06-05T00:31:08.873755+00:00 app[web.1]: The above exception was the direct cause of the following exception:
2019-06-05T00:31:08.873757+00:00 app[web.1]: 
2019-06-05T00:31:08.873759+00:00 app[web.1]: Traceback (most recent call last):
2019-06-05T00:31:08.873761+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app
2019-06-05T00:31:08.873763+00:00 app[web.1]: response = self.full_dispatch_request()
2019-06-05T00:31:08.873765+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request
2019-06-05T00:31:08.873767+00:00 app[web.1]: rv = self.handle_user_exception(e)
2019-06-05T00:31:08.873777+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception
2019-06-05T00:31:08.873779+00:00 app[web.1]: reraise(exc_type, exc_value, tb)
2019-06-05T00:31:08.873781+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise
2019-06-05T00:31:08.873783+00:00 app[web.1]: raise value
2019-06-05T00:31:08.873785+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request
2019-06-05T00:31:08.873787+00:00 app[web.1]: rv = self.dispatch_request()
2019-06-05T00:31:08.873789+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request
2019-06-05T00:31:08.873791+00:00 app[web.1]: return self.view_functions[rule.endpoint](**req.view_args)
2019-06-05T00:31:08.873793+00:00 app[web.1]: File "/app/ochazuke/api/views.py", line 59, in issues_count_data
2019-06-05T00:31:08.873795+00:00 app[web.1]: timeline = get_timeline_data(category, start, end)
2019-06-05T00:31:08.873797+00:00 app[web.1]: File "/app/ochazuke/helpers.py", line 118, in get_timeline_data
2019-06-05T00:31:08.873799+00:00 app[web.1]: issues_list = category_issues.filter(date_range).all()
2019-06-05T00:31:08.873801+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3161, in all
2019-06-05T00:31:08.873803+00:00 app[web.1]: return list(self)
2019-06-05T00:31:08.873805+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
2019-06-05T00:31:08.873807+00:00 app[web.1]: return self._execute_and_instances(context)
2019-06-05T00:31:08.873809+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3342, in _execute_and_instances
2019-06-05T00:31:08.873811+00:00 app[web.1]: result = conn.execute(querycontext.statement, self._params)
2019-06-05T00:31:08.873813+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 988, in execute
2019-06-05T00:31:08.873823+00:00 app[web.1]: return meth(self, multiparams, params)
2019-06-05T00:31:08.873826+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
2019-06-05T00:31:08.873828+00:00 app[web.1]: return connection._execute_clauseelement(self, multiparams, params)
2019-06-05T00:31:08.873830+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
2019-06-05T00:31:08.873832+00:00 app[web.1]: distilled_params,
2019-06-05T00:31:08.873834+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
2019-06-05T00:31:08.873836+00:00 app[web.1]: e, statement, parameters, cursor, context
2019-06-05T00:31:08.873838+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
2019-06-05T00:31:08.873840+00:00 app[web.1]: util.raise_from_cause(sqlalchemy_exception, exc_info)
2019-06-05T00:31:08.873842+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
2019-06-05T00:31:08.873849+00:00 app[web.1]: reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-06-05T00:31:08.873851+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
2019-06-05T00:31:08.873854+00:00 app[web.1]: raise value.with_traceback(tb)
2019-06-05T00:31:08.873856+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
2019-06-05T00:31:08.873858+00:00 app[web.1]: cursor, statement, parameters, context
2019-06-05T00:31:08.873860+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
2019-06-05T00:31:08.873862+00:00 app[web.1]: cursor.execute(statement, parameters)
2019-06-05T00:31:08.873864+00:00 app[web.1]: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: issues_count
2019-06-05T00:31:08.873873+00:00 app[web.1]: [SQL: SELECT issues_count.id AS issues_count_id, issues_count.timestamp AS issues_count_timestamp, issues_count.count AS issues_count_count, issues_count.milestone AS issues_count_milestone
2019-06-05T00:31:08.873876+00:00 app[web.1]: FROM issues_count
2019-06-05T00:31:08.873878+00:00 app[web.1]: WHERE issues_count.milestone = ? AND issues_count.timestamp BETWEEN ? AND ?]
2019-06-05T00:31:08.873880+00:00 app[web.1]: [parameters: ('needsdiagnosis', '2019-05-01', '2019-06-01')]
2019-06-05T00:31:08.873883+00:00 app[web.1]: (Background on this error at: http://sqlalche.me/e/e3q8)
2019-06-05T00:31:08.884390+00:00 app[web.1]: 10.63.66.107 - - [05/Jun/2019:00:31:08 +0000] "GET /data/needsdiagnosis-timeline?from=2019-05-01&to=2019-05-31 HTTP/1.1" 500 290 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:69.0) Gecko/20100101 Firefox/69.0"
karlcow commented 5 years ago

Ah understood, forgotten to specify the right DB URL for production.

karlcow commented 5 years ago

strange. because if I ask.

heroku config -a webcompat-metrics-pr-108

I get the config for DATABASE_URL why did it stick with sqlite?

karlcow commented 5 years ago

ahah I had a typo in the config file. This is fixed.

@laghee switching the status of this PR to actual PR.

It's huge, it's ugly and a bit messy because it shows my missteps. If you prefer. I'll rebase everything with a cleaner set of commits.

karlcow commented 5 years ago

conflicts, I guess I should probably merge master and push again.

karlcow commented 5 years ago

@laghee Resolved the conflicts.

karlcow commented 5 years ago

ah I messed up triage.json Let see.

karlcow commented 5 years ago 
(ochazuke) 15:06:22 ~/code/ochazuke
→ nose2
..............
----------------------------------------------------------------------
Ran 14 tests in 0.065s

OK
(ochazuke) 15:06:30 ~/code/ochazuke
→ flake8 .

Alleluiah. Tests ok. No conflicts. pfew.