Some top-insights are returning 500 ISE on dev

[Nebrethar]

Describe the bug:

http://localhost:5000/api/unstable/repo-groups/21/top-insights 
http://localhost:5000/api/unstable/repo-groups/22/top-insights 
http://localhost:5000/api/unstable/repo-groups/23/top-insights 
http://localhost:5000/api/unstable/repo-groups/24/top-insights 
http://localhost:5000/api/unstable/repo-groups/25150/top-insights 
http://localhost:5000/api/unstable/repo-groups/25151/top-insights 

Calling these front page endpoint URLs results in 500 errors. I have included my trace at the bottom of this post. It seems to be related to the variable cms_id which does not seem to exist in this context.

To Reproduce: Steps to reproduce the behavior:

1. Go to the front page of augur - dev
2. Open console
3. See errors

Expected behavior: I did not know what to expect, but it wasn't that.

Screenshots

Desktop (please complete the following information):

• OS: Windows 10
• Browser: Chrome 76

Additional context

Traceback (most recent call last):
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedColumn: column "cms_id" does not exist
LINE 3:                 ri_date AS date, cms_id AS rating, ri_fresh ...
                                         ^
HINT:  Perhaps you meant to reference the column "repo_insights.ri_id".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/augur-0.9.8-py3.6.egg/augur/server.py", line 368, in generated_function
    data = self.transform(func, args, kwargs)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/augur-0.9.8-py3.6.egg/augur/server.py", line 312, in transform
    data = func(*args, **kwargs)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/augur-0.9.8-py3.6.egg/augur/datasources/augur_db/augur_db.py", line 2991, in top_insights
    results = pd.read_sql(topInsightsSQL, self.db, params={'repo_group_id': repo_group_id, 'num_repos': num_repos})
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/pandas/io/sql.py", line 436, in read_sql
    chunksize=chunksize,
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/pandas/io/sql.py", line 1218, in read_query
    result = self.execute(*args)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/pandas/io/sql.py", line 1087, in execute
    return self.connectable.execute(*args, **kwargs)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2169, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
    e, statement, parameters, cursor, context
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column "cms_id" does not exist
LINE 3:                 ri_date AS date, cms_id AS rating, ri_fresh ...
                                         ^
HINT:  Perhaps you meant to reference the column "repo_insights.ri_id".

[SQL:
            SELECT rg_name, repo.repo_group_id, repo_insights.repo_id, repo_git, ri_metric, ri_value AS value,
                ri_date AS date, cms_id AS rating, ri_fresh AS discovered
            FROM repo_insights JOIN repo ON repo.repo_id = repo_insights.repo_id JOIN repo_groups ON repo.repo_group_id = repo_groups.repo_group_id
            WHERE repo_insights.repo_id IN (
                SELECT repo_id
                FROM repo
                WHERE repo_group_id = %(repo_group_id)s
                AND repo_id IN (SELECT repo_id FROM repo_insights WHERE data_collection_date > CURRENT_DATE - INTERVAL '10' DAY GROUP BY repo_id, ri_id HAVING 304 > count(repo_insights.repo_id) ORDER BY ri_id desc)
                LIMIT %(num_repos)s
            )
        ]
[parameters: {'repo_group_id': '21', 'num_repos': 6}]
(Background on this error at: http://sqlalche.me/e/f405)2019-09-03 07:46:42 DESKTOP-1815CFA augur.server[161] ERROR Exception on /api/unstable/repo-groups/23/top-insights [GET]
Traceback (most recent call last):
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedColumn: column "cms_id" does not exist
LINE 3:                 ri_date AS date, cms_id AS rating, ri_fresh ...
                                         ^
HINT:  Perhaps you meant to reference the column "repo_insights.ri_id".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/augur-0.9.8-py3.6.egg/augur/server.py", line 368, in generated_function
    data = self.transform(func, args, kwargs)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/augur-0.9.8-py3.6.egg/augur/server.py", line 312, in transform
    data = func(*args, **kwargs)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/augur-0.9.8-py3.6.egg/augur/datasources/augur_db/augur_db.py", line 2991, in top_insights
    results = pd.read_sql(topInsightsSQL, self.db, params={'repo_group_id': repo_group_id, 'num_repos': num_repos})
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/pandas/io/sql.py", line 436, in read_sql
    chunksize=chunksize,
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/pandas/io/sql.py", line 1218, in read_query
    result = self.execute(*args)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/pandas/io/sql.py", line 1087, in execute
    return self.connectable.execute(*args, **kwargs)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2169, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
    e, statement, parameters, cursor, context
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/mnt/c/Clones/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column "cms_id" does not exist
LINE 3:                 ri_date AS date, cms_id AS rating, ri_fresh ...
                                         ^
HINT:  Perhaps you meant to reference the column "repo_insights.ri_id".

[SQL:
            SELECT rg_name, repo.repo_group_id, repo_insights.repo_id, repo_git, ri_metric, ri_value AS value,
                ri_date AS date, cms_id AS rating, ri_fresh AS discovered
            FROM repo_insights JOIN repo ON repo.repo_id = repo_insights.repo_id JOIN repo_groups ON repo.repo_group_id = repo_groups.repo_group_id
            WHERE repo_insights.repo_id IN (
                SELECT repo_id
                FROM repo
                WHERE repo_group_id = %(repo_group_id)s
                AND repo_id IN (SELECT repo_id FROM repo_insights WHERE data_collection_date > CURRENT_DATE - INTERVAL '10' DAY GROUP BY repo_id, ri_id HAVING 304 > count(repo_insights.repo_id) ORDER BY ri_id desc)
                LIMIT %(num_repos)s
            )
        ]
[parameters: {'repo_group_id': '23', 'num_repos': 6}]
(Background on this error at: http://sqlalche.me/e/f405)

127.0.0.1 - [03/Sep/2019:07:46:42 -0500] - GET /api/unstable/repo-groups/21/top-insights HTTP/1.1
127.0.0.1 - [03/Sep/2019:07:46:42 -0500] - GET /api/unstable/repo-groups/23/top-insights HTTP/1.1
127.0.0.1 - [03/Sep/2019:07:46:42 -0500] - GET /api/unstable/repos HTTP/1.1

[Nebrethar]

Confirmed on a fresh checkout of dev

[sgoggins]

@gabe-heim : This is related it appears to our removal of the cms_id column as you were doing insights work? Can we fix the API quick?

LINE 3:                 ri_date AS date, cms_id AS rating, ri_fresh ...

[SQL:
            SELECT rg_name, repo.repo_group_id, repo_insights.repo_id, repo_git, ri_metric, ri_value AS value,
                ri_date AS date, cms_id AS rating, ri_fresh AS discovered
            FROM repo_insights JOIN repo ON repo.repo_id = repo_insights.repo_id JOIN repo_groups ON repo.repo_group_id = repo_groups.repo_group_id
            WHERE repo_insights.repo_id IN (
                SELECT repo_id
                FROM repo
                WHERE repo_group_id = %(repo_group_id)s
                AND repo_id IN (SELECT repo_id FROM repo_insights WHERE data_collection_date > CURRENT_DATE - INTERVAL '10' DAY GROUP BY repo_id, ri_id HAVING 304 > count(repo_insights.repo_id) ORDER BY ri_id desc)
                LIMIT %(num_repos)s
            )
        ]

[gabe-heim]

@sgoggins there is an api change that is needed as well as some frontend logic that references that column, I am fixing it now

[gabe-heim]

Fixed in 2e8e05f.