search

Clinical-Genomics / scout

VCF visualization interface
https://clinical-genomics.github.io/scout
BSD 3-Clause "New" or "Revised" License
152 stars 46 forks source link

Coverage report alchemy issue? #4860

Closed dnil closed 2 months ago

dnil commented 2 months ago

Thank you for taking the time to contribute an issue!

Describe the bug A couple of coverage report generation crashes:

2024-09-16 17:36:46,144 - scout.server.app - ERROR: Exception on /cust002/F0063456/pdf_report [GET] [in /venv/lib/python3.11/site-packages/flask/app.py:838]
Traceback (most recent call last):
 File "/venv/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
   response = self.full_dispatch_request()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
   rv = self.handle_user_exception(e)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask_cors/extension.py", line 194, in wrapped_function
   return cors_after_request(app.make_response(f(*args, **kwargs)))
                                               ^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
   rv = self.dispatch_request()
        ^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
   return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/worker/app/scout/server/blueprints/cases/views.py", line 280, in pdf_case_report
   data["coverage_report"] = controllers.coverage_report_contents(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/worker/app/scout/server/blueprints/cases/controllers.py", line 137, in coverage_report_contents
   html_body_content = resp.text.split("<body>")[1].split("</body>")[0]
                       ~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

AND

2024-09-16 17:36:30,966 - scout.server.app - ERROR: Exception on /reports/report [POST] [in /venv/lib/python3.11/site-packages/flask/app.py:838]
Traceback (most recent call last):
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
   self.dialect.do_execute(
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
   cursor.execute(statement, parameters)
 File "/venv/lib/python3.11/site-packages/pymysql/cursors.py", line 153, in execute
   result = self._query(query)
            ^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/cursors.py", line 322, in _query
   conn.query(q)
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 563, in query
   self._affected_rows = self._read_query_result(unbuffered=unbuffered)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 825, in _read_query_result
   result.read()
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 1199, in read
   first_packet = self.connection._read_packet()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 744, in _read_packet
   packet_header = self._read_bytes(4)
                   ^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 798, in _read_bytes
   raise err.OperationalError(
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/venv/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
   response = self.full_dispatch_request()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
   rv = self.handle_user_exception(e)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask_cors/extension.py", line 194, in wrapped_function
   return cors_after_request(app.make_response(f(*args, **kwargs)))
                                               ^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
   rv = self.dispatch_request()
        ^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
   return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/views.py", line 132, in report
   return render_template("report/report.html", **data)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask/templating.py", line 150, in render_template
   return _render(app, template, context)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask/templating.py", line 131, in _render
   rv = template.render(context)
        ^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/jinja2/environment.py", line 1304, in render
   self.environment.handle_exception()
 File "/venv/lib/python3.11/site-packages/jinja2/environment.py", line 939, in handle_exception
   raise rewrite_traceback_stack(source=source)
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/templates/report/report.html", line 252, in top-level template code
   {% if hidden %}
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/templates/report/layouts/base.html", line 36, in top-level template code
   {% block body %}
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/templates/report/layouts/base.html", line 51, in block 'body'
   {% block main %}{% endblock %}
   ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/templates/report/report.html", line 35, in block 'main'
   {{ overview_table(sex_rows) }}
   ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/jinja2/runtime.py", line 782, in _invoke
   rv = self._func(*arguments)
        ^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/templates/report/report.html", line 73, in template
   {% for sample in sex_rows %}
   ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/utils.py", line 80, in samplesex_rows
   for sample_id, chromosomes in samples:
 File "/venv/lib/python3.11/site-packages/sqlalchemy/orm/query.py", line 2813, in __iter__
   result = self._iter()
            ^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/orm/query.py", line 2827, in _iter
   result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
                                                 ^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
   return self._execute_internal(
          ^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2247, in _execute_internal
   result: Result[Any] = compile_state_cls.orm_execute_statement(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
   result = conn.execute(
            ^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
   return meth(
          ^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
   return connection._execute_clauseelement(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
   ret = self._execute_context(
         ^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
   return self._exec_single_context(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
   self._handle_dbapi_exception(
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
   raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
   self.dialect.do_execute(
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
   cursor.execute(statement, parameters)
 File "/venv/lib/python3.11/site-packages/pymysql/cursors.py", line 153, in execute
   result = self._query(query)
            ^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/cursors.py", line 322, in _query
   conn.query(q)
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 563, in query
   self._affected_rows = self._read_query_result(unbuffered=unbuffered)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 825, in _read_query_result
   result.read()
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 1199, in read
   first_packet = self.connection._read_packet()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 744, in _read_packet
   packet_header = self._read_bytes(4)
                   ^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 798, in _read_bytes
   raise err.OperationalError(
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
[SQL: SELECT transcript_stat.sample_id AS transcript_stat_sample_id, transcript.chromosome AS transcript_chromosome, avg(transcript_stat.mean_coverage) AS avg_1
FROM transcript_stat INNER JOIN transcript ON [transcript.id](http://transcript.id/) = transcript_stat.transcript_id
WHERE transcript.chromosome IN (%(chromosome_1_1)s, %(chromosome_1_2)s) AND transcript_stat.sample_id IN (%(sample_id_1_1)s) GROUP BY transcript_stat.sample_id, transcript.chromosome]
[parameters: {'chromosome_1_1': 'X', 'chromosome_1_2': 'Y', 'sample_id_1_1': 'ACC15259A4'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)2024-09-16 17:36:30,966 - scout.server.app - ERROR: Exception on /reports/report [POST] [in /venv/lib/python3.11/site-packages/flask/app.py:838]
Traceback (most recent call last):
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
   self.dialect.do_execute(
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
   cursor.execute(statement, parameters)
 File "/venv/lib/python3.11/site-packages/pymysql/cursors.py", line 153, in execute
   result = self._query(query)
            ^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/cursors.py", line 322, in _query
   conn.query(q)
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 563, in query
   self._affected_rows = self._read_query_result(unbuffered=unbuffered)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 825, in _read_query_result
   result.read()
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 1199, in read
   first_packet = self.connection._read_packet()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 744, in _read_packet
   packet_header = self._read_bytes(4)
                   ^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 798, in _read_bytes
   raise err.OperationalError(
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/venv/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
   response = self.full_dispatch_request()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
   rv = self.handle_user_exception(e)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask_cors/extension.py", line 194, in wrapped_function
   return cors_after_request(app.make_response(f(*args, **kwargs)))
                                               ^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
   rv = self.dispatch_request()
        ^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
   return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/views.py", line 132, in report
   return render_template("report/report.html", **data)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask/templating.py", line 150, in render_template
   return _render(app, template, context)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/flask/templating.py", line 131, in _render
   rv = template.render(context)
        ^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/jinja2/environment.py", line 1304, in render
   self.environment.handle_exception()
 File "/venv/lib/python3.11/site-packages/jinja2/environment.py", line 939, in handle_exception
   raise rewrite_traceback_stack(source=source)
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/templates/report/report.html", line 252, in top-level template code
   {% if hidden %}
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/templates/report/layouts/base.html", line 36, in top-level template code
   {% block body %}
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/templates/report/layouts/base.html", line 51, in block 'body'
   {% block main %}{% endblock %}
   ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/templates/report/report.html", line 35, in block 'main'
   {{ overview_table(sex_rows) }}
   ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/jinja2/runtime.py", line 782, in _invoke
   rv = self._func(*arguments)
        ^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/templates/report/report.html", line 73, in template
   {% for sample in sex_rows %}
   ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/chanjo_report/server/blueprints/report/utils.py", line 80, in samplesex_rows
   for sample_id, chromosomes in samples:
 File "/venv/lib/python3.11/site-packages/sqlalchemy/orm/query.py", line 2813, in __iter__
   result = self._iter()
            ^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/orm/query.py", line 2827, in _iter
   result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
                                                 ^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
   return self._execute_internal(
          ^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2247, in _execute_internal
   result: Result[Any] = compile_state_cls.orm_execute_statement(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
   result = conn.execute(
            ^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
   return meth(
          ^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
   return connection._execute_clauseelement(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
   ret = self._execute_context(
         ^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
   return self._exec_single_context(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
   self._handle_dbapi_exception(
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
   raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
   self.dialect.do_execute(
 File "/venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
   cursor.execute(statement, parameters)
 File "/venv/lib/python3.11/site-packages/pymysql/cursors.py", line 153, in execute
   result = self._query(query)
            ^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/cursors.py", line 322, in _query
   conn.query(q)
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 563, in query
   self._affected_rows = self._read_query_result(unbuffered=unbuffered)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 825, in _read_query_result
   result.read()
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 1199, in read
   first_packet = self.connection._read_packet()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 744, in _read_packet
   packet_header = self._read_bytes(4)
                   ^^^^^^^^^^^^^^^^^^^
 File "/venv/lib/python3.11/site-packages/pymysql/connections.py", line 798, in _read_bytes
   raise err.OperationalError(
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
[SQL: SELECT transcript_stat.sample_id AS transcript_stat_sample_id, transcript.chromosome AS transcript_chromosome, avg(transcript_stat.mean_coverage) AS avg_1
FROM transcript_stat INNER JOIN transcript ON [transcript.id](http://transcript.id/) = transcript_stat.transcript_id
WHERE transcript.chromosome IN (%(chromosome_1_1)s, %(chromosome_1_2)s) AND transcript_stat.sample_id IN (%(sample_id_1_1)s) GROUP BY transcript_stat.sample_id, transcript.chromosome]
[parameters: {'chromosome_1_1': 'X', 'chromosome_1_2': 'Y', 'sample_id_1_1': 'ACC15259A4'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

It looks like regular reports on this case work just fine now.

To Reproduce Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

northwestwitch commented 2 months ago

Tested now and the report from that case works.. What has changed? Was it the default report or some specific panel?

dnil commented 2 months ago

Tested now and the report from that case works.. What has changed? Was it the default report or some specific panel?

Agreed. Could have been something custom I guess. I’ve forwarded the question to the most likely user. Some restart or other temporary interruption on the MySQL side perhaps? For now it feels like nothing to worry much about.

dnil commented 2 months ago

From what I understand it came back as two empty reports and worked on the third try after filling in another comment. Probably transient.