search

eastgenomics / rd_wgs_workbooks_monitoring

Monitor progrees of RD WGS workbooks generation
0 stars 0 forks source link

Add database integration #2

Closed kjwinfield closed 4 months ago

kjwinfield commented 4 months ago

This has been tested up to the point of launching jobs. The script can grab JSON file IDs from the database, perform the JSON R number check and start eggd_generate_rd_wgs_workbook jobs for the JSONs that pass. The script also successfully updates the database throughout those stages.

Example terminal output from running:

   ReferralNumberID ReferralNumber  StatusReferralNumberID  ... CICode                     JSONFileID XLSXFileID
0                 1   r22057801644                       5  ...   None  file-GpVpqv04f83Y3J6jFZY0JfBq       None
1                 2   r22119920675                       5  ...   None  file-GpVpqvQ4f83Vb8qgpzJFQxqz       None
2                 3   r22317767696                       5  ...   None  file-GpVpqvj4f83gZk7V10pBgjjV       None
3                 4   r23000679682                       5  ...   None  file-GpVpxKj4f83Qz1P21xJ90bQX       None
4                 5   r23006110622                       5  ...   None  file-GpVpxP84f83Qz1P21xJ90bQZ       None

[5 rows x 8 columns]
Checking that the R number in JSON is correct...
Checking for records in staus JSONCheckPass or DXJobStarted
H:\rd_wgs_workbooks_monitoring\rd_wgs_workbooks_monitoring.py:156: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.
  df = pd.read_sql(query, conn)
Launching DNAnexus jobs...
Launched DX job job-GpYqpX8479V8PQk5K2pF9vv2 for r22057801644 (file-GpVpqv04f83Y3J6jFZY0JfBq)
Launched DX job job-GpYqpX8479V8JkgzFq0Pv5x5 for r22119920675 (file-GpVpqvQ4f83Vb8qgpzJFQxqz)
Launched DX job job-GpYqpXQ479V4K6PbzvY1zkzB for r22317767696 (file-GpVpqvj4f83gZk7V10pBgjjV)
Launched DX job job-GpYqpXQ479V8PQk5K2pF9vv7 for r23000679682 (file-GpVpxKj4f83Qz1P21xJ90bQX)
Launched DX job job-GpYqpXQ479V7v0p8xy668JQY for r23006110622 (file-GpVpxP84f83Qz1P21xJ90bQZ)
Pausing to allow jobs to complete...
Checking status of eggd_generate_rd_wgs_workbook jobs...
Job job-GpYqpX8479V8PQk5K2pF9vv2 for JSON file-GpVpqv04f83Y3J6jFZY0JfBq has status failed. XLSX report not generated; record in Shire will remain in status DXJobStarted and will run again on next running of this script.
Job job-GpYqpX8479V8JkgzFq0Pv5x5 for JSON file-GpVpqvQ4f83Vb8qgpzJFQxqz has status failed. XLSX report not generated; record in Shire will remain in status DXJobStarted and will run again on next running of this script.
Job job-GpYqpXQ479V4K6PbzvY1zkzB for JSON file-GpVpqvj4f83gZk7V10pBgjjV has status failed. XLSX report not generated; record in Shire will remain in status DXJobStarted and will run again on next running of this script.
Job job-GpYqpXQ479V8PQk5K2pF9vv7 for JSON file-GpVpxKj4f83Qz1P21xJ90bQX has status failed. XLSX report not generated; record in Shire will remain in status DXJobStarted and will run again on next running of this script.
Job job-GpYqpXQ479V7v0p8xy668JQY for JSON file-GpVpxP84f83Qz1P21xJ90bQZ has status failed. XLSX report not generated; record in Shire will remain in status DXJobStarted and will run again on next running of this script.`

These jobs failed due to JSON incompatibility with eggd_generate_rd_wgs_workbook which will need to be fixed within that app.

This change is Reviewable

pep8speaks commented 4 months ago

Hello @kjwinfield! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers:

Comment last updated at 2024-07-30 12:08:06 UTC