Missing log message for failed COVIDcast CSV files

[krivard]

There is some failure case that's not being adequately logged:

handling  /common/covidcast/receiving/jhu-csse/20200828_state_deaths_7dav_cumulative_prop.csv
deaths_7dav_cumulative_prop False
archiving as failed - jhu-csse

Correct handling of a failed CSV is logged like this:

handling  /common/covidcast/receiving/jhu-csse/20200828_county_deaths_incidence_num.csv
deaths_incidence_num False
 invalid value for Pandas(geo_id='.0000', val='7.0', se=nan, sample_size=nan) (geo_id)
archiving as failed - jhu-csse

[krivard]

The file in question: 20200828_state_deaths_7dav_cumulative_prop.csv.zip

[benjaminysmith]

I can take a look at this -- can you point me at the code where this error occurs?

[dfarrow0]

here's where the correct handling example is logged: https://github.com/cmu-delphi/delphi-epidata/blob/801f2729ea80b42891aa282c858c489a47049082/src/acquisition/covidcast/csv_importer.py#L302

it's unexpected that a file would be archived as failed without any corresponding log message describing the failure reason.

[benjaminysmith]

Thanks David -- that looks like enough to start debugging the issue. To understand more broadly, where does this code get called from? In one of the automation steps?

[dfarrow0]

yep, that's right. here's how it happens:

• task#33 runs step#63 every hour
• step#63 causes the steps in flow#7 to run sequentially
• the first step in flow#7 is step#60, which runs the python file csv_to_database.py
• csv_to_database imports and calls CsvImporter from file csv_importer.py

[benjaminysmith]

Still debugging, adding some findings:

1. The rows seem to parse fine (i.e., cc_rows = CovidcastRow.fromCsvRows(csv_rows, source, signal, time_type, geo_type, time_value, issue, lag, is_wip) returns a list of rows from the file:

ak 4.7801364393158705 None None al 40.11095411888978 None None ar 23.64538679687046 None None az 66.50701137105816 None None ...

2. The version of the code that executed is prior to this patch, which is why the error message shows the deaths_7dav_cumulative_prop False line -- this is the print(signal, is_wip) from line 80.

[benjaminysmith]

I suspect that the root cause of the problem may be that there is an infinite value in the data: pr inf None None -- my best guess right now is that this causes the mysql insert to fail, causing a return value of 0 rows. In the case where 0 is returned this falls through to archive_as_failed which would match the error messages in this issue.

[krivard]

Fixed in PR.