search

CUB-Libraries-CTA / counter-data-loader

Loads COUNTER database from JR1 report spreadsheets
1 stars 2 forks source link

Pre-process excel table input to determine if format errors are found (5) #55

Closed ericnienhouse closed 1 year ago

ericnienhouse commented 1 year ago

Problem: To identify errors in excel input file and entire load workflow is necessary. This is time consuming. It would be helpful to know in advance is an excel spreadsheet contains errors.

Estimate: 5

Acceptance Criteria:

Improve the pre-processing step to identify bad rows in excel file by row number (and not rename the file thereby preventing further processing).

bonnland commented 1 year ago

The preprocessing step (from preprocess-source-files.py) now will skip renaming Excel files if they contain rows with missing values.

For example when processing the Excel file TR_B3 2021_ProQuest.xlsx:

$ python preprocess-source-files.py /Users/Shared/AmazonMirror/workingFiles
Traceback (most recent call last):
  File "/Users/brbo3303/PycharmProjects/counter-data-loader/preprocess-source-files.py", line 80, in <module>
    raise Exception(error_msg)
Exception: TR_B3 2021_ProQuest.xlsx:  is missing part of (Title, Publisher, Platform) on these rows: [  171 172 173 174 175 176 177 178 179 180 181 182 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 ]

The error is placed in the log file errors.log, which is where other errors are being logged in the preprocessing step.

bonnland commented 1 year ago

Latest commit with these code changes: https://github.com/culibraries/counter-data-loader/commit/dfa0ca5ec62f54c57b88c12270a9f3b17cb8bd8b