Datamap sheet field issues

[yulqen]

If a sheet-block (however many rows pertaining to a particular sheet) of rows in the datamap has its sheet field missing/blank, then that row will now migrate during the export process, However, if the first row of the block is intact and the other sheet fields are missing for that particular block, then the data will actually be migrated.

This will cause a problem if the gap in the sheet entries spans the boundaries between blocks because the program won't know where the change in sheet happens.

Example:

cell_key,template_sheet,cellreference,type
Project/Programme Name,,C11,TEXT  # missing sheet herel the rest of the `Introduction` sheets will migrate
Department ,Introduction,C9,TEXT
Delivery Body ,Introduction,C10,TEXT
GMPP - IPA ID Number,Introduction,C12,TEXT
Controls Project ID number,Introduction,C13,NUMBER
Project Type (for IPA use),Introduction,C14,TEXT
Classification,Introduction,C15,TEXT
GMPP (GMPP - formally joined GMPP),Introduction,C16,TEXT
Reporting period (GMPP - Snapshot Date),Introduction,C17,DATE
Snapshot Date,Introduction,C18,DATE
Senior Responsible Owner (SRO),Introduction,C21,TEXT
Senior Responsible Owner (SRO) - Email,Introduction,C22,TEXT
Senior Responsible Owner (SRO) - ID number,Introduction,C23,NUMBER
SRO Sign-Off,Introduction,C24,DATE
Bad Spacing , Introduction, c35,TEXT
String Key,Summary,B3,TEXT
Big Float,Another Sheet,F17,NUMBER
Funny Date,Another Sheet,H39,DATE

cell_key,template_sheet,cellreference,type
Project/Programme Name,Introduction,C11,TEXT
Department ,,C9,TEXT  # second row sheet missing - all `Introduction` sheets will export
Delivery Body ,Introduction,C10,TEXT
GMPP - IPA ID Number,Introduction,C12,TEXT
Controls Project ID number,Introduction,C13,NUMBER
Project Type (for IPA use),Introduction,C14,TEXT
Classification,Introduction,C15,TEXT
GMPP (GMPP - formally joined GMPP),Introduction,C16,TEXT
Reporting period (GMPP - Snapshot Date),Introduction,C17,DATE
Snapshot Date,Introduction,C18,DATE
Senior Responsible Owner (SRO),Introduction,C21,TEXT
Senior Responsible Owner (SRO) - Email,Introduction,C22,TEXT
Senior Responsible Owner (SRO) - ID number,Introduction,C23,NUMBER
SRO Sign-Off,Introduction,C24,DATE
Bad Spacing , Introduction, c35,TEXT
String Key,Summary,B3,TEXT
Big Float,Another Sheet,F17,NUMBER
Funny Date,Another Sheet,H39,DATE

cell_key,template_sheet,cellreference,type
Project/Programme Name,Introduction,C11,TEXT
Department ,Introduction,C9,TEXT
Delivery Body ,Introduction,C10,TEXT
GMPP - IPA ID Number,Introduction,C12,TEXT
Controls Project ID number,Introduction,C13,NUMBER
Project Type (for IPA use),Introduction,C14,TEXT
Classification,Introduction,C15,TEXT
GMPP (GMPP - formally joined GMPP),Introduction,C16,TEXT
Reporting period (GMPP - Snapshot Date),Introduction,C17,DATE
Snapshot Date,Introduction,C18,DATE
Senior Responsible Owner (SRO),Introduction,C21,TEXT
Senior Responsible Owner (SRO) - Email,Introduction,C22,TEXT
Senior Responsible Owner (SRO) - ID number,Introduction,C23,NUMBER
SRO Sign-Off,Introduction,C24,DATE
Bad Spacing , , c35,TEXT # missing last `Introduction` and..
String Key,,B3,TEXT #.. first summary. This row likely end up targeting `Introduction` sheet
Big Float,Another Sheet,F17,NUMBER
Funny Date,Another Sheet,H39,DATE

The code handling this is currently at engine/repository/templates.py(39),_populate_workbook(). The test istest_populate_templates_from_master.py.test_export_continues_with_missing_sheet_in_datamap`.

The best solution here is to use the initial datamap check to ensure that every datamap row has the requisite fields, and warn the user before it procedes.

[yulqen]

New test required in test_datamap_parser.py - stub created. Need to focus on engine.utils.extraction.datamap_reader() which should reject any badly formatted datamap file. All exceptions raised here should be passed to main() and handled with appropriate log messages.

[yulqen]

Fixed by raising exception when sheet field is missing, and datamaps quits, asking the user to correct the datamap.