security-force-monitor / sfm-cms

Platform for sharing complex information about security forces. Powers WhoWasInCommand.com
https://whowasincommand.com
10 stars 3 forks source link

Update importer to accept country code as an argument and prepend error files with it #790

Closed hancush closed 2 years ago

hancush commented 2 years ago

Previously, entity sheets came to us with country code prefixes, which transferred to the error file names. This is no longer the case, causing errors for all countries in an import to show up in a single, e.g., persons-errors.csv file instead of being organized by country.

Update theimport_google_doc command to accept a country code argument and prepend it to error files.

Also update the docket Makefile to pass the new country code arg to the import command. N.b., the country codes are already provided in each docket. See: https://github.com/security-force-monitor/sfm-cms/blob/master/fixtures/import_docket.csv

smcalilly commented 2 years ago

and prepend it to error files.

@hancush do you mean to prepend it to the error filename? Or does the country code need to be in a new column? And then add each country code to the row in the error file?

tlongers commented 2 years ago

@hannah As discussed on email, we can provide the two letter country value via the import docket system (e.g ml for Mali).

However, us being us, I can see situations where we pass a sheet to the importer that still has the two letter ISO code in it (ml_units etc) . In this case, it should probably strip and ignore it, and defer to the value from the import docket.

hancush commented 2 years ago

Thanks for the clarification, @tlongers!

@smcalilly, great question. Prepend it to the filename. Also see Tom's note – it may happen that there are some legacy sheets with the country code still in the tab names. In that case, it should be stripped and superseded by the country code passed to the importer.