search

CascadesCarnivoreProject / Carnassial

Carnassial: Simplifying Remote Camera Data
Other
3 stars 0 forks source link

Really fix the CSV Import to make it robust. #8

Closed saulgreenberg closed 8 years ago

saulgreenberg commented 8 years ago
  1. If a user tries to change the CSV column order (or add/delete columns), data will be imported into the wrong column. SUGGESTED FIX: Check the column names against the datalabel. If they differ, either abort, or make sure updates are done to the datalabel as identified by the column name,
  2. If a user changes a value to a non-legitmate one (e.g., Foo instead of False for flags, non-numbers for counters, etc., strings that don’t match fixed choices) the data should be considered corrupted. SUGGESTED FIX: Check the data type against the format that is required (which you can find by examining the data label.) If it doesn’t conform, then indicate the problem and abort.
  3. [fixed previously] Changing File / Folder names will likely screw things up seriously. SUGGESTED FIX: Ignore any changes to File/Folder names
  4. [fixed previously] Importing often changes the format of the dates and time, which then affects the export format. This is because some spreadsheet packages – like excel – read in dates and time and changes it to a particular format. So this can lead to an undesired change. SUGGESTED FIX: Ignore any changes to File/Folder names, or parse the dates to change them into the standard Timelapse date representation.
saulgreenberg commented 8 years ago

Todd wrote: Could you give me some context on this comment in ImageDatabase.PopulateDataLabelMaps()? // don't type map user defined controls as if there are multiple ones the key would not be unique Given the effort the editor goes to ensure data labels are unique the constraint is removable, no?

Todd

Saul replied...

DataLabelFromStandardControlType holds the DataLabel and Label names for the standard controls found in all timelapse templates (e.g., File, Folder, etc), where it serves as a lookup table to get the DataLabel name from a control type (e.g., given a file type, it will return File).

ControlTypeFromDataLabel does the opposite, but for all controls i.e., given a data label name, it returns the type of the control (e.g., note, counter, date, etc).

I can reconstruct why I added the comment “// We don't want to add these types to the hash, as there can be multiple ones, which means the key would not be unique”

But It think it works something like this. There is (for example) only one DataType called (say) Date, so getting its DataLabel is a one to one mapping.

However, if I added (say) all Counts to the list, there could be multiple items of the type Count. So if I asked for the DataLabel accompanying the type Count, there could be multiple matches (i.e., one to many). This is why those types are excluded. So I guess the real answer is that this.DataLabelFromStandardControlType is only useful for finding the datalabels that matches the standard control types.

I’m not sure if the constraint is removable … would have to walk through the new code to see what is going on.

twest820 commented 8 years ago

[8a8c937]