Data Import + mapping Instrument error.

[tylerstright]

I am building a water temperature dataset containing two header fields, two detail fields, and when I assign the dataset to a project I include Instrument in the mix. I then go and add a few locations and instruments for testing.

EDIT : Duplicate checking is not enabled.

If I enter data manually, I select both Instrument and Location from select menus. I do tend to see some errors I don’t understand in devtools, but those don’t seem to interfere with the desired function.

If I import a CSV with only datetime and temperature fields, it takes me into the edit view and I am again able to select Instrument and Location (i.e. 1 location, 1 instrument = 1 Activity), along with observers/comments. Import success.

If I combine files in an attempt to import multiple activities at once, things start breaking. In my test file, I have two different sites, each with a different instrument (all existing in the Instruments List and Locations for the project). After selecting the CSV and mapping fields, it takes me to the preview where I see both Instrument and Location have maintained their values. However, the Activity count shows 0, and when I try to save, here's what is logged in the console:

[nowinski]

Hi Tyler,

Thanks for reporting this. Please send me your CSV file. Also, please let me know if duplicate checking is enabled for this data set? If so, which fields are included in the config?

Thanks Joe

[nowinski]

There are a few things going. Here's an update:

1. CDMS web app checks dataset.Config.EnableDuplicateChecking property in multiple places (modal-activities-grid.js, dataset-edit-form,js) but this property does not exist until duplicate checking is configured at the dataset level. We can avoid this for new datasets by setting a default value in datastore config (DatastoreController.cs line 430 ish):

datastore.DefaultConfig = "{\"AllowSaveWithErrors\":false,\"EnableDuplicateChecking\":true,\"DuplicateCheckFields\":[\"ActivityDate\",\"LocationId\"]}";

2. It's possible the multi-activity importer was not designed to handle Instruments and other special fields. Note that Instruments is a special header field (i.e. an Add button with extra functionality) on the data entry page but a normal grid field on the multi-activity import page. Ken is looking into this

3. Adding the instrument field to the duplicate check config causes that field to fail validation on the data entry page. Again, this is probably because instruments is a special header field.

[nowinski]

Pushed three commits to address these issues:

1. Automatically enable duplicate checking when new datastores are created and set ActivityDate and LocationId as default duplicate check fields. This change only applies to new datastores and related datasets. https://github.com/CRITFC/tribalCDMS-be/commit/bb67d702446faac86e1fd9ea77b2989035c75054

2. Instrument Id is a column in the Activity table and requires special handling for importing. Mapped Instrument names and Ids and added InstrumentId to the Activity dup check object. https://github.com/CRITFC/tribalCDMS-fe/commit/4ac2ebb3ba461589cf66e25e522cc7008a15a2df

3. Added special handling to grid-service.js to resolve validation issues on data entry page https://github.com/CRITFC/tribalCDMS-fe/commit/9d52330c54bc982d6929394509074e2db8b2e962

[nowinski]

Modified dataset-import.js to catch/validated unmatched instruments: https://github.com/CRITFC/tribalCDMS-fe/commit/4156114becb03d31002146c9fd54430243fc8990

Resolved related duplicate check conflict: https://github.com/CRITFC/tribalCDMS-fe/commit/83e59b9c8c501cc4f16e6c9204af5382728cc6c2