search

Indicia-Team / warehouse

GNU General Public License v3.0
5 stars 3 forks source link

Enhance csv import to update existing records #286

Open JimBacon opened 6 years ago

JimBacon commented 6 years ago

It is already possible to insert new sample and occurrence records in to the Indicia database by importing them from a CSV file. There is now a need to be able to update sample records from csv import.

It is anticipated that the external key field will form the basis of identifying when a record already exists in order to update rather than insert a new record. Where custom attributes are imported as part of an update then they may create new or update existing attributes according to circumstances. Multi-value custom attributes pose a particular problem but are not included in the current requirement. Deletion of records is not part of the current requirement.

Gary-van-Breda commented 6 years ago

There is a project just about to start (for the German Mycological Society) that has as part of its requirement that the CSV import functionality should be upgraded to allow updates of existing records. They are paying for this work.

JimBacon commented 6 years ago

Thanks. I have checked with my project manager and I can wait to receive your contribution. Un-assigning myself.

johnvanbreda commented 6 years ago

@Gary-van-Breda can you update this with the current status please? Is this released and usable?

JimBacon commented 6 years ago

@johnvanbreda, did you receive an update on this?

johnvanbreda commented 6 years ago

I've not used it myself but I understand that the code is done and working, but not tested in anger yet.

JimBacon commented 6 years ago

I've just tried it with some success but I can't achieve what I wanted.

My example is using the occurrence import on the warehouse with a data set including date, spatial reference, recorder, taxon and an external_key on each occurrence. The initial import was done a couple of years ago and I am now uploading an update.

Everything proceeds as previously but there is a new option to look up existing records. image I chose to look up occurrence records by external key and not lookup sample records. The upshot was that my occurrences were updated as hoped but they were linked to new samples. All the original samples were left but with no occurrences. On reflection, I understand why I need to look up samples as well.

I tried again, this time intending to also look up existing sample records. Unfortunately all the options are greyed out because my import does not contain the fields it proposes to use for matching samples. There is a Grid Ref and Date option but this does not appear to recognise dates in the vague date format of date_start, date_end and date_type..

johnvanbreda commented 6 years ago

Ok, let me know if you need me to look into a fix.

JimBacon commented 6 years ago

For reference, see https://github.com/Indicia-Team/warehouse/commit/f227225bcd64fe44f56a953b06de2ab1272b1aa3, https://github.com/Indicia-Team/warehouse/commit/04ce5b621f9e94c1ac0315d56c08f18b4b240bbc, https://github.com/Indicia-Team/warehouse/commit/2903f96031a47f4b95feb119589a3e152717470c, https://github.com/Indicia-Team/warehouse/commit/448ba5138acf6236a7486b470e4fd9aabf241ac1, https://github.com/Indicia-Team/warehouse/commit/3de32844e355f33a9d142f99fe6c84270c8e7c43, https://github.com/Indicia-Team/warehouse/commit/6e35123376a7bd4ca5761bd9f8631207cb59fd18 and https://github.com/Indicia-Team/client_helpers/commit/adc5dd3857b81a031a276b9f557e17fd72fc9bef