duplicate check - Githubissues

glamod / glamod-marine-processing

Code and configuration for marine data processing

Apache License 2.0

0 stars 2 forks source link

Closed ludwiglierhammer closed 3 days ago

ludwiglierhammer commented 3 months ago

We want to implement a duplicate check in the obs_suite

ludwiglierhammer commented 3 months ago

@aanderss:

What exactly are duplicates?
- identical lines are already removed in the cdm_mapper
- identicalreport_id/observation_id
- identicallatitude and longitude
- identicalreport_timestamp
Should we implement this check in leve1b or level1e script? I would say level1b.

ludwiglierhammer commented 2 months ago

I would recommend to implement the check in cdm_reader_mapper.operations and call it in level1b/1c.

ludwiglierhammer commented 3 days ago