search

Greenstand / Greenstand-Overview

Tree Tracking Fighting Poverty and Climate Change - This repository contains Contributing, Project Overview, Roadmap, etc
https://www.greenstand.org
GNU Affero General Public License v3.0
44 stars 11 forks source link

Remove Duplicates of Data Points #37

Closed Davidezrajay closed 5 years ago

Davidezrajay commented 6 years ago

There are several points in the DB that have the same timestamp, same image, same user etc.

dezidaisy commented 6 years ago

My Solution: As stopping the http requests from retrying may result in a loss of functionality as the connections are usually slow and unreliable, it’s best to delete entries from the database or on insert trigger from the database side delete a entry if a duplicate entry exists. You could also set a job to periodically delete duplicate entries from the database. https://www.pgadmin.org/docs/pgadmin4/dev/pgagent_jobs.html The above link is to create jobs in Postgres.

Davidezrajay commented 6 years ago

@dezidaisy Thanks for the research. Can you set this up?

dezidaisy commented 6 years ago

Yes, I could try.

dezidaisy commented 6 years ago

Ok I’ll create the job

Davidezrajay commented 5 years ago

The App is sending a duplicate every time the upload process is interrupted. Here are four tree data points...

Also see Android issue: https://github.com/Greenstand/treetracker-android/issues/127

62040 2019-01-17 03:11:34 2019-01-17 03:11:34 62025 2019-01-17 03:11:34 2019-01-17 03:11:34 62023 2019-01-17 03:11:34 2019-01-17 03:11:34 62021 2019-01-17 03:11:34 2019-01-17 03:11:34 62035 2019-01-17 03:11:34 2019-01-17 03:11:34 62028 2019-01-17 03:11:34 2019-01-17 03:11:34 62032 2019-01-17 03:11:34 2019-01-17 03:11:34 62027 2019-01-17 03:11:34 2019-01-17 03:11:34 62036 2019-01-17 03:11:34 2019-01-17 03:11:34 62039 2019-01-17 03:11:34 2019-01-17 03:11:34 62031 2019-01-17 03:11:57 2019-01-17 03:11:57 62050 2019-01-17 03:11:57 2019-01-17 03:11:57 62049 2019-01-17 03:11:57 2019-01-17 03:11:57 62047 2019-01-17 03:11:57 2019-01-17 03:11:57 62045 2019-01-17 03:11:57 2019-01-17 03:11:57 62042 2019-01-17 03:11:57 2019-01-17 03:11:57 62038 2019-01-17 03:11:57 2019-01-17 03:11:57 62037 2019-01-17 03:11:57 2019-01-17 03:11:57 62033 2019-01-17 03:11:57 2019-01-17 03:11:57 62034 2019-01-17 03:11:57 2019-01-17 03:11:57 62043 2019-01-17 03:12:12 2019-01-17 03:12:12 62057 2019-01-17 03:12:12 2019-01-17 03:12:12 62054 2019-01-17 03:12:12 2019-01-17 03:12:12 62044 2019-01-17 03:12:12 2019-01-17 03:12:12 62052 2019-01-17 03:12:12 2019-01-17 03:12:12 62041 2019-01-17 03:12:12 2019-01-17 03:12:12 62059 2019-01-17 03:12:12 2019-01-17 03:12:12 62060 2019-01-17 03:12:12 2019-01-17 03:12:12 62048 2019-01-17 03:12:12 2019-01-17 03:12:12 62046 2019-01-17 03:12:12 2019-01-17 03:12:12 62061 2019-01-17 03:12:26 2019-01-17 03:12:26 62064 2019-01-17 03:12:26 2019-01-17 03:12:26 62062 2019-01-17 03:12:26 2019-01-17 03:12:26 62056 2019-01-17 03:12:26 2019-01-17 03:12:26 62063 2019-01-17 03:12:26 2019-01-17 03:12:26 62065 2019-01-17 03:12:26 2019-01-17 03:12:26 62058 2019-01-17 03:12:26 2019-01-17 03:12:26 62055 2019-01-17 03:12:26 2019-01-17 03:12:26 62053 2019-01-17 03:12:26 2019-01-17 03:12:26 62051 2019-01-17 03:12:26 2019-01-17 03:12:26

Davidezrajay commented 5 years ago

Remove the duplicate images that the app sends. To be completed AFTER we complete #60

At @petersenpdx has written a python script to identify these points.

Davidezrajay commented 5 years ago

Thanks @sebastiangaertner for removing over 19183 more duplicates... over 23 thousand duplicates now removed from the map...