search

thoughtgap / arc-data-server

Data Wrangling with Web API for exported personal location records from Arc App https://bigpaua.com/arcapp
20 stars 3 forks source link

Read json exports: Avoid overlapping timelines #3

Open thoughtgap opened 5 years ago

thoughtgap commented 5 years ago

Right now it's in the user's responsibility to avoid duplicate data. The script reads duplicate data without questioning.

E.g. if a monthly export for 02.2019 and a daily export 2019-02-01.json are read, the data is queried and returned twice.

thoughtgap commented 5 years ago

At first only a hint that there is duplicate data makes sense.

Approach for finding out the overlapping time spans: https://derickbailey.com/2015/09/07/check-for-date-range-overlap-with-javascript-arrays-sorting-and-reducing/

PalminX commented 5 years ago

Even regular non-overlapping json files will contain duplicate timelineItems: timelineItems which cross midnight will be contained (identically) in both the current and the previous day's JSON file.

These are absolutely identical, down to the itemID, so they should be easy to filter out.