search

JCKalt / General-Work

Modeling database
0 stars 0 forks source link

Research Dynamo Sensor Data Loss #179

Open JCKalt opened 6 months ago

JCKalt commented 6 months ago

Data was clobbered. I submitted a request to restore the Feb 1 2024 sensor table backup.

This output from dnsen python src/dnsen.py $(python src/pghub.py 0xF100943CC6E3566C --sensors -q) --day-counts -q > dnsen-day-counts.ou Illustrates the problem. Numbers do not start matching the pgsen version of this command until 2024-01-08.

2023-12-20 727 2023-12-21 679 2023-12-22 702 2023-12-23 734 2023-12-24 735 2023-12-25 724 2023-12-26 721 2023-12-27 710 2023-12-28 674 2023-12-29 697 2023-12-30 691 2023-12-31 792 2024-01-01 627 2024-01-02 666 2024-01-03 717 2024-01-04 712 2024-01-05 671 2024-01-06 719 2024-01-07 6558 2024-01-08 17709 2024-01-09 16871 2024-01-10 17216 2024-01-11 16734 2024-01-12 17221 2024-01-13 15815 2024-01-14 15619 2024-01-15 15791 2024-01-16 16897 2024-01-17 16905 2024-01-18 17372 2024-01-19 16809 2024-01-20 17682 2024-01-21 17253 2024-01-22 16753 2024-01-23 17300 2024-01-24 16984

JCKalt commented 6 months ago

At least some of the missing data is in the Raw data from Steve Hiner. I created a script prlx:src/combine-steve-raw-csv-files.py that combines Steve's set of csv files into a single csv file.

I can see that some of that the earliest rows of data are not in dynamo but they are in postgres. Something deleted them after they were transfered into postgres.

JCKalt commented 6 months ago

Still dont' know the cause however, I reimported Steve's Raw and that has restored the data I observed as being missing.

I restored previous files but not sure what to do with them.