issues
search
disk91
/
watchium-issue
3
stars
0
forks
source link
Watchium Service not refreshing data
#28
Closed
disk91
closed
1 year ago
disk91
commented
1 year ago
Watchium service is not refreshing data
What happen
On January 12th one of the RAID0 HDD running the postgresql database died
Impact
Service is working with backup ETL provided by third party during restoration time
Primary ETL service can't sync anymore, accessible data are frozen until restore
Estimated duration 1 week at least
Workaround
using another Helium API during the repairing time
Current situation
2023-01-12 12:00 : investigation in progress
2023-01-12 13:00 : trying to recover the missing SSD
2023-01-12 14:00 : removing dead SSD
2023-01-12 14:30 : rebuilding LVM on 3 working SSD
2023-01-12 15:00 : downloading DB snap from dewi - 2023-01-08
2023-01-13 01:00 : switching on backup API thankfully provided by a third party (TY)
2023-01-13 09:30 : impossible to unzip the dewi database snapshot (corrupted)
2023-01-14 19:43 : new dump generated 2023-01-13, download in progress (2TB)
2023-01-15 16:30 : uncompressing the database dump
2023-01-16 00:20 : restoring ETL database
2023-01-17 11:30 : start to resync blockchain with fresh DB - 8200 blocks late
2023-01-20 13:00 : synchronization in progress - 3800 blocks late
2023-01-22 11:00 : synchronization done - primary api restarted
2023-02-22 22:00 : additional storage addition to retrieve initial storage
2023-02-23 17:00 : resynchronization done - end of the operations
Next steps
[x] restoring LVM configuration with a fresh logical volume
[x] restoring DB and blockchain data from foundation snapshot
[x] resync the DB to the top block
[x] replacing dead ssd
[x] restoring initial storage capacity
Watchium service is not refreshing data
What happen
Impact
Workaround
Current situation
Next steps