Database Import should not overwrite existing data

[Dainem21]

Hello

I had to change my phone. On the new phone I installed xDrip+ and started a new Dexcom G6 Sensor. All running fine. Then I wanted to ADD the old Database and I observed the following issues:

1) The data on the phone was replaced by the old one. So I lost all the new values. So it would be nice to have a real ADD of data.

2) I started the Dexcom with the Code and it was running fine. After the import of the old data, I was forced to calibrate with blood. Although the date and timespan of the sensor was still correct, xDrip+ lost the calibration.

Best regards and thanks for the great effort you all spend!

[definitivequestion]

Hence the warning. I agree it would be nice if the import database code was rewritten as SQL insert statements.

[Navid200]

@AdrianLxM I hope you don't mind me asking you. Why did this happen? In this scenario, what can the user do with xDrip on the old phone to ensure that it will not contain any data for the period it was not being used any longer? Would that avoid overwriting data on the new phone?

[Navid200]

@nickb24

[tolot27]

The current procedure is a complete restore of a backup database. This means, the data gets completely overwritten as it is typical for restore procedures. I suggest copying the data from the old phone to the new phone as soon as possible and prior adding treatments or calibrations at the new phone. The menu item is incorrectly named "Import database". Instead, it should be named "Restore database" to reflect the current behavior. Implementing a real import would be nice but gets complicated if the database is large and import fails somehow. Also conflicts can occur if the new phone got backfill data.

Another solution is to use Nightscout and keep the data their.

[Navid200]

This is the message that comes up.

Do we need to turn this into a bug and request there to be a more clear message explaining that the existing database will be overwritten? Or, is the existing message good enough and we can close this issue?

[tolot27]

I would turn it into an FR and change the title, accordingly.

[Navid200]

I don't have edit privilege on the title.

[SimonPhilpott]

I recently wiped the phone app data and cache in order to fix battery drain on my smartwatch SW3, usually this would involve saving all settings to SD card and exporting my database prior to this so that I can restore both.

However once the app storage was cleared, after restoring settings via the popup message I was prompted with the import database prompt which I accepted. Despite the import data success message, nothing was imported also the settings that were recently restored were wiped. I attempted to import the database again, but again the screen remained blank and no data points were restored - also my settings were once again cleared.

Looking in the log files for all of the recent data points I can see a lot of errors;

"Bg data was not calculated, working on raw data" followed by "Error getting data from getLatestBgForXMinutes. Returning", "libreBlock exists but no matching bg record exists"

This might be impacting the import as the data is not in the right format to display?

Note: I'm not asking Xdrip to overwrite anything, since the database after wiping the app cache & data is empty. This is a pure import and bring in the saved data

[Navid200]

@SimonPhilpott When I have reinstalled xDrip, I have been asked if I wanted to restore settings. I have never been asked if I wanted to restore the database. Are you sure it wasn't a question about restoring settings?

How many days of data did you have in xDrip when you exported the database? What is the name and size of the database file?

[SimonPhilpott]

@Navid200 Okay so I bit the bullet and wiped everything so I can clearly document this with screenshots, so there's no confusion. Same issue once again, with only about a days worth of data held in the exported database pre-wipe. This is a description of each screenshot taken after each step;

1) Pre app wipe. You can see the data from the last day here. 2) Wiping the data from the app 3) Launching Xdrip+ for the first time after wiping and setting the source back to Libre > MiaoMiao 1 (seen here as Bluetooth bridge...) 4) Settings backup detected 5) Settings are restored, this are my Alerts 6) Still no data, so... 7) Import database 8) The file 9) Success, supposedly 10) But no data, it should show right up to the point I wiped the app seen in screenshot 1 11) And it's wiped my settings, so I restored them manually again 12) And here is the prompt I mentioned that asked to restore the exported database it detected. I said no, it just wipes the settings again

[Navid200]

@SimonPhilpott Can you inspect the database file to see what it contains? As a first step, it will tell us if the problem is with export or import. It is an sqlite file. You can use a database browser.

[SimonPhilpott]

@Navid200 Okay so it opens, there are lots of pockets of data is there any particular category that would help? Is it poor behaviour to attach it here?

[Navid200]

If you look at BgReadings, do you see anything? You can also look at sqlite_sequence. It should give you a summary of all the contents. Does it make sense? Does the file contain enough data corresponding to what you exported? Or, is it empty?

You can attach it if you don't consider the contents to be private.

[SimonPhilpott]

BgReadings has 3015 entries, in sqlite there are nearly 12k entries under user error. Not sure if that's meaningful. I've attached my sqlite file since I don't really have a problem with 24h of data for bug testing purposes. export20210421-165626.zip

[Navid200]

I just imported your database. I can see readings starting early morning of April 11, 2021 and ending noon of April 21, 2021, yesterday.

Do you not see those results?

[Navid200]

There are also blood glucose readings starting from September 2020. I suppose those are from a Bluetooth blood glucose meter.

[SimonPhilpott]

I do not see those results no, what version of Xdrip+ are you running? That's pretty interesting what you've seen fro results going back so far, I suppose I didn't think the MiaoMiao held the saved data back that far. To be honest it would be a better place to restore from since it's held so far back and useful should I need to flush the phone app but still have enough data for reporting should I need to show my Endio anything. Is it worth me raising it as a feature?

[Navid200]

I used the April 13, 2021 release. What do you see if you tap on history and go back 2 days?

It looks like the export has worked properly for you. Since I have been able to import successfully. Let's find out why you are having trouble importing. So, please answer my question above.

I don't know what you are considering to propose as a feature. In short, if there is a feature that many can benefit from and there is no work-around for, it may be merged. However, I know many people will benefit if xDrip included a GPS capability. I go hiking and I have to keep switching between xDrip and my GPS app. Submitting a request to add a GPS feature to xDrip, although many will benefit from, is still selfish because xDrip is a CGM; it is not a GPS; it is not a watch; it is not a simulator; it is not a diet guide; it is not ...

Look at the number of open issues, 2 of which currently have been opened by you. Why do you think there are so many open issues with nobody paying attention to them for years?
The more complicated we make xDrip, the harder it becomes to maintain.

[SimonPhilpott]

@Navid200 I can see calibrations, notes and any carb inputs but no data whatsoever.

I can appreciate how pointless submitting new features might seem. They are always open for consideration - you never know what the appetite might be and the discussion does at least challenge or distil an idea. We simply have more users than developers - I'm working on an alternative theme but that plays to my strengths and Java is not one of them. I do agree on simplification, the settings menu has become a mess of options and attempts to fix X for whatever unique circumstance the bug is experienced within. Github is great, but it does not make up for better development processes. Diabetes hardware is a mess of standards until that standardises this is the way things are, no?

[Navid200]

Can I post a snapshot of what I see? It may help you. But, it's your readings. So, I won't if you don't want me to.

[Navid200]

@SimonPhilpott If you can see calibrations, notes and carb inputs, the import has worked for those at least, right? It's not like it cannot import the file at all.

What do you see if you look at history?

[pcastelo]

Hi guys i have the same issue with android 11, i test same version xdrip with older android and work without any problem letme know if i can help or need some screenshot

[Navid200]

@pcastelo If you have attempted to import a database file into xDrip and it has confirmed that it was imported. But, it did not, and your phone is Android 11, would you please post in this tread instead? https://github.com/NightscoutFoundation/xDrip/issues/1705

[Navid200]

We have renamed the menu item from "Import" to "Restore" to avoid this confusion. There is currently no capability to import a database leaving the existing database intact. When you get a new phone, you should transfer the database before you set up your new phone as a collector.

[Navid200]

@jwoglom @tzachi-dar

1- Is it possible to create a new feature to allow importing only the BG readings from a database file in without overwriting the entire database?

2- Is there a way to combine two databases outside xDrip?

[Navid200]

@SimonPhilpott and everyone else,

1705- xDrip has a problem with restoring database for some Android 11 phones. This has been reported in the following issue and a work-around has also been identified: https://github.com/NightscoutFoundation/xDrip/issues/1705

1045- xDrip is currently incapable of importing a database. It can only overwrite the existing database. This issue (https://github.com/NightscoutFoundation/xDrip/issues/1045) is about that.

These are two different issues. Would you please delete your posts and images in this thread that are not about 1045 and instead are about 1705? This will allow each thread to focus on one specific matter.

Thanks

[air-ship]

I hope this helps with an xDrip+ database restore on a new phone...

Phones: Old phone > Sony Xperia L3 / Android 8.1. New Phone > Samsung Galaxy S10e / Android 12 Version: xDrip+ nightly build 8th September 2022 / xDrip-plus-20220908-4aeee42.apk

Here's how I managed to restore a cloud backup. Not quite the way I thought it would work. I suggest you read through the steps and comments to get an overview of the process.

1. On my old phone I made a cloud backup in xDrip+ > There dots menu > Import / Export features > Cloud Backup. I tried it with my primary Google account and couldn't get a backup to complete. No idea why so I can't advise on troubleshooting this. I then tried another Google account and managed to successfully create a backup: xDrip-Backups/xdrip-backup-I3312-Xperia-L3.xdrip

2. On the new phone I tried the standard cloud backup restore but this didn't work because xDrip+ writes an empty file with the name of the new phone (xdrip-backup-SM-G970U1-Rees's-S10e.xdrip). The xdrip-backup-I3312-Xperia-L3.xdrip is never found by xDrip+. No idea why even though the file exists. Read on.

3. So I downloaded the backup file > xdrip-backup-I3312-Xperia-L3.xdrip. Next I There dots menu > Import / Export features > Cloud Backup > SELECT BACKUP LOCATION grey button. I had uploaded xdrip-backup-I3312-Xperia-L3.xdrip to my S10e's Download directory (could be whatever directory you want). Next, in SELECT BACKUP LOCATION I chose USE ALTERNATE FILE. Navigated to my S10e's Download directory and chose the file xdrip-backup-I3312-Xperia-L3.xdrip.

4. Once I completed step 4 xDrip+ then recognised the backup and listed in the info about the backup above the three grey buttons in Cloud Backup. At this stage you know you have your backup selected for restore. Now tap the RESTORE FROM BACKUP grey button. Choose "YES" in the "Backup Source Confirm", "Are you really sure?" > YES. The restore now runs and in my case it was successful. xDrip+ restarts and (hopefully) your data is imported.

5. My final test was to forget my MiaoMiao on my old phone (in xDrip+ > System Status) and my new automatically connected to the MiaoMiao because I had tested this previously.

6. I then went back to the step 1 and tested it again. Success! Notes:

7. When you perform your cloud backup it's worth setting your screen display timeout to 30 minutes (if possible). I found if my phone slept then the backup didn't complete. Could just be my old phone's eccentricity.

8. Be patient when xDrip+ makes the cloud backup. My phone took about 15 minutes to upload 110MB.

9. The cloud backup file is overwritten each time when you backup. Keep this in mind if you are testing everything before a full commit. It might be worth backing up each cloud file either in your Google Drive or locally on your computer.

10. I understand that the xDrip+ versions should match on each phone. I guess this is because additional data may be written to the database file which means that a newer version of xDrip+ is expecting additional data (which may make the database incompatible).

11. Be patient. It's not a quick process. After all, the data in xDrip+ is valuable, especially when it goes back over three years as it does in my case.

12. Unrelated to databases but you will find if switching manufacturers phones that your alert settings for low bg / high bg / calibration etc will lose their sound files. In y case I used the Sony alerts and my Samsung doesn't have these. I had to go Google hunting to find them. This took ages. I'm 54 so I'm a creature of habit and my brain is subconsciously trained to the alert sounds, especially at night.

I hope this helps.