Data disappeared in iOS devices | Photos do not synchronize at all in both Android/iOS

[hideoshiraishi]

After solving the following issue, I am now facing two critical issues regarding data synchronization. https://github.com/opengisch/qfieldcloud/issues/567

Currently I do not have enough information, however, some of the surveyors, who are all using iOS devices, claimed that their data suddenly disappeared after pushing changes.

Organization: Bayan_Project Project name: TerracedPaddyField QField version 2.7.0 (iOS) and 2.7.2 (Android)

I downloaded data uploaded from each device in QFieldSynch plugin and realized that some parts of survey data are not synced properly (means that collected data were not downloaded from cloud). Then I asked them to confirm whether they pushed changes. They had collected data in their devices, however, after pushing it, they said that it disappeared from their screen. I am sure they implemented the survey and they had data in their device.

I am not sure what is going on and will confirm the device tomorrow, but it is critical issue. Our project will be failed if the collected data is gone.

This issue seems similar to this: https://github.com/opengisch/qfieldcloud/issues/570

Most of disappeared data are point data. I am not sure whether the disappeared data is listed in "changes" in QFieldCloud or not, but if I want to check it, I should confirm coordinates in "geometry" in Raw JSON as that is only the way to confirm location of collected data, shouldn't I?

Another critical issue is that photos taken in mobile devices are not uploaded at all. At the first day of the survey, around 60 pictures could be synced. After that, it reached 1GB storage capacity, then I added storage to synch photos and all other data 1 week after the survey, however photos do not synch at all. The following image shows the setting in QGIS.

I also tried to transfer data from iOS devices via iTunes based on the following info, but there is no option to do that.

https://docs.qfield.org/get-started/storage/

I hope someone provice solutions or advices to avoid the worst case scenario - the data vanished. I have paid for the service and did not expect this situation.

I will update the post if I getdetailed information. I posted it before obtaining clear formation as this issue affect our project timeline. I need any supportive information.

[hideoshiraishi]

I confirmed the issue a bit detail. There are several changes "applied" to cloud data but are not downloaded when we retrieve it from "QFieldSync" plugin.

Here is an example "01e94b73":

The coordinates are "Point (437231.23705789429368451 9085655.39667976088821888)"

Location of this point is depicted in red circle here, but when we donload the data, it does not appear.

Also, photos "DCIM/facility_20230312171153214.jpg" and "photo_2": "DCIM/facility_20230312171201624.jpg" are not downloaded.

This trouble has been confirmed not only one case, but also perhaps around 50 cases. Also, some parts of features are synched, but most of photos are not downloaded (perhaps a few hundreds of photos). We are now manually plotting the data again from raw JSON file one by one, but it takes time and photos cannot recovered. We prefer more systematic way to recover it all.

[wanderingnature]

I have experienced similar issues with photos that are not uploaded. After doing field work I sync the mobile device (iOS) and can find the point data (features and related observations) but not the associated photos. The link to the photos is correct (name and folder) but no photo is transferred. The photos exist on the device still and I can browse the data there and see the associated photos. I was only able to upload all the photos after multiple sync actions from the plugin. Finally, it took me editing the observation and feature layer again and adding a random photo from the device camera. Somehow this managed to cause all the photos to be pushed. At that time the device showed only two changes, yet then the final 20+ images were pushed as well. There are no messages in the log on the device at any time related to the sync process. Several days ago I also 'lost' 9 new features and those related photos on the device and in QGIS - I couldn't tell if it was due to sync issues or why. I checked all online versions of the GPKG online and locally and none had those features.

[hideoshiraishi]

@wanderingnature Thank you for your information, it seems similar to my problem. In my case, there were two other issues we faced during/after the field survey as we lost most photos in iOS devices.

1. Sync eraces data in mobile devices Some surveyors experienced that their locally collected data dissapeared after synchronization. I suppose it happened when they synched the data before pushing their changes, which might mean that their local data was overwritten by the cloud project which have not applied their local changes. Therefore, I adviced them not to synch but push their changes.

2. Local data dissapeared It is related to the issue 1. We gave up pusheing photos and left mobile devices for a few weeks after the survey. We discussed how to salvage photos and decided to take screenshots from QField one by one, change file names based on the information kept in QField manually and transfer the files into our master DCIM folder in QGIS. However, after launching QField, although I have not seen it actually, all (two) surveyors who were using iOS device reported that their local data dissapeared perhaps due to automatic sync after launching QField. It might completely erace our local data and we gave up salvaging photos.

In this way we lost around a hundreds of photos unfortunatelly. We decided not using iOS device anymore but using android deviced to take data via locally connected cables. I cannot determine the cause but perhaps it (photo unsynch) might happen in the environment with poor internet connection as our survey site was in mountain with very slow internet. I did not set the limitation of image resolution. It might be better to set it to reduce data volume.

[wanderingnature]

@hideoshiraishi

After some struggle with this issue I have the following observations:

On iOS it seems that only PUSH action from cloud app will actually upload photos. I am unsuccessful using synchronize to get photos uploaded on iOS. I have tried this several times in last few days as I made updates and the only time my photos are uploaded is when I PUSH them. If I synchronize I see the changes in the GPKG layers which reference the photo files but the photo files are not uploaded. After a PUSH I check the project Changes page on web and see if the updates are successful. Then I check the files to see if they are there. I've also started using the SDK/CLI to manually download and archive the photos in case of any other issues. It has taken more then one PUSH to get all files uploaded. It's a bit of a pain when really this should all just work... Several times I have had to make a dummy change to a feature and attach a bogus photo and then the PUSH button is active again so you can use it.

[hideoshiraishi]

@wanderingnature Thank you for your information! Making a dummy change for pushing unuploaded photos is a good quick solution. I am not sure whether it works for me as surveyors says that their data has gone after the synchronization. Anyway I will try it.

By the way, I have never used SDK/CLI but can I take photos from iOS devices through it?