Sharing images from other apps removes EXIF location on Android > 10

AykutCevik commented 4 years ago

Steps to reproduce

Take a picture with a camera app which writes also GPS data into the picture
Ensure GPS data is written into the picture with different apps like file browser or gallery app
Upload this picture with the Nextcloud Android app version 3.11.1 via the share menu from gallery app
Download picture from Nextcloud instance with a browser and look at the GPS data of the file
See that the GPS data is missing or more likely set to zero

Expected behaviour

Pictures shouldn't lose their GPS data while uploaded through the app

Actual behaviour

Pictures are missing GPS data after upload

Can you reproduce this problem on https://try.nextcloud.com?

Upload through the browser to Nextcloud works fine, just app removes the informations

Environment data

Android version: 10 (OxygenOS 10.3.3) Device model: OnePlus 6T Stock or customized system: Stock Nextcloud app version: 3.11.1 Nextcloud server version: 18.0.5 Reverse proxy: Nginx

Logs

Web server error log

Nothing worth to mention

Nextcloud log (data/nextcloud.log)

Only related to "Maps" app while trying to analyze the image since the GPS data is wrong.

Nothing worth to mention

I faced this issue since I saw that the pictures weren't showing up in the Maps app of Nextcloud. So this is not a new issue. Also I can surely say that the problem started before 20th of April, so also with a prior version. I can share pictures in a PM.

Appreciate any ideas for solving this issue. Thank you for your great work!

strugee commented 4 years ago

3\. Upload this picture with the Nextcloud Android app version 3.11.1

How are you uploading the picture? Auto Upload? The "Upload files" button? The "Upload content from other apps" button?

Stock or customized system: FALSE

Does this mean you have a stock system? (I.e. the version of Android that came with your phone)

Reverse proxy: FALSE

I believe this question is asking what web server you access Nextcloud through.

AykutCevik commented 4 years ago

@strugee updated the issue.

Trying several other uploads, I found out that the issue may not be related to the Nextcloud app itself.

I've uploaded pictures through the share menu from both apps:

OnePlus stock gallery app
Simple gallery app (com.simplemobiletools.gallery.pro)

Both uploads resulted in setting the GPS data to zero.

Upload through a file browser like "Amaze" or through the Nextcloud app itself didn't change the GPS data.

Tricky one! I will contact the app maintainers and reference to this issue here.

AykutCevik commented 4 years ago

I have to reopen it again since I'm not quite sure how to debug this issue without brainstorming/information from here.

The "simple gallery" app provider said that they don't change the image beforehand. What could be the problem?

InfamousUser commented 4 years ago

I think it is usual for sharing application to remove metadata, it used to be a huge privacy leak sending images over IMs that turned out to contain user's exact coordinates. Not sure if cloud backup tools should do it, they should not IMO, but it could be normal, somebody will know more about this.

AykutCevik commented 4 years ago

Also to reference a comment which describes exactly the same thing: https://github.com/nextcloud/maps/issues/361#issuecomment-614571815

strugee commented 4 years ago

Upload through a file browser like "Amaze" or through the Nextcloud app itself didn't change the GPS data.

How exactly did you upload in all of these apps? Did you use the "Share" button in Amaze? In Simple Gallery? In the stock gallery? In general the more information and specifics contained in a bug report, the better.

AykutCevik commented 4 years ago

@strugee as I said in point 3: via the share menu from gallery app This was the approach for all of the apps. The gallery app is stock, yes.

tobiasKaminsky commented 4 years ago

NC is not changing any metadata. You should easily find the causing app via uploading with different apps. If you can really reproduce it via NC please re-open.

AykutCevik commented 2 years ago

Sorry, but I have to reopen this. It is still an issue, if I share a picture through the gallery or files app, nextcloud removes/nulls the GPS data all the time: grafik

I tested this on a OnePlus 6T with Android 11 as well as on Samsung S21 Ultra with Android 12. On Samsung, I was even able to tick a checkbox prior sharing to keep the GPS data. Nextcloud itself removes them unfortunately.

AykutCevik commented 2 years ago

@tobiasKaminsky can we reopen this issue please? It prevents me from uploading images through gallery apps which is way more convenient than doing it through the Nextcloud app.

fm-sys commented 2 years ago

Stripping EXIF data is part of the android Q privacy feature set.

Starting with Android 10 (or 11?), apps need to explicitly specify that they want to receive Metadata.

So it's indeed a bug in the nextcloud client, and not the fault of speciffic gallery apps.

You can read more about it here:

AykutCevik commented 2 years ago

@AlvaroBrey any chance we get feedback on this issue since @fm-sys thankfully added more information?

ghost commented 2 years ago

Hopefully a resolution soon on this

2lian commented 2 years ago

I found a solution, not sure it works everywhere, but it fixed every upload done with the Nexcloud app (notably instant upload). (I have a OnePlus7 with Android 11)

Go to your phone Settings -> Privacy -> Permission Manager -> Files and Media. Find Nexcloud in the list and make sure to "Allow management of all file". This works for me

Screenshot_20220807-110434res

Previously this setting was on "Allow access to media only".

ghost commented 2 years ago

I get gps data on my Samsung S21 + with auto uploads on. It started to work. I have no idea what changed.

2lian commented 2 years ago

Tested some more. Funny thing is only instant upload and "Upload files" from the "all app" menu is working. The share button in Gallery app (this app has the same auth as Nextcloud) does not work, nor the "Upload files" for the "shared" menu in the Nextcloud app.

AykutCevik commented 2 years ago

Uploading within the Nextcloud app already worked before in my case, it is cropping GPS data when you use the share functionality of other apps like gallery and files apps. They are way better in showing, organizing and selecting files and images than the Nextcloud app which makes it hard to use the in-app upload of NC regularly.

Poldi1977 commented 2 years ago

The problem also occurs under Android 13 on a Pixel 4.

AykutCevik commented 2 years ago

Clearly, this is a case of data loss! If I wouldn't have found out and would just have all my photos in the nextcloud instance I would never be able to get back my GPS data in the photos. How is this issue not getting more attention? How many users may be already affected by this?

InfamousUser commented 2 years ago

Nobody seems to care about these issues. And the amount of them is vast.

strugee commented 2 years ago

I don't speak for the development team but as a Nextcloud user I would encourage anyone who cares about seeing this get solved to open a PR. There seems to be a path forward described in https://github.com/nextcloud/android/issues/6248#issuecomment-1147167524 which should help you.

stephan-l commented 1 year ago

Is there any hope to get this fixed? Without valid metadata auto upload is completly useless.

The solution in privacy dashboard is not usable as there are only two options on Pixel 7.

tobiasKaminsky commented 1 year ago

Can you test this: https://github.com/nextcloud/android/pull/11161#issuecomment-1345998605 ?

AykutCevik commented 1 year ago

Works for me @tobiasKaminsky

stephan-l commented 1 year ago

Seems to work for me too @tobiasKaminsky

dbmaxpayne commented 1 year ago

I just stubled upon this too. Very frustrating as I had already deleted the originals on the phone when I was wondering where the GPS info is -.- Either way, this works on Google Pixel 7 Pro https://github.com/nextcloud/android/issues/6248#issuecomment-1207315271

AykutCevik commented 1 year ago

@dbmaxpayne definitely! This data loss has to be addressed in a unit or integration test. I'm really wondering how at NC or nobody else faced this issue so far - maybe they don't know? People should check the maps feature in NC after upload to see if their photos are placed on the map or not.

dbmaxpayne commented 1 year ago

I mean I really didn't notice until now and look at how old this thread already is. I only noticed, because my workflow has changed due to a new phone. Back then I always copied the files off the phone over the InstantUpload ones to get rid of them on the phone and to circumvent any upload issues I sometimes had with my old Samsung S7. I always wondered why Nextcloud on Windows would upload the files again without noticing they are actually the same. Now I know. They were never the same -.-

As I work quite a lot with the GPS data for example, I'd also vote for some sort of check in the Android app's code if it is allowed to upload the original file and display a warning if not. However, I am not deep enough into Android programming as to suggest a fix myself.

Either way it's a weird behavior by Android to remove data without any notification at all. Nextcloud seems to just be a victim here.

whitemanthedj commented 1 year ago

+1 on everything said. Would love to have gps data automatically uploaded with user permission. Media location not working from my auto upload setup from my Samsung S10. Have noticed for a while that my photos didn't show up on the map, but just getting around to pursuing a fix and the magical world of open source.

rikrdo89 commented 1 year ago

have anyone found a fix to this issue for the Sealife client on android? It seems like no one is responding to the issues opened by users. (https://github.com/haiwen/seadroid/issues/946)

vitis586 commented 1 year ago

This is indeed a problem but it is not entirely problem with Nextcloud. I am also missing all location data on my auto uploaded files but when I tried sharing photos from other apps (ie. uploaded a photo as a file to Telegram and downloaded to my PC), I was missing the GPS data too. Not missing, but zeroed out as @AykutCevik shows. When I uploaded the same photo from Google Photos gallery, I have the GPS data present and accurate. The steps below helped me to keep the GPS data in photos.

Is there a way for Nextcloud to ask for this specific permission? As you can see the depth of the setting is quite substantial and it took me quite some time to find it.

This is a bit different on my Pixel 7 Pro - https://github.com/nextcloud/android/issues/6248#issuecomment-1207315271

Settings - Security and Privacy - Privacy - Permissions manager - Files - (small print on the bottom) See more apps that can access all files - Nextcloud - Allow access to manage all files

obrazek

brilthor commented 1 year ago

I would suggest that the nextcloud client should be notifying users if it's not getting GPS data as part of autoupload and how to work around it until there is a transparent solution.

I like many others found out about this data loss issue when looking at backups and trying to figure out why they were corrupted which is a terrible place to put a user in

Cebrain commented 1 year ago

Have to say this is a huge bug for me, that the location data is removed while upload. It started for me in January 2023 that the data went missing, dont know exactly what changed.

I am really happy i still have the files on my phone so no location data was lost and i moved the files manually.

nomandera commented 1 year ago

Please consider family deployments, (non skilled users), when resolving.

Currently if the client permissions are not correct, images may be uploaded with missing metadata. I cannot trust my users to understand anything to do with permissions and I cannot rely on initial setup to persist as Android routinely suggest removes permissions from applications under its own control.

Currently in these scenarios my family photos result in silent data loss that I cannot mitigate. It would be better if image auto-uploads did not work at all unless the full necessary permissions were granted. Please fail closed in all cases, even edge cases, where any data loss can occur.

In the interim please urgently add a client warning. I ended up here after hours of head scratching debugging why on earth some of our holidays photos are missing Lat/Long and it is certain many others will unknowingly be in the same boat.

reox commented 1 year ago

It would be better if image auto-uploads did not work at all unless the full necessary permissions were granted.

I guess there are cases where you want auto upload but strip sensitive metadata (okay maybe not that often tbh, but you never know). I would vote for a solution that neither silently ignores the metadata nor never uploads anything without granting that permission. In my opinion, the best thing to do is to inform the user during initial setup and ask what to do.

In the interim please urgently add a client warning.

I sign that. I also spend half a day scratching my head what the heck was going on there...

reox commented 1 year ago

Is it possible that Android resets this permission after a while? I noticed that it suddenly stopped working and the permission to read all files was removed again.

vitis586 commented 1 year ago

Mine permission are still the same since I set it up. Did you use Nextcloud app? Android removes permissions after some time of inactivity.

reox commented 1 year ago

I used it for the auto-upload - does that not count as usage? but that might be the case.. I disabled the auto-removal for nextcloud now as well.

nomandera commented 1 year ago

Some info that I am not sure has been posted so far. I was hit with this bug but my initial symptoms were lack of auto-upload deleting the redundant copy on android on successful upload.

This eventually led me here to the real after I noticed some GEO tags missing from my families photos.

So purely anecdotally it appears that in my case at least the failure to upload a proper copy of the image also manifests as failure to delete.

Unfortunately I manually deleted some photos from my phone essentially losing the geographic information forever when I was sill under the mistaken impression this was a much simpler problem.

The client MUST alerts for missing permissions. The current situation of silent data loss is horrendous.

vb0 commented 1 year ago

Can the client in the first place detect that it's getting sanitized/censored files from the OS? How could it tell that it didn't receive a "proper copy" if it didn't have access to the original? Sure, there might be something that can be done "out of band" in the API but it's highly unlikely as it's some logic to be explicitly programmed and it doesn't seem to be activity on this since 2020.

Other than that I'd go one step further about alerting: the information should be present in any file transfer notification. Frankly the OS should be doing it in the first place when it starts to zero out stuff in your files, I don't understand why they don't do it as it seems to be a feature many people are very vocal about it, if Android caters to these even better, just pop up a notification "I just sanitized for you 1, 5, 200 files that had GPS location you didn't want to preserve/share/transfer/etc.". Let them feel better that Android is protecting them from themselves while letting also users that know better something is happening. But this is what it is, next best thing would be for NextCloud (and anything similar) to alert the user about this.

AykutCevik commented 1 year ago

Can the client in the first place detect that it's getting sanitized/censored files from the OS?

It is not about detecting it, it is about requesting the sufficient rights from the apps perspective and warn a user if rights are missing to access the file properly with all the data.

vb0 commented 1 year ago

It is not about detecting it, it is about requesting the sufficient rights from the apps perspective and warn a user if rights are missing to access the file properly with all the data.

That is just one way of detecting (or at least assuming) you aren't getting the complete file, but as I said it's some extra logic that for sure it isn't yet programmed (as NextCloud doesn't even know to request the right permissions yet AFAIK, never mind keeping track they weren't granted). My question was triggered by the other previous comment mentioning files that weren't uploaded properly not being deleted too, was this by chance or NextCloud somehow detected it isn't getting the originals?

strugee commented 1 year ago

My question was triggered by the other previous comment mentioning files that weren't uploaded properly not being deleted too, was this by chance or NextCloud somehow detected it isn't getting the originals?

It was probably by chance. That behavior is likely triggered by an error being thrown, interrupting the upload process before it has a chance to delete the file on the local device. By contrast, this bug doesn't cause files to fail to upload - rather it causes the wrong file (the sanitized/scrubbed version of the picture instead of the original) to be uploaded.

nomandera commented 1 year ago

Can i just clarify these responses because as I see it my logic HAS to be incorrect or this would be an absolutely horrendous critical bug that has been open for potentially 2+ years.

If it was probably by chance that the final file deletion was blocked by permissions on the device does this not mean that in some scenarios the NextCloud client is uploading files with incomplete metadata and then deleting the copy with complete metadata i.e. losing data.

Edit: I have now determined that as well as images videos have had GPS tags stripped as well.

DAveShillito commented 1 year ago

I think a recent update may have fixed the issue. I just took a picture, and it auto uploaded and the version that arrived on my PC did have the location in the metadata.

troygeiger commented 1 year ago

I think a recent update may have fixed the issue. I just took a picture, and it auto uploaded and the version that arrived on my PC did have the location in the metadata.

I also just found that the location does auto-uploading on my Galaxy phone after discovering there is a location tag option in the camera app that was toggled off. Turned it on and sure enough, the location was in Nextcloud in the next picture I took.

vb0 commented 1 year ago

Still happening on the latest non-beta from Play Store (3.26.0) on Pixel 7 with Android 13 (September update).

When testing it's worth sharing the following:

permissions, especially if you gave the special "All files access" one (it's a different UI for that). We know it works with that, and it's the current workaround.
which software version/branch and where it comes from (Play Store, F-Droid, side-loaded, etc.)
what's the phone manufacturer and which software is running (both as Android version/last update and if it's something like LineageOS).

asm0dey commented 1 year ago

Thanks @vitis586, your fix worked for me! Is there any way to resync all the data now to upload the information to nextcloud?

Cebrain commented 1 year ago

Thanks @vitis586, your fix worked for me! Is there any way to resync all the data now to upload the information to nextcloud?

You could create a new folder on your phone with the problematic media files. Enable Auto Upload for this folder with the Upload Setting "Upload all existing files" and "Overwrite existing files"

You could also do this on you normal camera folder, if you want reupload everything.

vitis586 commented 1 year ago

Thanks @vitis586, your fix worked for me! Is there any way to resync all the data now to upload the information to nextcloud?

You can do what @Cebrain suggested. That is the most straightforward way how to do it.

I my case I had already done some changes to the EXIF before I noticed that it is missing location data. I then used exiftool to extract the position data to csv and then used this to update the already uploaded files.

nextcloud / android