search

OpenArchive / Save-app-ios

Secure Mobile Media Preservation
https://open-archive.org/
GNU General Public License v3.0
11 stars 3 forks source link

NextCloud (& possibly others) naming folders with wrong metadata - using upload date but should be capture date - does not match capture metadata #182

Closed foundscapes closed 2 years ago

foundscapes commented 3 years ago

Phone: Google pixel 2 OS: Android 10 Server: NextCloud Instance for WITNESS

Two issues:

  1. Folders should be titled by date of capture, not by date of upload, but they are currently showing the latter.
  2. Hashes not matching original file hash

Notes from org: the video that she uploaded to NextCloud seems to be not the same as the original video, and the neither the nextcloud file nor the original file have the hash (I tried MD5 and all the SHA's) that is reported in the JSON file. Please see the following:

Yvonne1

metadata from airdropped file has original iphone metadata, whereas NextCloud-downloaded file does not:

Yvonne2

not sure what kind of hash is used in the JSON, but does not correspond to any of the SHA's or MD5 of either the nextcloud or the airdropped file:

Yvonne3

This is a similar issue on Android, so it may be useful for us to coordinate efforts. See Issue #226

tladesignz commented 3 years ago
foundscapes commented 3 years ago

Screen Shot 2020-12-22 at 1 59 08 PM @elric-wamugu this is another screenshot of the discrepancy of time/date info

foundscapes commented 3 years ago 
* The checksum aka. hash is SHA256. Please check, if the checksum really doesn't match.

* The 2 files cannot be identical. They're in different formats. Obviously the device original uses Apple's QuickTime Movie as container format, whereas the exported one uses the MPEG-4 container format. This is due to the fact that app's can't just read the original video file and send that, but, on iOS, have access only through a transcoding pipeline for export, as original video files are typically too large to send them over the net. So that is completely on purpose. Since we're transcoding anyway, we use a more widely-supported format than the original.

* Regarding the folder naming: The upload collection folder name is created, when the first file is added to a project's new upload batch ("collection" in app's concept parlance). So it's not exactly upload date. Anyhow, please don't mix different issues in one - create another issue, if you want to have something done there. However, please note that there needs to go some more thinking into what timestamp to use here exactly. Capture time of what? Of the first file of an upload batch? The median of all files? The last one?

Thank you @tladesignz the issue here is that we need the creation date/time of the file to be in the name of the uploaded file and correspond to the original file - can we do this?

apologies if this is a separate issue, will work with @elric-wamugu to organize better.

tladesignz commented 3 years ago

Thank you @tladesignz the issue here is that we need the creation date/time of the file to be in the name of the uploaded file and correspond to the original file - can we do this?

Ok, so now we're talking about a single media file, not about the collection folder name, is that correct?

  1. We can use a different naming scheme on files, yes. Currently, on iOS we're using what the OS is giving us, if it's giving us any. If not, we're using a randomly generated UUID plus an extension which works for the file format.

  2. There are different timestamps available. The filesystem has creation (always) and modification timestamps (mostly, but not always), and different file formats may contain other timestamps inside as contained metadata. We can definitely use the filesystem timestamps to create names, that's ok. About the metadata contained inside the file - that's going to get difficult, as not every file format has that and if so, they might not have the same meaning, and it will be difficult to write code to read that for a lot of different file types. Even so, there's the question if the file is really an original or something transcoded on import into the app, and that metadata might or might not differ from the original.

  3. At last, there's the question of how to format that timestamp in the file name: Should we use a (technical) internationally unified format, should we hardcode a typical format of one region (like the US way of displaying a date and time) or should we localize to what the device is set to?

So, what should it be?

foundscapes commented 3 years ago

@ozgegn and @elric-wamugu can you help with these answers? We can discuss on slack

foundscapes commented 3 years ago

@elric-wamugu can you test this please?

elric-wamugu commented 3 years ago

Refer to https://github.com/OpenArchive/Save-app-android/issues/229

johnhess commented 2 years ago

Closing this in favor of a more closely scoped issue specifically covering filenames as timestamps.

https://github.com/OpenArchive/Save-app-ios/issues/195