The 64-bit sha256 checksum calculated for each file and stored in submission.hash_sha256 satisfies initial duplication detection. Post #238 a new use cases arose whereby updates to the reference track need to be accommodated e.g. user ingests initial file representing the reference track and subsequently ingests another file with literally ONLY the flag_as_reference metadata value changed.
This would require updates to only for the metadataflag_as_reference value and data_position.flag_as_reference for an existing submission. This would prevent us from (re)ingesting all of the data and track again.
Essentially we need to extend the concept of submission.hash_sha256 to something like
submission.file_sha256: a SHA256 calculation of a raw binary file
submission.metadata_sha256: a SHA256 calculation of the metadata portion of a raw binary file
submission.data_sha256: a SHA256 calculation of the data portion of a raw binary file
submission.track_sha256: a SHA256 calculation of the track position data portion of a raw binary file
There are some nuanced complexities associated with this proposal e.g., we need to determine if calculating a SHA256 hash directly from a file would yield the same hash as when one is calculated for a PostgreSQL response supposedly representing the same data.
I would add a second use case that is an extension of the one described above.
A eTUFF file, in its core design, is meant to be a self-contained, self-described file to house both metadata and data, the latter of which include measurement (sensor) data and positional data (track).
If a eTUFF is used as intended, a user may want to include a new, best available track as the reference track, and include it as part of an updated and complete eTUFF file.
So the user proceeds to generate a new eTUFF (ver 2), and swapped out the "old" track in the original eTUFF file (ver 1).
In essence: "metadata" have not changed (ver 1 and 2 both have referencetrack_included = 1) and "measurement data" have not changed. Only "positional data" have changed.
The 64-bit sha256 checksum calculated for each file and stored in
submission.hash_sha256
satisfies initial duplication detection. Post #238 a new use cases arose whereby updates to the reference track need to be accommodated e.g. user ingests initial file representing the reference track and subsequently ingests another file with literally ONLY theflag_as_reference
metadata value changed. This would require updates to only for themetadata
flag_as_reference
value anddata_position.flag_as_reference
for an existing submission. This would prevent us from (re)ingesting all of the data and track again.Essentially we need to extend the concept of
submission.hash_sha256
to something likesubmission.file_sha256
: a SHA256 calculation of a raw binary filesubmission.metadata_sha256
: a SHA256 calculation of the metadata portion of a raw binary filesubmission.data_sha256
: a SHA256 calculation of the data portion of a raw binary filesubmission.track_sha256
: a SHA256 calculation of the track position data portion of a raw binary fileThere are some nuanced complexities associated with this proposal e.g., we need to determine if calculating a SHA256 hash directly from a file would yield the same hash as when one is calculated for a PostgreSQL response supposedly representing the same data.