ISSUE-239 Implement signature calculation for each file

[lewismc]

WIP to address #239 @renato2099 we can add some unit tests to cover the new functions. This is not 100% yet. In the future we will need to think about the real use case which demands updates to metadata after the file has ben ingested. We would need to recompute the SHA256.

[lewismc]

@renato2099 this is very close now. I am getting a problem when ingesting the attached file.

tagbase-server-postgis-1         | 2023-05-16 14:15:46.068 UTC [337] tagbase@tagbase ERROR:  insert or update on table "data_histogram_bin_data" violates foreign key constraint "histogrambindata_histogrambininfo_fkey"
tagbase-server-postgis-1         | 2023-05-16 14:15:46.068 UTC [337] tagbase@tagbase DETAIL:  Key (bin_id, bin_class)=(1, 1) is not present in table "data_histogram_bin_info".
tagbase-server-postgis-1         | 2023-05-16 14:15:46.068 UTC [337] tagbase@tagbase CONTEXT:  SQL statement "WITH moved_rows AS
tagbase-server-postgis-1         |         ( DELETE
tagbase-server-postgis-1         |          FROM proc_observations a USING observation_types b,
tagbase-server-postgis-1         |                                         submission c,
tagbase-server-postgis-1         |                                         data_time_series d
tagbase-server-postgis-1         |          WHERE a.variable_id = b.variable_id
tagbase-server-postgis-1         |            AND b.variable_name LIKE 'TimeAt%'
tagbase-server-postgis-1         |            AND a.submission_id = c.submission_id RETURNING a.submission_id,
tagbase-server-postgis-1         |                                                             c.tag_id,
tagbase-server-postgis-1         |                                                             a.submission_id AS bin_id,
tagbase-server-postgis-1         |                                                             cast(substring(variable_name, '(\d+)') AS int) AS bin_class,
tagbase-server-postgis-1         |                                                             a.date_time,
tagbase-server-postgis-1         |                                                             a.variable_value,
tagbase-server-postgis-1         |                                                             d.position_date_time,
tagbase-server-postgis-1         |                                                             a.variable_id)
tagbase-server-postgis-1         |       INSERT INTO data_histogram_bin_data
tagbase-server-postgis-1         |       SELECT *
tagbase-server-postgis-1         |       FROM moved_rows"
tagbase-server-postgis-1         |  PL/pgSQL function execute_data_migration() line 121 at SQL statement
tagbase-server-postgis-1         | 2023-05-16 14:15:46.068 UTC [337] tagbase@tagbase STATEMENT:  COPY "proc_observations" FROM stdin WITH DELIMITER AS ',' NULL AS '\N'

I will take a look later on...

[lewismc]

iccat_gbyp0008_ArgosTrans_eTUFF.txt

Something to do with sequencing again possibly...?

[renato2099]

Something to do with sequencing again possibly...?

yeah I was going to say this ☝️ because this PR does not change current ingestion logic, it only changes the sha256 signature , I will take a look a look as well