SUSE-Enceladus / public-cloud-info-service

Find IPs of service endpoints, and information about published images.
Apache License 2.0
5 stars 8 forks source link

Add validation for date fields #104

Closed KeithMnemonic closed 2 years ago

KeithMnemonic commented 2 years ago

It would be very helpful to have DB checks/constraints to ensure the following dates fields are always in chronological order

publishedon deprecatedon deletedon

I.e deleted must be greater than or the same as deprecated, and deprecated must be greater or the same as published.
This also implies deletedon is greater or the same as published.

rtamalin commented 2 years ago

So this will likely require cleanup for existing entries that don't conform to those rules, e.g.

postgres=# select name, state, publishedon, deprecatedon, deletedon from amazonimages where deprecatedon is not Null and deprecatedon < publishedon;
             name              |  state  | publishedon | deprecatedon | deletedon  
-------------------------------+---------+-------------+--------------+------------
 suse-sles-11-sp1-v1.02.x86_64 | deleted | 2012-07-20  | 2012-03-06   | 2012-09-06
 suse-sles-11-sp1-v1.02.x86_64 | deleted | 2012-07-20  | 2012-03-06   | 2012-09-06
 suse-sles-11-sp1-v1.02.x86_64 | deleted | 2012-07-20  | 2012-03-06   | 2012-09-06
(3 rows)
postgres=# select name, state, publishedon, deprecatedon, deletedon from microsoftimages where deprecatedon is not Null and deprecatedon < publishedon;
                                 name                                 |  state  | publishedon | deprecatedon | deletedon  
----------------------------------------------------------------------+---------+-------------+--------------+------------
 b4590d9e3ed742e4a1d46e5424aa335e__suse-opensuse-13.1-20141216-x86-64 | deleted | 2015-01-05  | 2014-11-04   | 2016-01-14
(1 row)
rtamalin commented 2 years ago

Note that we can programmatically fix these issues in an Alembic driven data migration script; the correct deprecatedon date for a deprecated image would be the publishedon date for the replacement image, which we should be able to lookup in the same table.

However if we go this route that means we also have to ensure that data_update.py doesn't undo the fixes, or else defer landing the fixes for this issue until after we stop using data_update.py.

KeithMnemonic commented 2 years ago

@rtamalin Remember the use case where only one version of an image was ever published so there is not a replacement image. pint-data-manager will handle this as @today or a user supplied deprecation date

rtamalin commented 2 years ago

@KeithMnemonic Not sure that that will impact things in any way though, as the validation we are planing should still apply, namely that publishedon <= deprecatedon <= deletedon, and if the "user" specifies a date that fails that validation for either deprecatedon or deletedon, they will get an error ;-)

rtamalin commented 2 years ago

Seeing the data_update.py fail today because it is trying to compare a publishedon (== None) with a date stamp... Wasn't seeing this error last week in my testing, and latest pint-data data changes are not actually the cause, so it may be a subtle one related to execution ordering, e.g. last week it triggered the validator for publishedon before doing so for deprecatedon or deletedon, but this week it is triggering for deprecatedon first... So it sees like we may need to re-do this validator, either with logic to avoid hitting the issue, or instead validate on inserting/updating images records.