search

SUSE-Enceladus / public-cloud-info-service

Find IPs of service endpoints, and information about published images.
Apache License 2.0
5 stars 8 forks source link

changeinfo URLs should end with a / #113

Closed rtamalin closed 2 years ago

rtamalin commented 2 years ago

Add a validator that ensures that the changeinfo URLs end in a /.

Also add a data migration step which will make the same change to all the existing changeinfo URLs in the database for all of the provider images tables.

Update the data_update.py XML data extraction to automatically add a trailing slash, if not present, to the changeinfo entries loaded from the XML data, ensuring that future incremental data updates from XML data don't incorrectly report data updates when processing older entries that didn't end with '/'.

Bump version to 2.0.9.

Relates: #98

rtamalin commented 2 years ago

Example relative to a "production" equivalent database:

% ../bin/schema_upgrade.sh --host pdm-dev --user snotty --password MasterSlobs --database postgres upgrade
INFO:__main__:Creating version control
INFO:__main__:Upgrading schema
INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
INFO:alembic.runtime.migration:Will assume transactional DDL.
INFO:alembic.runtime.migration:Running upgrade 8c5fc3cd9b18 -> 14ae3b1b5e81, image dates validation
INFO:alembic.runtime.migration:Running upgrade 14ae3b1b5e81 -> e937749e3f8b, servers partial unique keys
INFO:alembic.runtime.migration:Running upgrade e937749e3f8b -> ee82c541fae0, changeinfo urls end in slash
INFO:ee82c541fae0_changeinfo_urls_end_in_slash.py:alibabaimages: Updated 97 changeinfo entries
INFO:ee82c541fae0_changeinfo_urls_end_in_slash.py:googleimages: Updated 21 changeinfo entries
INFO:ee82c541fae0_changeinfo_urls_end_in_slash.py:microsoftimages: Updated 701 changeinfo entries
INFO:ee82c541fae0_changeinfo_urls_end_in_slash.py:Committing 819 changeinfo fixes
Pint database schema migration successfully completed.
% ../bin/data_update.sh --host pdm-dev --user snotty --password MasterSlobs --database postgres update
INFO:__main__:Updating data
INFO:__main__:Extracting XML data for 'alibaba' provider...
INFO:__main__:Extracting XML data for 'oracle' provider...
INFO:__main__:Extracting XML data for 'microsoft' provider...
INFO:__main__:Extracting XML data for 'amazon' provider...
INFO:__main__:Extracting XML data for 'google' provider...
INFO:__main__:No new entries found for model 'AlibabaImagesModel'
INFO:__main__:No new entries found for model 'OracleImagesModel'
INFO:__main__:No new entries found for model 'MicrosoftImagesModel'
INFO:__main__:No new entries found for model 'MicrosoftServersModel'
INFO:__main__:No new entries found for model 'MicrosoftRegionMapModel'
INFO:__main__:No new entries found for model 'AmazonImagesModel'
INFO:__main__:No new entries found for model 'AmazonServersModel'
INFO:__main__:No new entries found for model 'GoogleImagesModel'
INFO:__main__:No new entries found for model 'GoogleServersModel'
Pint database successfully updated.
rtamalin commented 2 years ago

Note that the changes included in this PR are a subset of those found in the draft PR #105 which I will re-work to be based upon these changes next.