bcgov / traction

Traction is designed with an API-first architecture layered on top of Hyperledger Aries Cloud Agent Python (ACA-Py) and streamlines the process of sending and receiving digital credentials for governments and organizations.
https://digital.gov.bc.ca/digital-trust/tools/traction/
Apache License 2.0
52 stars 50 forks source link

Script to clean-up old reservation fields from database #810

Closed esune closed 12 months ago

esune commented 1 year ago

As a follow-up to #773, we should provide a migration script that collects the data - if available - from the old enumerated reservation fields, saves it to a JSON object in the appropriate database column and drops the old unnecessary columns from the table.

Unsure on whether there are best practices to be aware of for plugin-ins for scenarios like these - c.c.: @usingtechnology

usingtechnology commented 1 year ago

My $.02 is to avoid leaning on or treating this like a relational database because it is not. I am unsure if anyone has metrics on how growing fields and record size affect performance. There is no ORM or SQL scripts so migrations can be tricky. I need to do more to know all the sticking points, but dropping fields can be an issue. If any data gets left behind, it will blow up when it attempts to marshal itself out of the record.

There are update/upgrade steps, and ACA-Py does a lot of updates and changes to the record structures, so it is very much doable, just not in the way one usually thinks. See upgrade command. Doesn't look like we've done upgrades that change the existing record data yet.

I guess look over the upgrade process and see what/how to leverage that concept to manipulate the record types you need. Kind of an interesting thought experiment with regard to plugins... how do we support/enable this requirement for plugins to update their record types and data? There is a version in the plugin, but I don't think there is anything that persists the version so you know that you are jumping from plugin version 1 to version N.

And I suppose the other question would be, do you need this? Are there reservations in PROD that need persisting? Are there environments/instances that cannot be reset? Eventually, you/we will need to tackle the version -> data mismatch problem but maybe not now?

esune commented 1 year ago

@usingtechnology thanks for the input. Based on your feedback and on the fact that we do not have any production instance that requires that data I think we might just leave this for now, and revisit if/when the situation requires it.