Patch system

[kalisp]

Comment in WIP (You may comment, changes are to be expected)

Currently anytime an update to MongoDB structure is needed as a part of a Pype upgrade, queries must be captured in update notes and then run manually during the upgrade process.

This feature should make it more automatic. The idea is to have a list of applicable db patches, mark patches that were run successfully and on any start of Pype check if all db patches present in installation were already applied. If not, run sequentially patches and mark all successes, revert (if possible) to original state if error happened.

Parts of the system:

• Delivery mechanism of patches itself and storage
• Marking of applied patches
• Triggering mechanism
• Rollback

Delivery mechanism MongoDB updates could be stored only as plain file with one or multiple queries, but better would be for each update create Python file (class) that would contain possibilities to run:

• global update (run on all collections in DB)
• project update
• Ftrack (any other third party API) update

It is recommended to create one patch (python script) per feature request, or bug fix, not a single big file per release. There is no actual dependence of patches on release itself. (It means that patches are not labeled by version number.)

Recommended name notation for patch: pype\upgrade_patches\001_patch_name.py Number in name is important for sorting

Python script will be implementing class with 3 functions: (TODO) global_db_update - everything that should be running on all projects project_db_update(project_name) - db update specific for project_name - useful for testing of updates, limiting only on single project external_api_update(project_name=None)

Marking of applied patches New collection 'upgrade_patches' will be added to pype mongoDB. It will contain documents:

[
  { 
    name: '001_my_first_patch',    
    applied_dt: '02-10-2020 15:55:00',
    type: enum('global', 'project'),     # it is applied on all projects or should be only on specific
    aplied_on_projects: [],                  # list of projects that were modified (only project that were there during upgrade ?)
    mongo_updated: true,                 # change to mongoDB structure/content
    external_api_updated: [{ api: 'ftrack'}] # any external APIs updated, FTrack most probably for now
    triggered_by: John_Doe (?)                  # id of Pype installation who triggered this
}
]

Triggering mechanism Each start of Pype will list all files in pype\upgrade_patches and compare found file names (without extension) with names from upgrade_patches. All missing will be marked to run sequentially. (Could we do "Do you accept updating DB" dialog during the start of Pype?) (? Limit this check only on runs in 'debug' mode to limit accidental triggering?) (TODO - figure out what to do if patch will be applied only on single project as a test, then it should be applied on rest. How that would be possible if upgrade should be a part of regular start of Pype. Environment variable, command line argument?) All parts(methods) of single patch will be run sequentially, only if all parts finish successfully, next patch will be run. (Investigate possibility of single transaction for all Mongo update queries.)

Rollback It might prove necessary to backup the state of DB before updates for possible rollback if error is found. Not sure if build something to backup/restore directly in Pype OR somehow use external mongodump + mongorestore scripts. In the latter case we would need to distribute these with Pype installation.

It might be useful to rename 'broken' collections and keep them on server for future debugging. Restore then would recreate all collections from backup.

(Or maybe rename all collections instead of backup, that would allow us to maybe highlight changes after update. Not sure if it is actually possible to rename the collection though.)

In case of error similar document should be created on upgrade_patches with additional 'error' field for trackback of issue for possible debugging.

Notes and possible issues It might be useful to store versions of Pype to each of the patches and stop starting of Pype with a lower version then its stored in DB. Currently there is no 'main' (wanted to called it master but thats a no-no now :-p) Pype that could trigger this, so there might be an issue with unwanted start of upgrade by someone who gets their hands on installation prematurely. Maybe introduce command line flag? (But then it might happen that updated installation package is distributed to people, but upgrade it not run yet (because of timing/personal issues).

Miro board for design https://miro.com/app/board/o9J_kjoG3R8=/?moveToWidget=3074457347479145837&cot=1

[cuID:OP-1035]

[create-issue-branch[bot]]

Branch feature/595-Patch_system created!

[antirotor]

good work, something I had in mind for some time. Rollback should be done by patch executor code for whole db. And it should be locked by first Pype instances that will trigger update. For example - we will roll out new version. It will have in patches dir file P01_upgrade_foo_in_bar-v3.1.2.py. When it finds out that this patch is not applied in db, it will test for lock there. If none present, it will create one, apply patch and remove lock. We will store in database list of applied patches to track where we are. Those could be UUID stored in each individual patch file.

[kalisp]

Comments after last weekly: Each project record in avalon DB should have added:

• 'version' number
• applied_patches - list of dictionaries of {NAME_OF_PATCH: ddMMYYYY}, denoting if and when the project was updated (for debugging purposes)
• 'freeze' field (boolean) - if present and set to True >> Patch system won't be checking it as it is explicitly telling no patch should be done (production vs staging, some project should stay on stable old version of Pype etc.)

Each patch class should contain the property "version" that matches the Pype(avalon) version. It denotes to which version of Pype this patch brings. The value of version would be used for testing of each of the projects, the patch system checks if project does not have higher version number, in that case patches for lower version won't be scheduled, as it is expected that project was created with updated up-to-date version of DB and no changes are necessary.(Eg. new project structures are created from correct schemas, there is no "base" schema that needs to be patched all the time.)

Backup of databases will be done OUTSIDE of Patch system (command line utilities should be used for better performance, there should be "Admin toolbox" GUI created which would contain helpful tools for admins only - for upgrading, purging etc.)

Structure of Patch log record in 'pype':

• affects - list of targets that are modified by this patch (['avalon_db', 'pype_db', 'ftrack','settings'...]
• description - dictionary {'avalon_db': 'Changed task from list to dictionary', 'ftrack': 'Modified custom attributes'} - to visually check what patch was actually doing - for debugging purposes
• applied_on - dictionary of {'global': MMDDYYYYHHmmss,'project_A':MMDDYYYYhhmmss} - to highlight when patch was applied, if it was 'global' - not project depended changes (either Avalon, Ftrack or Settings) or 'PROJECT_NAME' - changes run per project
• implemented_by_PR - id of pull request that is implementing this patch - for debugging

Structure of AbstractPatch should be simplified and separated into 2 main methods:

• global -- should be run only ONCE per patch -- could contain different targets (avalon_db, settings, ftrack project changes ...)
• project_changes(project_name) -- run repeatedly for each of the projects (but only ONCE per project!) -- could be run at different times (first run on 'test_project' >> check >> OK >> run on all projects) -- it is 'patch author' responsibility to exclude any projects that shouldn't be upgraded -- it is 'upgrader' responsibility to run patch in correct configuration (eg. if patch is run on single test project, patch needs to be triggered to run on rest of the projects)

(Note outside of scope of Patch system - all projects in Avalon DB should contain 'tag' - "production"|"staging" - each event server should contain same tag and handle only appropriate projects, eg. production event server handles only "production" projects - that way staging and production could run in parallel for testing. Issue will be in Ftrack - that would probably need to be completely separate installation.)

[kalisp]

Implemented changes according to last comment.

State of 'project' document of upgraded project:

Look of log document:

[mkolar]

A few points towards the stored data.

Project['version'] should probably be renamed to project['pype_version'] we should keep in mind that we are building on top of avalon default structure for the most part, and I'd prefer being a bit more explicit.

Considering we're adding multiple extra data entries wouldn't it make sense for us to group them under one field? Also you're missing the applied patch dates in the example.

Regarding the log record. Do we need affects array and description as disctionry? Looks to me that the dictionary is enough as it hold all the information.

[kalisp]

Ad log record: I am a bit hesitant to keep only description, as values in array are used as a decision points in the process. I am worried that people would rather skip 'description' portion of patch implementation than separate value, not called 'description'. But if we create a process that description is really important here, I am not against merging that array together.

[BigRoy]

How relevant is this still with AYON on the horizon? Should we close @mkolar ? Or does some of the original issue apply to AYON's backend as well?