Upgrade marshmallow dependency on api

[sastels]

Description

As a Notify developer I need to keep our dependancies up to date.

WHY are we building?

• Marshmallow is at 2.21.0, released 2020-03-05. - It is the last 2.* version (latest is 3.20.0).
• Minor upgrades to some of our other dependancies require at least 3.0.0

WHAT are we building?

• upgrade marshmallow to at least 3.0.0
• currently with 3.0.0 our tests fail to collect:

  ERROR tests/app/celery/test_tasks.py - ValueError: Invalid fields for <UserSchema(many=False)>: {'user_to_service', 'user_to_organisation'}.
  ERROR tests/app/dao/test_services_dao.py - ValueError: Invalid fields for <UserSchema(many=False)>: {'user_to_service', 'user_to_organisation'}.
  ERROR tests/app/dao/test_templates_dao.py - ValueError: Invalid fields for <UserSchema(many=False)>: {'user_to_service', 'user_to_organisation'}.
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  We've known about this bug for a while

  VALUE created by our solution

We can upgrade our other dependancies

Acceptance Criteria

• [ ] marshmallow upgraded to 3.0.0 or higher
• [ ] test pass
• [ ] smoke tests pass
• [ ] everything looks normal in admin

QA Steps

• [ ] smoke tests in staging or prod
• [ ] admin looks fine in staging or prod

[jimleroyer]

I started on the task and replicate GDS did in their upgrade, minus a few differences.

[sastels]

work progresses, no big issues found so far

[jimleroyer]

I continued work yesterday on this. I am in the weeds with it as I applied most changes but now I need to debug. There is some code that does not align well with the strict default required by the schema definition by marshmallow, which GDS did but GCNotify didn't over the past years since the fork.

[jimleroyer]

Made some progress but moving back to the shelves as we have a higher priority task to tackle.

[jimleroyer]

Made further changes to the schema.

[jimleroyer]

I am at 99% test working, have to review the remaining 1% and then test locally extensively.

[ben851]

• Progress blocked by incident on Thursday
• Friday was full of support issues and meetings

[ben851]

• Jimmy resolved some tests, but is running into some bugs on certain tests.

[sastels]

branch / PR https://github.com/cds-snc/notification-api/pull/1844

[jimleroyer]

Jimmy and Steve worked on the task and got all automated tests to run. Next step is to run locally and test it out.

[jimleroyer]

Jimmy working on his local setup; his login is not working.

[sastels]

Steve will try using this branch to create a new user (where Jimmy is having an issue with the verification email not being sent)

[jimleroyer]

Steve and Jimmy made it work after finding a few related and unrelated issues. Jimmy to test more thoroughly.

[ben851]

Jimmy performed some tests and broke Admin. Jimmy will re-evaluate his fix to ensure existing functionality doesn't break.

There are some fields relating to mail/letter functionality that are still being referenced in admin that we tried to remove, but caused issues.

New card to remove letters could be created to tackle this reference removal.

[jimleroyer]

Tested the application locally, fixed a bug but now it seems to be working good. The PR would need a review.

[jimleroyer]

Andrew is taking a look at the PR for review. We might do a bug bash after deploying it to staging.

[jimleroyer]

Jimmy to reply to Andrew's questions PR.

[jimleroyer]

Answered to Andrew's question and waiting for this reply. Will tested and approved the PR so it would be ready to go.

[sastels]

Still being reviewed

[sastels]

good idea to do a bug bash on staging before merging to prod

[jimleroyer]

Bug bash is scheduled for tomorrow at 13:30 EST.