Identify and prioritize pressing app refactors and problems

[lyzadanger]

We need to determine what's most urgent to remediate in the app—things that should be addressed before extending the app with more features.

[seanh]

This will grow as we start working on improving the app and discover more things, but here's an initial list of the "main things" after doing a first pass over the code:

Need to be able to operate the app going forward (e.g. to be able to diagnose problems users are seeing):

• [ ] Don't raise HTTPBadGateway.
  CloudFlare messes with it.

• [ ] Improve logging of both LMS app and h.
  Perhaps by setting up AWS CloudWatch?

• [ ] Improve error and crash reporting of LMS app.
  Showing better error messages to users: both a friendly / generic error and a details one.
  Making sure that errors are logged to Sentry with a decent amount of detail.
  Related GH issue: https://github.com/hypothesis/lms/issues/174

• [ ] Fix known LMS app crashes. There are 16 unresolved issues in Sentry, and that's with many issues failing to be reported to Sentry at all. At some point these all need to be fixed -- the app shouldn't be crashing for users.

• [ ] Replace the metrics / "reports" feature with something viable.

Needed to be able to maintain and extend the app (fix bugs, add new features, make changes):

• [ ] De-tangle the view code.
  View decorators. Tangled helper functions. These need to be sorted out before it's reasonable to extend and maintain the app.

• [ ] Add a services layer and refactor various things into services.
  There's a handful of services that the new provisioning feature code could benefit from, e.g. an h API client service.

• [ ] Add a centralised validation package.
  The app needs one place to do request parameter validation and generate error messages etc consistently, instead of spreading validation logic throughout the entire app leading to long functions and duplication.
  Related GH issues:

  • https://github.com/hypothesis/lms/issues/175.
  • https://github.com/hypothesis/lms/issues/137
  • https://github.com/hypothesis/lms/issues/134
  • https://github.com/hypothesis/lms/issues/133
  • https://github.com/hypothesis/lms/issues/63

• [ ] Isolate the unit tests.
  The app doesn't have properly isolated unit tests and, given the poor design of the code, this is very bad news for extending and maintaining the app.

• [ ] Add a feature flagging mechanism.
  The app needs a feature flagging mechanism similar to h's where feature flags can be turned on and off using an admin dashboard. We're going to need this to develop new features.

• [ ] Completely rewrite the app's OAuth 2.0 support.
  The app asks user's to authorize it on every single launch (failing to re-use access tokens), it has no refresh token support at all, it does OAuth authorization in an iframe, and it has various OAuth-related crashes: https://github.com/hypothesis/lms/issues?q=is%3Aissue+is%3Aopen+label%3AOAuth The OAuth code is also some of the worst, most tangled code in the app and it interacts with other code (e.g. LTI launch views) in a way that forces that code to be tangled too.

Smaller things:

• [ ] Copy over the new Tox stuff from h.
  No need to create a dev venv manually, make shell and make sql, etc.

• [ ] Get Black code formatting in place.

• [ ] Move traversal resources out of config.
  These belong in an lms.traversal package not in lms.config.

• [ ] Fix import style throughout.
  Packages shouldn't reach deeply into each other etc.

• [ ] Remove unnecessary "coding" comments.
  These aren't needed in Python 3.
  We may want to look for other Python 3 things futurize, modernize and linters might tell us about.

• [ ] Tidy up module-level functions into @staticmethods and @classmethods.

• [ ] Remove unised dependencies.
  Evaluate the app's dependencies and see if there's any we can remove.