PTV-1887 - linting, formatting, updating reps, fixing tests

eapearson commented 1 year ago

This set of changes aims to put the codebase into a stable, updated state in order to be able to make further changes. The heart of, and initial set of, the changes was to apply uniform formatting via black, linting via pylint and sonarlint. Once these fixes were made, tests were run and any broken tests fixed until there were no errors. However, once tests were passing, it was clear that not all tests were being invoked - code coverage was at about 50% and there were many tests skipped due to incorrect async usage and a few other issues. In fact, code coverage reports were not being generated either. So the next effort was to update all dependencies, including adding a missing dependency required for asynchronous testing. Once through this, coverage was up to 84%. Finally, once the changes were captured in a pull request, a series of changes were made to address any issues raised by sonar cloud or codify until they were all resolved. Some of these issues are specific to repeated values used in tests, which were simply annotated with # NOSONAR rather than rework the tests, as there are dozens if not hundreds of inline test data, and probably a 30-40 instances of duplicated test data which needed annotating. There are no functional changes made around design. All code changes were made to address code quality issues raised by linting and related tools.

I did not include some changes that would help keep the codebase in shape. E.g. there should be tools (preferably docker-based) to apply formatting, linting checks, certainly ad-hoc but also via pre-commit hook and in the GitHub actions. Other welcome additions would be tightening up the type annotations, including mypy (currently 69 errors, and would require several more dependencies), and separating developer dependencies from those required by the service (runtime.)

eapearson commented 1 year ago

Hello @bio-boris. Here are the changes without the functional changes to fix and improve uploading. I'll start working on those next,

eapearson commented 1 year ago

@bio-boris I've made a few changes since yesterday. Mostly just around release notes, and a few odds and ends. It would be nice to get control over codacy and other code quality tools to get them to align with developer tools. It mostly does, but e.g. codacy uses "remark-lint" whereas markdownlint seems more popular and widely supported.

After this, I'll have a PR with the functional changes for fixing upload.