TASK: Robust Metamist Integrations

[milo-hyben]

This PR is implementing items for task Robust Metamist Integrations on the road map.

Add a re-try mechanism to catch failing to connect to metamist when writing analysis entries. Gracefully catch and handle failures, when writing to metamist fails. Add integration tests to validate these behaviours.

TODO

• [x] Increase test coverage for metamist.py (currently on main 42.8%) to as close to 100% as possible WIP, this PR latest coverage at 76%

• [ ] Two functions in metamist are with comments 'not used anywhere' (process_existing_analysis, find_joint_calling_analysis), Should we remove? It would make test coverage at 91% after this PR.

• [x] Make API call more robust, esp. differentiate between thrown API Exception It seems we just need to differentiate between ServiceException and the rest of ApiException. ServiceException can be retried or other we should stop.

• [x] If getting HTTP 504 (Gateway Timeout), try to reconnect, addressing this issue: https://github.com/populationgenomics/production-pipelines/issues/585 Error Details: It fails on

  "/cpg_workflows/status.py", line 73, in complete_analysis_job
  a_id = aapi.create_analysis(project=project_name, analysis=this_analysis)

  Exception:

  ServiceException(http_resp=r)
  metamist.exceptions.ServiceException: (504)
  Reason: Gateway Timeout

• [ ] investigate the logic in update_analysis, seems to always set the status, even when API fails: https://github.com/populationgenomics/production-pipelines/blob/f4801c6804c56f865f2047019ad679f9c59b7de4/cpg_workflows/metamist.py#L318

• [x] During writing unit tests discover this condition is wrong, missing brackets around or, basically it would never throw MetamistError for invalid index file extension: https://github.com/populationgenomics/production-pipelines/blob/f4801c6804c56f865f2047019ad679f9c59b7de4/cpg_workflows/metamist.py#L635

[vivbak]

Two functions in metamist are with comments 'not used anywhere' (process_existing_analysis, find_joint_calling_analysis), Should we remove? It would make test coverage at 91% after this PR.

Yes, but could we do it in a separate PR (merged before this one). If something goes wrong it's easier to track down the cause!