Improve bill-teams - [merged] #63

Closed alexander-haller closed 4 months ago

In GitLab by @mhxion on Nov 6, 2023, 15:36

Merges refactor-bill-teams -> dev

This PR adds the following 3 improvements to bill-items.

Split `bill-items` into sub-commands

For our purposes, we mostly care about invoice generation. We may also wish to have some invoice customization capability from the CLI. Up until now, bill-teams would only produce structured data in JSON/YAML/TEXT formats. To make room for invoice generation it only make things simpler if we split the two actions into sub-commands: info and generate-invoice. Screenshot_2023-11-06_at_3.09.55_PM

info would work the same way bill-teams used to work. generate-invoice will run info for us and generate invoice(s) based on the teams data from info. This gives us more freedom to extend and improve the CLI functionalities of generate-invoice (e.g., generate aggregated invoice every 6-month). This also solves the issue with making --export the default behavior for bill-teams. generate-invoice sub-command would implicitly assume --export is turned on. E.g.,

$ elapi bill-teams generate-invoice
# This one command will fetch teams data, convert the data to Markdown (for now), 
# and export to the directory defined in elapi.yaml configuration file.

Retry after failure

This has been a long-awaited feature! This PR finally adds support for a retry method for bill-teams. Lots of refactoring had to made so retry can work seamlessly. I believe how the retry logic (i.e., when to trigger the retry and for which error) should be implemented is something that's best left to be discussed at our meetings. For now, the retry logic a follows an exponential increment in time. In total there will be 5 retry attempts.

$ elapi bill-teams generate-invoice <- first run (retry count = 0)
...
1st Network error <- 1st retry is triggered (retry count = 1)
> Retry will wait 60 seconds before running bill-teams again.

2nd Network error <- 2nd retry is triggered (retry count = 2)
> Retry will wait 2 x 60s = 120 second before running bill-teams again.

3rd Network error <- 3rd retry is triggered (retry count = 3)
> Retry will wait 2 x 2 x 60s = 240 second before running bill-teams again.
# It will go on like this until retry count reaches 5.

The total retry wait period spans over an hour. Worth noting, the retry is triggered against network-related errors, and not against configuration-related errors (e.g., missing API token).

We're using a library tenacity to accomplish this retry. Changing any of the above-mentioned behavior shouldn't be too hard if you have something entirely else in mind for how to retry 🙂.

Async by default

elAPI now runs bill-teams asynchronously by default. Previously, an --async flag was required which added completely unnecessary maintenance overhead! As discussed earlier, with async, bill-teams can finish generating invoice of 2500 members in under 5 minutes.

No temporary caching

elAPI no longer needs to store sensitive data in /var/tmp/elapi. The only reason we had it in the first place is out of fear of data loss and for debugging purposes. elAPI is much more stable now than before! The cleanup feature will be completely removed in an upcoming PR.

In GitLab by @mhxion on Nov 6, 2023, 15:37

requested review from @alexander-haller

approved this merge request

uhd-urz / elAPI