frappe / press

Full service cloud hosting for the Frappe stack - powers Frappe Cloud
https://frappe.cloud
GNU Affero General Public License v3.0
279 stars 195 forks source link

fix: stop processing in create_usage_records job if it took more than 15min #2260

Open tanmoysrt opened 2 weeks ago

tanmoysrt commented 2 weeks ago

In create_usage_records we process the usage of sites synchronously in a batch size of 200.

we hold lock on the record at the time of submission of usage record.

https://github.com/frappe/press/blob/2f435f835d81e6df33610fbd4090e6bcf4289c89/press/press/doctype/usage_record/usage_record.py#L64-L68

When there is some wait for lock on a record and it delay the overall job, it can exceed 15 minute. Meanwhile, next job started and start processing same record, hence more errors.

Multiple instance of same job running Screenshot 2024-11-06 at 9 43 36 AM

If we can stop process further records after 15 minute, it can reduce the errors.

codecov[bot] commented 2 weeks ago

Codecov Report

Attention: Patch coverage is 25.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 39.90%. Comparing base (2f435f8) to head (58fff1c). Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
press/press/doctype/subscription/subscription.py 25.00% 3 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #2260 +/- ## ========================================== + Coverage 29.94% 39.90% +9.95% ========================================== Files 371 371 Lines 28078 28082 +4 ========================================== + Hits 8408 11206 +2798 + Misses 19670 16876 -2794 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.