It could be a problem with using the folio_get_all function. Some suggestions for refactoring:
use the storage endpoints instead
expand the lists into mapped tasks (so that retries are not retrying the whole list again)
try using folio_get instead of folio_get_all for the _get_all_invoice_lines (we realistically will probably not have more than 200 invoice lines per invoice)
maybe we can keep folio_get_all for the _get_all_ids_from_invoices function?
if we keep getting errors for the _get_all_ids_from_invoices function, then we should reconsider the DAG schedule when trying to run this with a from_date of 8/28/2023. Maybe we do catchup=True, in the DAG configuration in order to spawn many DAG runs with only weekly paymentDate ranges.
See #1259 for some background.
It could be a problem with using the
folio_get_allfunction. Some suggestions for refactoring:_get_all_invoice_lines(we realistically will probably not have more than 200 invoice lines per invoice)_get_all_ids_from_invoicesfunction?_get_all_ids_from_invoicesfunction, then we should reconsider the DAG schedule when trying to run this with a from_date of 8/28/2023. Maybe we docatchup=True,in the DAG configuration in order to spawn many DAG runs with only weekly paymentDate ranges.