Empty budget data error on fetching

[cocox]

When trying to run it it gives the following error:

Traceback (most recent call last): File "/app/firefly-iii-monthly-email-report/monthly-report.py", line 478, in main() File "/app/firefly-iii-monthly-email-report/monthly-report.py", line 82, in main spent = budgetsSpentCategories['data'][0]['attributes']['spent']

IndexError: list index out of range

[cocox]

I think problem is with last dataset that looks like: {'data': [], 'links': {'first': 'http://fireflyiii-core:8080/api/v1/budgets/11/limits?limit=65&page=1', 'last': 'http://fireflyiii-core:8080/api/v1/budgets/11/limits?limit=65&page=1', 'self': 'http://fireflyiii-core:8080/api/v1/budgets/11/limits?limit=65&page=1'}, 'meta': {'pagination': {'count': 0, 'current_page': 1, 'per_page': 65, 'total': 0, 'total_pages': 1}}}

[fauxir]

Thanks for sending this through, I will have a look and see why it shows 65 items on the page since the limit is 50. Bear with me.

[fauxir]

Right, after some testing I concluded that the final data set provided by your instance of Firefly is either corrupted or there is something wrong going on. I would recommend checking your budgets and seeing if there is anything "abnormal" you can spot before going ahead with the solution provided below.

To combat that I've added a conditional that will check if the data provided is non-empty. That should allow you to "skip" any empty datasets and get the populated ones.

You can find the patched version on the Non-empty-budget-data-patch branch. Give it a go and let me know if it works for you so I can merge that into the main branch.

[cocox]

In the branch it gives this error.:

raceback (most recent call last): File "/app/firefly-iii-monthly-email-report/monthly-report.py", line 481, in main() File "/app/firefly-iii-monthly-email-report/monthly-report.py", line 83, in main if len(budgetsSpentCategories['data']) > 0:

KeyError: 'data'

[fauxir]

Interesting one, how many budgets do you have set up?

[cocox]

I have about 54 active budgets and 2 inactive budgets.

[fauxir]

Nice one, I've finally managed to replicate the issue. Working on a fix!

[fauxir]

I've just merged the patch on the main branch. It looks like Firefly creates a blank budget ID when you delete a budget that will send this message on fetch: {'message': 'Resource not found', 'exception': 'NotFoundHttpException'} thus the error above. Now it should jump over that error message and fetch the correct budget data. Let me know how it works for you.

[pvieira84]

I'm having an issue with the same instruction but with error message "IndexError: list index out of range"

I have 4 active budgets and 1 inactive

[fauxir]

To double-check, are you using the latest version of the script?

[pvieira84]

I am, sorry for not mentioning it

[fauxir]

Right, can you think of anything you have done to the budgets? Changed positions, deleted, deleted multiple budgets? This will be important so I can replicate the bug and work on a fix. Any information is welcome.

[fauxir]

I suspect that the problem might be here: if budgetsSpentCategories == {'message': 'Resource not found', 'exception': 'NotFoundHttpException'}. This checks if you're getting an error message instead of the budget data details. If you have a different error message then that's why you're getting the error above. I will need to test a few assumptions in order to fix this. Bear with me.

[pvieira84]

I have not explored much with Budgets yet. This is what I have currently:

• id 1: active
• id 2: active
• id 3: deleted
• id 4: inactive
• id 5: active
• id 6: active

[fauxir]

I've just updated the script on a test branch that is available here: https://github.com/fauxir/firefly-iii-monthly-email-report/blob/EmptyBudgetDataErrorOnFetching/monthly-report.py

Taking into consideration that I cannot review your data due to privacy matters and I cannot replicate the bug locally the fix was implemented on the assumption that if the data extracted is not of the required type a loop will go to the next budget ID looking for that specific type. Once found it should be able to continue generating the the report.

Let me know how this works.

[pvieira84]

I just tried your new version but I'm getting the same error. Can you not try to simulate my behavior with the demo site?

[zarguell]

I'm getting a similar error - I suspect because of the budget index finder/handling logic for empty budgets.

Here's my error, I packaged the script up in a simple container and am mounting my custom config to run it:

➜  firefly-report sudo docker run --rm -v ./config.yaml:/app/config.yaml firefly-iii-monthly-email-report
Traceback (most recent call last):
  File "/app/monthly-report.py", line 496, in <module>
    main()
  File "/app/monthly-report.py", line 98, in main
    spent = budgetsSpentCategories['data'][0]['attributes']['spent']
            ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
KeyError: 'data'

For me, I have 6 budgets, but I deleted a bunch at some points, so I have some gaps in indexes. So if I test index 5 - 9 in debug console, 5 and 9 return data, but 6 through 8 don't. It looks like the code hits the line 89 check for if budgetsSpentCategories == {'message': 'Resource not found', 'exception': 'NotFoundHttpException'}: for the first iteration of a non-existent budget (index 6), but then it assumes that 7 must exist, instead of iteratively incrementing the budgetFetchIndex until the next active budget is found (which needs to be 9 in my case). I'm thinking the code needs some sort of loop to increment until a valid budget is found?

Here's my debug console output for getting budgets 5 through 9, and the value of my "budgets" array

s.get('https://firefly/api/v1/budgets/5/limits?start=2024-04-012024-04-30').json()
{'data': [{...}, {...}], 'included': [{...}], 'meta': {'pagination': {...}}, 'links': {'self': 'http://firefly/api/v1/budgets/5/limits?limit=50&page=1', 'first': 'http://firefly/api/v1/budgets/5/limits?limit=50&page=1', 'last': 'http://firefly/api/v1/budgets/5/limits?limit=50&page=1'}}

s.get('https://firefly/api/v1/budgets/6/limits?start=2024-04-012024-04-30').json()
{'message': 'Resource not found', 'exception': 'NotFoundHttpException'}

s.get('https://firefly/api/v1/budgets/7/limits?start=2024-04-012024-04-30').json()
{'message': 'Resource not found', 'exception': 'NotFoundHttpException'}

s.get('https://firefly/api/v1/budgets/8/limits?start=2024-04-012024-04-30').json()
{'message': 'Resource not found', 'exception': 'NotFoundHttpException'}

s.get('https://firefly/api/v1/budgets/9/limits?start=2024-04-012024-04-30').json()
{'data': [{...}], 'included': [{...}], 'meta': {'pagination': {...}}, 'links': {'self': 'http://firefly/api/v1/budgets/9/limits?limit=50&page=1', 'first': 'http://firefly/api/v1/budgets/9/limits?limit=50&page=1', 'last': 'http://firefly/api/v1/budgets/9/limits?limit=50&page=1'}}

budgets
[{'name': 'Restaurants', 'budgeted': None, 'spent': 1.00}, {'name': 'Groceries', 'budgeted': None}, {'name': 'Fun', 'budgeted': None}, {'name': 'Transportation', 'budgeted': None}, {'name': 'Miscellaneous', 'budgeted': None}, {'name': 'Bills', 'budgeted': None}]
special variables
function variables
0:
{'name': 'Restaurants', 'budgeted': None, 'spent': 1.00}
1:
{'name': 'Groceries', 'budgeted': None}
2:
{'name': 'Fun', 'budgeted': None}
3:
{'name': 'Transportation', 'budgeted': None}
4:
{'name': 'Miscellaneous', 'budgeted': None}
5:
{'name': 'Bills', 'budgeted': None}
len():
6