[Feedback]: Custom Reports

carkom commented 10 months ago

Thanks to everyone for their input in getting this merged. Thanks to the maintainers for the reviews. Anything not listed please feel free to add to the conversation. Cheers!

Update: moving completed items down and consolidating most recent feedback items

Known Issues:

[x] Newly hidden categories do not update the dashboad cards. The actual report correctly excludes newly hidden categories.
[x] Table Totals need a callback

Upcoming Additions:

[x] Add interval selection (by: day, week, month, year)
[x] Implement disabledList so all disabled functions can get updated/maintained in one place
[ ] Drag and drop dashboard

UI suggestions:

[ ] Condsider adding Rolling Averages (like Category Spending graph)
[ ] Add Net Worth numbers
[x] Click graph items (table cell, donut section, single bar, etc.) and see transaction that contribute to that cell *all done except table graph
[ ] onHover first columns (make faster)
[x] Change Category sort order
[ ] Refactor show hidden and show empty toggles

Version 1 Known Issues:

[x] Need to find a way to freeze first row column when scrolling (table graph) (fixed by #1996)
[x] Would like to freeze bottom "totals" row as well. Not sure it's possible (table graph) (same as above)
[x] Table Rendering is funky when using filters (makes me think my rendering could be done much better) (same as above)
- Add tnum to table formats
[x] Load Payees is intermitent (must navigate to payee page to load them into app before it will work)
[x] Hidden Groups crash Bar Graph
- [x] Hidden Groups crash Table Graph
[x] Error when changing "split" drop down in "time" mode for StackedBarGraph
[x] The Range: "all time" is not working.
[x] Loading time for large budgets is quite slow (fixed by #1988)
[x] dataset does not update on "Type" change
[x] Add interval selection (by: day, week, fortnight, month, year)

Upcoming Additions:

[x] Saving the graph to the "Reports" homepage (#1924)
- Drag and drop to reorder
- "..." to rename or delete on homepage
[x] Make relative saved graphs update when opened (eg. current month)
[x] Enable "Show Label" button
[x] Enable "Show Legend" button
[x] Add range options "year to date" and "last year"
[x] Figure out logic for when/how to show labels on each graph
[x] Create Live/Static choice for date ranges.

UI suggestions from reviewers (open to discussion, leave your comment!):

[x] Legend and Summary on by default (changed to global variables)
[ ] Condsider adding Rolling Averages (like Category Spending graph)
[x] Display percentages in pie chart (argument here is that it could get really busy with 30+ categories)
- It's been suggested to only display pieces with more than 10%
[ ] Collapse Groups in matrix table (not entirely neccessary since groups can be found in the "split" menu). A nice functionality to consider though.
[x] Remove hover highlight on Bar/StackedBar
[ ] Add Net Worth numbers
[x] Add select all and unselect all buttons to categorySelector
[x] Use global variables for view (legend/summart/labels)

Backend planned additions/changes to improve code/functionality:

[ ] Improve state variables
[ ] Implement disabledList so all disabled functions can get updated/maintained in one place

Kidglove57 commented 10 months ago

Great work! It looks as if it must have been a huge task for which we are very grateful indeed.

My main use will be of the Table view. As you say, the alignment of numbers (particularly larger numbers) in the table still needs some work and also this alignment seems inconsistent between various time-scale views.

The "year to date" and "last year" options will be great additions!

Kidglove57 commented 10 months ago

In my dream world I would also love to be able to click through the table entry to see the underlying transactions!

youngcw commented 10 months ago

Hidden category groups break the processing. The error below is what gets printed, but the core of the issue is that the categoryGroupCalcData ends up with a null entry for each hidden category.

default-spreadsheet.tsx:350 Uncaught (in promise) TypeError: Cannot read properties of null (reading 'id')
  ¦ at default-spreadsheet.tsx:350:19
  ¦ at Array.map (<anonymous>)
  ¦ at default-spreadsheet.tsx:349:29
  ¦ at Array.map (<anonymous>)
  ¦ at default-spreadsheet.tsx:332:40
(anonymous) @ default-spreadsheet.tsx:350
(anonymous) @ default-spreadsheet.tsx:349
(anonymous) @ default-spreadsheet.tsx:332
Promise.then (async)
(anonymous) @ useReport.js:11
sS @ react-dom.production.min.js:244
lM @ react-dom.production.min.js:286
lP @ react-dom.production.min.js:283
lO @ react-dom.production.min.js:281
ly @ react-dom.production.min.js:273
i_ @ react-dom.production.min.js:127
(anonymous) @ react-dom.production.min.js:267

Shazib commented 10 months ago

Awesome work. Couple issues i've noticed.

Selecting a payee filter seems to show it as deleted

Flipping between the 'Type' when in 'Time' mode crashes the app.

Kidglove57 commented 10 months ago

“ Flipping between the 'Type' when in 'Time' mode crashes the app” I have been unable to reproduce this crash on my live budget. Although I do get the crash when selecting a Split other than category EDIT I should have added that this happened in the stacked bar graph. But seems to be fixed in 1930.

Redbox3070 commented 10 months ago

Really happy to have this feature!

The Range: "all time" is not working.
Reducing Range in table view does not work properly I saw the initial save report feature (https://github.com/actualbudget/actual/pull/1924) and it's sleek
not a big fan of the grey hover color for bar charts and I would remove it in the reports page

idea for the future that I had: net worth view which displays a breakdown of account balance. Ideally with an area chart

MatissJanis commented 10 months ago

Overall: super impressive work! This is a MASSIVE improvement to Actual. I will personally be using it loads as well! Thanks for all the hard work!

My feedback (sorry for nitpicking):

[x] loading indicators - I think we're missing these; for large budgets (such as mine) it takes ~40 seconds for the split_by:payee to load the data - and there is no loading indicator while this is happening.
[ ] if I change from split_by:payee to split_by:group while payee is still loading - I will get a weird "flash" of the payee data after it finishes loading - I think the previous calculations don't get cancelled
[x] the category visibility selector gets hidden on small screens with no ability to scroll in it - I think we're missing a minHeight
[x] would be nice to add on-hover tooltips to the icons (i.e. show summary); the native title attribute shows up with a bit of a delay

Kidglove57 commented 10 months ago

When reporting on categories it would be great to have a select and deselect all button. Particularly the latter, In order to quickly narrow down the categories required for a particular report.

carkom commented 10 months ago

When reporting on categories it would be great to have a select and deselect all button. Particularly the latter, In order to quickly narrow down the categories required for a particular report.

Can you specify what part of the page you are referring to?

Kidglove57 commented 10 months ago

“Can you specify what part of the page you are referring to?” Sorry for being unclear! I am referring to the sidebar in the Custom Reports screen where all Groups and Categories are ticked. I would like to be able to “deselect all” (rather than individually deselecting Grouos or Categories. This would make it quicker for me to then individually select just a few categories to compare.

Shazib commented 10 months ago

“ Flipping between the 'Type' when in 'Time' mode crashes the app” I have been unable to reproduce this crash on my live budget. Although I do get the crash when selecting a Split other than category EDIT I should have added that this happened in the stacked bar graph. But seems to be fixed in 1930.

Sorry there is an additional step in the sequence

switch from Total to Time, then change the Split

Here is a clip from demo.actualbudget.org

https://github.com/actualbudget/actual/assets/4405777/34451f1a-0463-4c73-bc5f-cf8892f65be9

carkom commented 10 months ago

* [x]  loading indicators - I think we're missing these; for large budgets (such as mine) it takes ~40 seconds for the `split_by:payee` to load the data - and there is no loading indicator while this is happening.

Can you check this on #1930? I put in a fix but can't test it since my budget isn't that big...

Kidglove57 commented 10 months ago

* [x]  loading indicators - I think we're missing these; for large budgets (such as mine) it takes ~40 seconds for the `split_by:payee` to load the data - and there is no loading indicator while this is happening.
Can you check this on #1930? I put in a fix but can't test it since my budget isn't that big...

My budget goes back to Dec 2019 so I tested this in Netlifly. I used "all time", in "table view" and in "time mode". Loading time changing from category to payee was 13 seconds in Safari, although no indication that loading was happening.

Kidglove57 commented 10 months ago

PS: Thank you so much for adding in Year to Date and Last Year!

MatissJanis commented 10 months ago

* [x]  loading indicators - I think we're missing these; for large budgets (such as mine) it takes ~40 seconds for the `split_by:payee` to load the data - and there is no loading indicator while this is happening.
Can you check this on #1930? I put in a fix but can't test it since my budget isn't that big...

Sadly it's still quite slow and doing these "glitches". Tricky to explain what I'm seeing visually.. if you added privacy mode for the Y axis - I could share a screen recording.

carkom commented 10 months ago

Sadly it's still quite slow and doing these "glitches". Tricky to explain what I'm seeing visually.. if you added privacy mode for the Y axis - I could share a screen recording.

Due to the way recharts handles YAxis tick labels I'm not sure I can use the "privacyFilter" implementation. I've used it everywhere else (tooltips, legends, etc) but the best I can do for Axis is replace the numbers with a static string variable. I chose '...' for now. Open to changing the string if there's a better or more consistant UI with how we do it elsewhere in the app. Cheers!

MatissJanis commented 10 months ago

Here you go:

https://github.com/actualbudget/actual/assets/886567/6cedf0be-1714-4cb2-a0e2-f9639673a2f4

Akellade commented 10 months ago

The control side bar isn't very reactive. There seems to be a max height. This is clear on the categories selection, where there is plenty of space below, but it introduces a scroll bar element :

Setting the overflow-y behaviour of the ul element from scroll to auto had the desired result for me.

carkom commented 10 months ago

Here you go: Screen.Recording.2023-11-21.at.08.12.28.mov

I've added a loading indicator. Still not sure what to do to help speed that up though. Payees loading can be quite slow for larger SQL files. Also, not sure how to cancel current data load when something new is clicked - currently it just waits for first one to complete then loads the next one.

Maybe make individual query files and call them via switch statement based on "mode" "groupBy" and "graphType" selections?

Kidglove57 commented 10 months ago

I have also asked for feedback on this in the Project Reports discussion.

My thoughts are then when the Expense view is selected in Reports this should show the net in each expense category, after any refunds. In turn, the Income view should only show Income categories.

At present receipts into an expense category are shown as Income.

Just wondering if this behaviour is intended? Thanks!

youngcw commented 9 months ago

At present receipts into an expense category are shown as Income.

@carkom Im looking at one of my categories that is specifically for reimbursements and it isn't acting like you described on discord. I can select expenses only and get what is expected, but net and income views don't show the money coming into the category. My guess is that this is due to the income all being in the form of off-budget to on-budget transfers. Those don't seem to count towards the table data.

carkom commented 9 months ago

@carkom Im looking at one of my categories that is specifically for reimbursements and it isn't acting like you described on discord.

Heya! I'm not sure which discord post you're referencing. I posted some suggestions on how it could work but wasn't saying that is how it currently does.

For further understanding of what/how data is displayed. By default the report hides off-budget, hidden categories/groups, empty ($0) categories, and uncategorized transactions. You can choose to show any/all of this data by ticking the corresponding checkbox in the sidebar.

I can select expenses only and get what is expected, but net and income views don't show the money coming into the category. My guess is that this is due to the income all being in the form of off-budget to on-budget transfers. Those don't seem to count towards the table data.

FYI, the app labels transfers as uncategorized transactions that have a "payee.transfer_acct" not equal to null. It shows these in the transactions list of the accounts pages but does not add them to any budget totals since they have no category.

If you'd like to see income for an account select the table graph and Time/Account/Net for the sidebar choices. You can easily see all the income for that account in the "assets" column and all the expenses in the "debts" column.

Kidglove57 commented 9 months ago

Just chipping in here but maybe I read the question wrongly? The off budget to on budget transfers seem to be key to this comment. They do not of course have a category on the off budget side of the transfer. However, on receipt into the on budget account that side would be categorised.

is @youngcw in fact saying that these categorised inflows are not appearing as income in the Custom Reports OR as a net off against the expense?

youngcw commented 9 months ago

@Kidglove57 that was what I was trying to get at. The transfers don't get counted as income even though they are categorized in that reimbursement category. That category has a balance of zero on the budget. If I set the custom table report to "net" I only see the expenses and nothing shows up under "income".

youngcw commented 9 months ago

It would be nice to be able to hide the side bar and top bar settings options, especially if viewing a saved report.

carkom commented 9 months ago

Just chipping in here but maybe I read the question wrongly? The off budget to on budget transfers seem to be key to this comment. They do not of course have a category on the off budget side of the transfer. However, on receipt into the on budget account that side would be categorised.

On the receipt side, where you say they have a category, what category would that be?

If you click on Payee and then "make transfer" it converts the transaction to a transfer and locks the category. You litterally can't set the category unless you remove the transfer flag. My contention is that it's not possible for a transfer to also have a category...

carkom commented 9 months ago

@Kidglove57 that was what I was trying to get at. The transfers don't get counted as income even though they are categorized in that reimbursement category. That category has a balance of zero on the budget. If I set the custom table report to "net" I only see the expenses and nothing shows up under "income".

Can you tick all these boxes and tell me if you see what you're expecting.

carkom commented 9 months ago

@youngcw @Kidglove57 I think what we are missing here is examples. Can either of you share an example file that I can play with? Feel free to adjust a demo file to match what you are seeing so you don't have to share anything personal.

youngcw commented 9 months ago

My contention is that it's not possible for a transfer to also have a category...

It is most certainly possible, I have lots of transfers with categories. Here is a demo budget that has an example. Look at the category Reimburse:Off-budget Reimburse. It has a charge then a transfer from off-budget. The budget table shows a balance of zero, but the data table shows only the expense even when selecting "net". 2023-11-27-_test-budget.zip

I tried selecting all the optional checkboxes and I still got the same issue

Kidglove57 commented 9 months ago

On the receipt side, where you say they have a category, what category would that be?

If you click on Payee and then "make transfer" it converts the transaction to a transfer and locks the category. You litterally can't set the category unless you remove the transfer flag. My contention is that it's not possible for a transfer to also have a category...

I wonder if we are at cross purposes here? If I make a transfer from an "Off Budget Account" into a "For Budget" account then Actual will insist that I enter a category but on that one side only - i.e. for the transaction side that is entering the "For Budget" account. I can choose whichever category is appropriate.

At the moment, the categorised receipt into the For Budget account is not reflected whatever box or dropdown is selected. I am guessing this must be because transfers are ignored completely - as indeed they should be between "For Budget" accounts. So it is just this very specific use case that falls through the cracks ie uncategorised transfer from Off Budget become categorised transfer on the For Budget side.

youngcw commented 9 months ago

~~I haven't checked any transfers going the other way, on-budget to off-budget. Those would need to be counted as expenses in the table.~~ Transfers that go from on-budget to off-budget also are not showing up right. My mortgage category is really off because part of it shows up as a transfer towards my off-budget loan balance.

carkom commented 9 months ago

My contention is that it's not possible for a transfer to also have a category...

It is most certainly possible, I have lots of transfers with categories. Here is a demo budget that has an example. Look at the category Reimburse:Off-budget Reimburse. It has a charge then a transfer from off-budget. The budget table shows a balance of zero, but the data table shows only the expense even when selecting "net".

My testing was from on-budget to on-budget accounts. I see that the on to off and off to on act differently. Thanks for bringing it to my attention. I'll have a look.

carkom commented 9 months ago

Thanks for getting me there gents. The test file helped. I see the issue now. Cheers!

carkom commented 9 months ago

Okay, does this meet expectations for your test file?

First transaction is Offbudget debit Second is credit to "Off-budget Reimburse" Third is debit to "Off-budget Reimburse"

Is this how you'd expect that to look?

youngcw commented 9 months ago

Is this how you'd expect that to look?

I would expect the table to report 0 when looking at net since there was a matched inflow/outflow. That screenshot is what I see now

Kidglove57 commented 9 months ago

Thanks for getting me there gents. The test file helped. I see the issue now. Cheers!

Totally understand and thanks for your patience - I'm sure there are many users who would not have this issue because they keep all their accounts on budget.

In the meantime I am greatly enjoying using the table view (to be honest this is the only one I will use as I am not very "visual" - prefer detail to a sweeping overview.)

Anyhow I just wanted to thank you again for the enormous amount of work you have gone through to bring two of my favourite features to Actual - this and Saved Filters!

carkom commented 9 months ago

Is this how you'd expect that to look?

I would expect the table to report 0 when looking at net since there was a matched inflow/outflow. That screenshot is what I see now

So if you look at the "total" column for net on the "off-budget reimburse" category. It is equal to 0.

I'll put in a PR for testing with your own budgets.

youngcw commented 9 months ago

When changing settings with large data tables, either to a large one or away from a large one, I get misaligned numbers. I can fix this by changing to a bar chart and back to a data table. It seems like the in code data table isn't updating properly since this should be a one month view.

carkom commented 9 months ago

When changing settings with large data tables, either to a large one or away from a large one, I get misaligned numbers. I can fix this by changing to a bar chart and back to a data table. It seems like the in code data table isn't updating properly since this should be a one month view.

Yep, I've already got it on my list. It's a rendering issue I need to fix. Thanks for the call out!

Teprifer commented 9 months ago

Bug: When viewing table data, hidden categories do not show anything in the chart area except totals and average when selected. For time series it shows the months, but still not the categories.

Teprifer commented 9 months ago

Issue: UI is unclear as to why no data will show after changing split by.

Example steps: 1) Split by category, untick all. 2) Change to split by account Result, no data shows, and no way to identify this is because no categories are selected (category list is hidden).

This may also apply to other variations of changing split-by.

After playing some more, I think the category list will may almost always need to be present, regardless of split by.

Small suggestion:

'Time' mode isn't disabled when looking at 'area' chart, but is incompatible with it. Suggest disabling the button when viewing an area chart and add tool tip advising this is why e.g. "Time mode is not available for this chart type."

Note the reverse is already true, when viewing time mode, the area chart option is disabled.

Likewise add similar text for when other buttons are disabled.

When using table view and scrolling to the right, the categories get hidden.

Suggestion: Pin the categories column to the screen when scrolling horizontal so they are always visible.

For visibility, enforce a minimum width for the categories column (gets a bit too squished with more data columns)

The same idea for the header row, pin in place so it's always present when scrolling vertically.

It doesn't make sense for Range to show any of the current options, if either the From or To month is manually changed.

Suggest range should contain a 'Custom' option, and changing either From or To month values triggers the Range to change to the "Custom" option.

Feature suggestion:

When in total mode, for bar and donut, maybe an 'order by' option? E.g. Most to Least (or large to Small) and the reverse, 'by group', 'by group, then large to small' and the reverse, 'by category order' (as in the order they are in the budget)

Minor suggestion: I found by accident clicking the bar chart icon again changes how the graph bars stack, suggest putting an explanation of what is changing in to the tool tip of the button, or elsewhere on screen.

Minor, by feel, show summary panel should be a remembered setting (ie, once toggled on, always on, even when leaving reports and coming back)

carkom commented 9 months ago

Bug: When viewing table data, hidden categories do not show anything in the chart area except totals and average when selected. For time series it shows the months, but still not the categories.

Use the tick boxes in the Display section in the sidebar to see this data. It is hidden by default.

Teprifer commented 9 months ago

Bug: When viewing table data, hidden categories do not show anything in the chart area except totals and average when selected. For time series it shows the months, but still not the categories.

Use the tick boxes in the Display section in the sidebar to see this data. It is hidden by default.

Ahh, so, your comment confused me as I couldn't see an option which reflected hidden categories, so I setup the same report again and played with those tick boxes before going "nah uh" :)

'Off Budget Items" seems to be the one you're referring to, I wouldn't consider "hidden" to mean "off budget" - they're still on budget, just not visible. Furthermore, prior to hiding them, they definitely were on budget, and their balances are reflected in the overall budget numbers. I thought that option was for 'off budget' accounts (since you can split by account).

If it's just for hidden categories, then I'd suggest renaming it to 'show hidden categories', if it's for both, I'd suggest renaming it to 'Show Hidden Categories and Off Budget Accounts", which is a lot of words I'll have to admit.

To address that I'd suggest separate tick boxes as they are different things.

p.s. I don't have off budget accounts, so not familiar with how they're interact with this custom reporting.

youngcw commented 9 months ago

The data table has started to not align the date header properly. The bottom totals do the same thing.

carkom commented 9 months ago

The data table has started to not align the date header properly. The bottom totals do the same thing.

I have a fix for this waiting in the wings. Also comes with improved scrolling behaviors and better row rendering. 😁

carkom commented 9 months ago

Ahh, so, your comment confused me as I couldn't see an option which reflected hidden categories, so I setup the same report again and played with those tick boxes before going "nah uh" :)

'Off Budget Items" seems to be the one you're referring to, I wouldn't consider "hidden" to mean "off budget" - they're still on budget, just not visible. Furthermore, prior to hiding them, they definitely were on budget, and their balances are reflected in the overall budget numbers. I thought that option was for 'off budget' accounts (since you can split by account).

If it's just for hidden categories, then I'd suggest renaming it to 'show hidden categories', if it's for both, I'd suggest renaming it to 'Show Hidden Categories and Off Budget Accounts", which is a lot of words I'll have to admit.

To address that I'd suggest separate tick boxes as they are different things.

p.s. I don't have off budget accounts, so not familiar with how they're interact with this custom reporting.

I can see the use case for 2 separate checkboxes. 😁

Teprifer commented 9 months ago

Sorry one more minor feature request, currently a category group is either selected, or not, I propose a third state for cases like this, where one or more, but not all, of the categories are selected:

Mock up:

HStep20 commented 8 months ago

I love these reports so much. Ive only ever missed Firefly III for its reporting capabilities, so Ive been super happy to explore these ones.

I did find a bug when it comes to the 'Amount' filter. When used to filter expenses, using the 'greater than' filter doesn't seem to show any actual data on the chart:

As it turns out, behind the scenes, it seems like the chart is counting expenses as Negative Dollars (which makes sense), even though it displays them as Positive (line go up).

In order to actually see data when the Expenses are selected as the reporting data, you have to invert the filter selection to look like: Less than -$150:

Im not sure how the data is retrieved via a filter, but I think that using the Absolute Value of the filtered data would solve this. Otherwise, it may be better to invert the chart for Expense reports so that they look like negative values when viewed.

As an aside, there is some weird behavior with the value filter. I figure its a separate Actual bug, but I definitely put in less thatn $150 as my filter, and its showing as $1.50. Ive seen similar things where Ill add a value filter for something like $100.00 and it will translate it to $10000

carkom commented 8 months ago

These are all known issues that are not unique to the custom report. It's a problem with the filter in every part of the app.

Check issue #1935 for neg/pos views.

We discussed the direction of the payments graph...yes they are negative values but ended up deciding that it's more difficult for the general user to interpret negative graphs and so opted to flip it.

For your "aside" issue have a look at https://github.com/actualbudget/actual/issues/1977

It's been fixed and closed but it sounds like there's still something similar happening for you?

carkom commented 8 months ago

I love these reports so much. Ive only ever missed Firefly III for its reporting capabilities, so Ive been super happy to explore these ones.

Thanks!

actualbudget / actual

[Feedback]: Custom Reports #1918