Button in the popup to export any time-based trend

[idpaterson]

This update ports over the "Export Daily History to CSV" button from my userscript now leveraging the API framework of Monarch's extension along with several other enhancements. Since inserting the export button into the page did not work out this pull request accomplishes the same with a "Download current trend daily balances" button in the popup. No content script so none of the insurmountable issues that derailed #23.

The button is enabled when viewing an exportable trend like Net Income Over Time or Spending Over Time for 5 selected accounts.

     

Export any time-based trend

The fetch functions have been generalized to work for any time-based trend which is pretty much just passing in a report type and a list of accounts that become filters. When working with trend state the extension fetches the daily balances for the exact time period you're looking at – no optimization to avoid fetching zero months. I had it in the original userscript but since the popup is such a nicer way to export all accounts I think the use cases for trend exporting would not demand that kind of optimization. Just use a custom date range if you're concerned about loading too much data 😉

The formatBalancesAsCSV function now supports the oddball net worth and net income trends that include extra columns. zipTrendEntries() handles the API response for those trends since they can contain two TrendEntry rows per date. Same trimming of zero values at the end of the CSV for all trend types.

Making API requests correspond to Mint trend state

The current trend state is sourced from a Redux store on a component in the Mint Trends page. It provides all of the data necessary to reproduce the same API queries that Mint used to chart the trend.

This Redux store was difficult to find and the original implementation used a different data source that was convoluted and unreliable. Original discussion is no longer relevant, but preserved below:

Okay, so... localStorage.trends-state-ASSETS and friends track the current state of the user's selections on the Trend page. As a user I select the accounts that I want to see but localStorage.trends-state-* lists only deselectedAccounts. The concept of "all accounts" differs based on the report type so it's inconvenient that we only know which are deselected.

Weird, but no problem since the trends API seems to support affirmative and negative filters [{ matchAll: true, filters: [...] }, { matchAll: false, filters: [...] }]. Except, taking that deselectedAccounts value and plugging it in to the matchAll: false filters results in a 400 bad request. I tried everything I could think of but was not able to figure out if the API supports filtering based on the deselected accounts.

So, getAccountTypeFilterForTrend() was born. We now have a mapping ACCOUNT_CATEGORY_BY_ACCOUNT_TYPE of all account types (from an enum in the Mint source) to category (asset or debt). That seems to work for most of the report types to determine which accounts are eligible but NET_INCOME is a special little darling.

I found some code in Mint that maps each trend report type to the specific account types that are automatically disabled. The code can be found by searching for accountsToFilter, it's in the trends ui library.

The CashAccount and InsuranceAccount types are always excluded if the trend is filtered by account. If "All accounts" are selected then these are both included (if the trends backend includes them, it at least does for NET_INCOME). I assume InsuranceAccount is a legacy thing, I do not have one and was unable to add one.

To test I compared the daily balances table shown for "Last Month" trends to the daily CSV. I exported the "All accounts" data and compared the numbers, then deselected one old account that has no transactions in the last month and compared those. Mint produces different numbers for these two cases not because my old savings account had transactions in the last month, but because CashAccount and InsuranceAccount are excluded only if some other account is excluded as described above.

Finally, the daily export button has been disabled on trends filtered by tag, category, or merchant. Those filters are not represented in localStorage.trend-state-* and there's not much value in trying to work something out. If anyone needs that functionality for some reason, the userscript can do it since it just repeats the last trend API request. We do not have access to that data since it required code modification that made the userscript very difficult to set up.

[MilesBHuff]

Can you please provide a simple summary of why and where we need to be clicking this new button? It's easy to get lost in all the details in your OP.

[idpaterson]

Thanks for following up on this, it looks like Mint has pushed the sunset date so this might still have a chance to help people.

This adds a third button to the extension popup that should be visible when you click the extension icon. The button is enabled if you're viewing a Mint trend page and when clicked exports daily balances for that trend over the time period shown in Mint. Unlike the existing per-account export it allows combining accounts to export trends like "all student loan payoffs" or net worth.

When I was using this back in November to export very old trends I noticed that the Mint API took a lot longer to respond (and often timed out with no data at all) for old data. Severe rate limiting (like 1 request per 5 seconds) had no effect, it seemed to be a general Mint API issue. The Mint UI similarly failed to load trends most of the time.

If this is still an issue, a possible improvement would be to fetch data in the reverse order, collecting daily balances for the most recent transactions first moving back in time until the API gives up. A partial export could be more useful than no data at all.

Honestly though I don't know if anyone other than me finds this useful. I considered asking people on the subreddit but assumed that the Monarch dev team would not have time to review this one anyway.

[MilesBHuff]

Thank you for the quick follow-up and notes. I'll definitely what you've said about old trends in-mind when I export my Mint data near the end of December.

When I'm ready to export, I plan to combine the active PRs into my own fork and then use that instead of the current version of the plugin. I'm hoping (likely in vain) that Monarch addresses the open issues and PRs before then.

So there is at least one person who isn't you who finds what you've done useful. Thank you for taking the time to share your work.

[idpaterson]

Thanks! I assumed you were from Monarch 😄

I'll do some more testing on this to see if it can be improved and to see if the Mint API is still in a woeful state.

[idpaterson]

Today's update addresses issues I was having with pulling the current trend state from localStorage. Mint tracks the trend there for the purposes of resuming it if you refresh the page, but it is extremely unreliable. Occasionally accounts get unset, dates change, and it is even more unpredictable if you have multiple tabs open. It often did not accurately reflect the UI state unless you had just recently configured the trend and you can see this in Mint by refreshing the trend page and occasionally ending up at a different trend.

Now after excessive digging around in the DOM to find the pieces that I needed, then discovering a Redux store with all of the data in one place, the extension instead pulls the current trend state from React. This works with multiple browser tabs open and avoids the debacle of transforming deselectedAccounts into the selected accounts.

This does not yet support category, tag, and merchant filters available on the Income and Spending trends. I have started working on that because I have several tags that I like to track with annual trends. I can just get the data from Monarch now but hopefully it will be helpful to others.

Oh also the Mint API is speedy again, at least for me. No rate limiting issues so far testing this out.

[idpaterson]

Earlier I had interpreted a response with no Trend array as a timeout, but after adding support for exporting trends filtered by tag or category I noticed that Trend is always omitted when every value in the time period is zero. I was exporting some very infrequent tags that had large gaps between old and new data. The old data loaded fine but the middle gap always errored out. Since there are no dates in the response the extension just generates the rows with zero amounts.

This was great news since my past experience with API issues may have just been due to exporting gappy data.

[idpaterson]

Here are some examples of things you can export with the "Download current trend daily balances" button:

1. Net worth
2. Credit card spending (e.g. select "All credit cards" in the Spending trend)
3. All investments (e.g. select "All investments" in the account dropdown on the Assets trend)
4. Spending by category (e.g. select "Category: Groceries" under "Show the transactions that match" on the Spending trend)
5. Income by merchant (e.g. select "Merchant: FizzBuzz Ltd" on the Income trend)
6. Spending by tag and merchant (e.g. select "Tag: My Tag" and "Merchant: Target" under "Show the transactions that match" on the Spending trend)
7. Single account balances, useful after fixing a problem seen in the all accounts export (e.g. select "My Bank Account" on the Assets trend)
8. Net income (though why would you want daily net income?)
9. Any trend you want, as long as one of the "over time" options is selected in the trends sidebar.

[oaosman84]

@idpaterson we're really sorry we're so behind on reviewing things here.. one thing that could help, could we split out any code in here related to rate-limiting of the existing functionality from the new functionality into separate PRs?

[idpaterson]

Good idea, at the time it was the only way I could make progress on this branch but it's a good idea to separate it. I'll check for any other standalone bug fixes, too.

[idpaterson]

Thanks for reviewing the small ones, I’ll rebase this PR when I get a chance!

[vanessa]

@idpaterson Is this PR ready for review? Thanks!

[idpaterson]

Yep it is up to date with main and ready to review.

[MilesBHuff]

Any updates on this?

[idpaterson]

It’s not going to happen, Monarch definitely has more important work to do and we’re closing in on the Mint shutdown date.

[MilesBHuff]

It’s not going to happen

Seems you were right. :\ Thank you for trying to get this merged, regardless.

[idpaterson]

You’re welcome! Fortunately most of the improvements along the way to the trend button made it in via separate pull requests.

Monarch has been very nice to work with, they reached out recently offering to add more code review time for contributions to this extension. However, with the shutdown looming I decided it was not worth taking the risk of pushing new code. If an undetected mistake were introduced in the past couple weeks it could have left a lot of people with imperfect CSVs.

Thanks for following up, I hope your migration went well