Handle $0 "extra comment" transactions

[bdrister]

Frequently, "extra comments" on a transaction will appear in the OFX stream as extra $0 transactions following the transaction to which they refer.

<STMTTRN>
  <TRNTYPE>CREDIT
  <DTPOSTED>20161027160000.000
  <TRNAMT>123.45
  <FITID>FITID20161027123.45ABCDE
  <NAME>AMAZON MKTPLACE PMTS AMZN.COM/BI
</STMTTRN>
<STMTTRN>
  <TRNTYPE>DEBIT
  <DTPOSTED>20161027160000.000
  <TRNAMT>-0
  <FITID>FITID20161027-0.0ABCDE
  <NAME>XXXEXTRASTUFFXXX
</STMTTRN>

Most financial software handles this transparently, using the second "transaction" as a comment on the first one. The ledger transactions produced by ledger-autosync include them as-is, however.

I've attempted to implement this functionality on my end, but ran into the problem that it makes ledger-autosync's deduping not work, as if there's more than one ofxid in the notes, it uses only the last one, so either the main transaction or the extra comment transaction will fail to be deduped.

Ideally, ledger-autosync would abstract away this annoyance of the OFX/financial system and do the merging automatically. As a fallback, recognizing more than one ofxid would allow me to solve it on my end. Were that to also not be feasible, ledger-autosync stopping once it first encountered a transaction already in the ledger would at least bound the damage to {0,1} extra transactions per import, instead of the potentially dozens currently.

[egh]

Do you think that it would be sufficient to filter all transactions with a TRNAMT = -0? That would be simple. A little more difficult would be filtering all transactions with TRNAMT = -0 and a date equal to another transaction.

[bdrister]

I've filtered them out on my end at the moment; that's straightforward. It's trying to actually maintain that extra data (merging the comment into the main transaction) that's not currently possible without reimplementing deduping.

[egh]

Ah, I see. I think we can do that. But how do you think the extra data should be merged in? I can't find any examples of files like this except what you have provided. Should it be appended to the payee?

[bdrister]

Interesting. I think at least two of my banks do this, so I'd assumed it was fairly commonplace.

I think the minimum necessary change here is deduping supporting multiple ofxid tags on a transaction, since that's necessary no matter who's doing the merging. (Hopefully that's just a current ledger-autosync limitation, and not a fundamental thing with the ledger format!) Once that exists, if ledger-autosync was going to do the merging then the content makes the most sense as a ; comment on the transaction. But if this is so rare that it's not worth doing in ledger-autosync, I can do it on my end as long as I can avoid breaking deduping when I do so.

[egh]

I think it might be possible to work around the dedup issue by ignoring transactions that follow another transaction with the same timestamp and a TRNAMT = 0, if that is sufficient to identify the transaction as an "extra comment" transaction.

[bdrister]

I don't have any 0-amount transactions that aren't this kind of comment, so this solution is functionally identical (for my particular data set) to the current situation of just discarding all 0-amount transactions on my end.

[egh]

Great! I think this is possible. I'll look into it.

[egh]

I have added some basic support for suppressing these transactions in https://github.com/egh/ledger-autosync/commit/d687e9c5787d5aaecb81fa445bd643add86ade93. It should be possible to something more sophisticated with them, but I'm not sure what. Because the process that converts from an OFX transaction to a ledger entry happens on a per-transaction basis, it would be a bit hard to merge the entries without complicating the code.

Thanks for your patience!

[egh]

@bdrister I'm going to close this issue for now. Let me know if there is anything else I can help with for this.