Support CSV exports without Balance amounts

[siddhantgoel]

Sometimes the Balance (Saldo) amount is missing from the CSV exports, but the ECImporter expects it to be there.

The "Saldo" metadata entry should be made optional.

[szabootibor]

Balance (Saldo) is optional in the exported CVS; the user can omit it by this checkbox: I have noticed however, that a beancount balance assertion is not emitted by the importer, even if balance is present in the input. Do you wish a fix for this?

[siddhantgoel]

Yep, if you could that would be great! It's probably also connected to the fixed META_KEYS length check we do (I actually can't recall why I implemented it this way).

Would be cool if we could make the "file read" independent of the number of metadata entries present. Such that when we read the file, we always read a bunch of lines until the next empty line and process segments one by one (first segment is considered the header, the second one containing the meta entries, and the last one containing the actual transactional data). This way we wouldn't have to have any fixed number of meta entries to process and might be a bit more flexible when reading/processing the file.

[szabootibor]

My issue is a bit different, shall I probably open a new one?

The ECImporter reads Saldo, though. But this value is never used, i.e., a balance assertion is not emitted.

However, the usage of the value Saldo is not straightforward, because Saldo apparently represents the balance of the account at the moment the CSV is generated, and not at the closing date of Zeitraum metadata. If a date filter with different end date is used, transactions that contribute to the balance but missing from the CSV might exist. Example: I make an export on April 15 of the transactions made in March. Consequently, using the value of metadata Saldo for the balance assertion with the closing date of Zeitraum metadata, Beancount will indicate a balance error until the transactions between April 1 and 15 are imported. Even if no different end date is selected on the CSV generation, some transactions may be booked by the bank on the same day later, and these transactions will make the exported balance later invalid.

So I think the Saldo metadata entry is not especially useful anyway.

However, each transaction has a Saldo column. We could use it instead of the Saldo metadata, for example to make an assertion for the closing balance: take the value of Saldo colum of the last transaction in the CSV, and emit a balance assertion with date of the next day (Beancount interpretes balance as initial balance on the specified day).

We have 2 problems with this approach:

1. the user may select a sorting for the CSV other than ascending by date; in this case we have to determine the last transaction, which is not straightforward by multiple transactions on the last day
2. similarly as outlined above, additional transactions may be booked on the same day after generating the CSV, that could make the balance assertion invalid later.

I think we can solve problem 1.. problem 2. remains responsibility of the user.

[szabootibor]

I have made some thoughts about this issue and I propose the following solution:

1. The value of Saldo meta shall be still ignored
2. The value of Saldo column of the last transaction together with the closing date of Zeitraum meta plus one day shall be used to output a closing balance assertion.
3. An opening balance assertion could be produced as follows: balance is Saldo minus Betrag from the first transaction together with opening date of Zeitraum meta.
4. The balance assertion shall only be produced if there is at least one transaction and the order of transactions selected by the user on CSV export is either ascending or descending by date (Datum aufsteigend or Datum absteigend). In other cases the importer may emit an info message to stderr clarifying why no balance assertion is produced.

I think this would keep the importer simple at the cost of a small limitation of the balance assertion functionality.

[siddhantgoel]

Oh yeah, good catch. Your proposal sounds good :+1: with one slight modification: we probably don't need to rely on the user sorting. We could just go through all the transactions and pick the first/last transactions based on the Buchung value. Each such transaction has a Saldo value associated with it which should ideally be the same for all transactions booked on the same day.

Thanks for taking this up! Let's continue the discussion on the PR you opened.

[siddhantgoel]

Solved as part of https://github.com/siddhantgoel/beancount-ing-diba/pull/74 .