Line break within transaction, colon right after newline

railslove / cmxl

your friendly MT940 SWIFT file parser for bank statements

http://railslove.com

MIT License

46 stars 25 forks source link

Line break within transaction, colon right after newline #30

Closed mkilling closed 5 years ago

mkilling commented 5 years ago

Hi, I’m currently having problems with Deutsche Bank. The error message I’m getting is:

Cmxl::Field::LineFormatError: Wrong line format: ":08 Karten?25nr. 5355999999999975  Origi?26nal 49,00 USD 1 EUR/1,\r\n12385?27 USD  Entgelt 0,44 EUR?30DEUTDEDBFRA?31DE1950070024000402\r\n0480?32DEUTSCHE BANK"

The transaction that’s causing the issue is:

:61:190425D44,04NMSCNONREF
:86:106?109075/658?20EREF+000000000193592204?21MREF+CN3R3U?22CRED+DE7
600200000132558?23SVWZ+STARTER//8449273399/US?24 22-04-2019T03:46
:08 Karten?25nr. 5355999999999975  Origi?26nal 49,00 USD 1 EUR/1,
12385?27 USD  Entgelt 0,44 EUR?30DEUTDEDBFRA?31DE1950070024000402
0480?32DEUTSCHE BANK

Note the linebreak and the :08 right after the newline, which does not indicate a new section but is just a part of the transaction details that has been inconveniently split.

Unfortunately I don’t know right now if this is a configuration issue from my side or a case that Cmxl currently cannot handle...

bumi commented 5 years ago

Hey, thanks for reporting this. that sounds like an interesting edge case... The current check does look for linebreaks and lines starting with a :.
https://github.com/railslove/cmxl/blob/master/lib/cmxl/statement.rb#L32

This should probably be changed to use a regex that tests for actual MT940 tags. (something like /:\d{2}: ? Do you want to make a PR for this?

misterfinster commented 5 years ago

I see this as well:

/usr/local/bundle/gems/cmxl-1.4.0/lib/cmxl/field.rb:56:in `parse': Wrong line format: ":48  Folge?25nr. 007  Verfalld. 1222  Or?26iginal 300,00 CHF 1 EU\r\nR/1,1?274259 CHF  Entgelt 5,99 EUR?30XXXXXXXXXXX?31DE999999999999\r\n99999999?32Deutsche Bank AG" (Cmxl::Field::LineFormatError)

with carriage returns mid-field it seems?

bumi commented 5 years ago

@misterfinster yes this looks similar and seems to be also a statement from Deutsche Bank. thanks for reporting it.

can anybody help with the potential fix mentioned above? => optimizing the splitting of every field and making sure we do not have line breaks in the data or making sure we support all these line breaks.

mkilling commented 5 years ago

Thanks everybody for weighing in on this. I think /:\d{2}:would definitely be an improvement but it would not solve the issue fully.

Maybe I’m naive, but: Is there anything in the format spec about handling cases like these?

misterfinster commented 5 years ago

I just tested this commit in combination with the consuming libraries and it has exactly fixed my issue.

117agu commented 5 years ago

I just tested this commit (only changed statement.rb) and it fixed my issue with comdirect. schurig/ynab-bank-importer#43

bumi commented 5 years ago

ok, that is awesome! thanks for the feedback!

it is great that we have a test in the PR and all the other tests are green, too. @Uepsilon can you review please? what are your thoughts on the changed line parsing? #31