Flag identical transactions

[katefike]

Problems

• What if the bank sends the same TXN email twice?
• What if I forward the same TXN twice?
• It's impossible to tell if some TXNs are duplicates or I just multiple of the same thing on the same day. For example, there are multiple Uber charges for -0.51 on the same day - how many are actually on my statement?

Solution

For a given day and bank, check if there are other TXNs with the same merchant/payer and raw amount

• date
• bank
• account (if applicable)
• merchant/payer
• raw amount

Pytest

1. Load a named mbox containing two or more emails - the emails may or may not be duplicates

DONE WHEN

• For the same TXN, load a forward from gmail and a forward from CloudHQ, ensure the second TXN is flagged with the txn ID of the first (file="duplicate_txns_gmail+cloudHQ_forwards.mbox")
• For the same TXN, load 2 gmail forwards (file="identical_txns_2_gmail_forwards.mbox")
  • The 2 gmails mbox (identical_txns_2_gmail_forwards.mbox) had mysterious mbox loading errors. However, identical_txns_gmail+cloudHQ_forwards.mbox loads successfully. One way the 2 Gmail forwards mbox differs from Gmail+cloudHQ mbox is that Gmail is nesting the 2 gmail forwards. Maybe Gmail interprets the 2 forwards as one email, and mb2md doesn't know how to interpret that into /Maildir? I.e. gmail-specifc mbox logic that chokes mb2md.

[katefike]

Troubleshooting Pytest

root@9754f7b4e7a9:/# mb2md -s test_data/example_data/unparsable_emails.mbox -d /home/incoming/Maildir/
Fatal: Source is not an mbox file or a directory!

Error appears when refresh_inbox() attempts to load the mbox file. Error is caused by not using an absolute or fully qualified path. Solution is to change path to ./test_data/example_data/

[katefike]

Troubleshooting Pytest

root@9754f7b4e7a9:/# mb2md -s test_data/example_data/unparsable_emails.mbox -d /home/incoming/Maildir/
Fatal: Source is not an mbox file or a directory!

Error appears when refresh_inbox() attempts to load the mbox file. Error is caused by not using an absolute or fully qualified path. Solution is to change path to ./test_data/example_data/

Refactored refresh_inbox to use a lot of error catching but something was going wrong with this style of subprocess calling. Reverted back to original use of subprocess.call without

    _delete_maildir_success, delete_maildir_output = call_subprocess_with_output(
        f"{container} rm -r {maildir_path}"
    )
    print(f"INFO: Deleted existing Maildir/, if any: {delete_maildir_output}")

    recreate_maildir_success, recreate_maildir_output = call_subprocess_with_output(
        f"{container} mkdir {maildir_path}"
    )
    if recreate_maildir_success is False:
        print(f"CRITICAL: Failed to recreate Maildir/: {recreate_maildir_output}")

    load_mbox_success, load_mbox_output = call_subprocess_with_output(
        f"{container} mb2md -s {mbox_path}/{mbox_name} -d {maildir_path}"
    )
    if load_mbox_success is False:
        print(f"CRITICAL: Failed to load mbox: {load_mbox_output}")

    (
        modify_maildir_permissions_success,
        modify_maildir_permissions_output,
    ) = call_subprocess_with_output(
        f"{container} mb2md -s {mbox_path}/{mbox_name} -d {maildir_path}"
    )
    if modify_maildir_permissions_success is False:
        print(
            f"CRITICAL: Failed to modify Maildir/ permissions: {modify_maildir_permissions_output}"
        )

def call_subprocess_with_output(command):
    success = False
    try:
        output = subprocess.check_output(command, stderr=subprocess.STDOUT).decode()
        success = True
    except subprocess.CalledProcessError as e:
        output = e.output.decode()
    except Exception as e:
        # check_call can raise other exceptions, such as FileNotFoundError
        output = str(e)
    return (success, output)