Headless usage not working

chises commented 1 year ago

Hello,

I am sorry but I am not able to use the headless feature. wget -O - -q http://172.17.1.11:8083/?automate=true\&config=dkb.json or curl -X GET 'http://172.17.1.11:8083/?automate=true&config=dkb.json' Output:

<html>
<head>
    <title>Choose Tan Device</title>
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css"
          integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh" crossorigin="anonymous">
</head>
<body>
<div id="content" class="container">
        <h1>Choose a device for your tan method.</h1>

            <p>There is only one tan device available.</p>

            <p>Automatically continuing to next step in 2 seconds.</p>
    <script>
        window.onload = function() {
            setTimeout(function() {
                document.forms["next-form"].submit();
            }, 2000);
        }
    </script>

    <form name="next-form" action="." method="post">
        <input type="hidden" name="step" value="STEP2_LOGIN">
        <div class="form-group">
            <label for="bank_2fa_device">Tan Device</label>
            <select class="form-control" name="bank_2fa_device" id="bank_2fa_device" required>
                                    <option value="cHiNokia8">
                        cHiNokia8
                    </option>
                            </select>
        </div>

        <button type="submit" class="btn btn-primary">Collect statements from your bank</button>
    </form>
</div>
</body>
</html>

Any hint what is wrong here? :)

chises commented 1 year ago

@Bur0k can u assist here?

Bur0k commented 1 year ago

Seems i never actually tested it with curl or wget - only in a browser.

I guess this is because the bank wants you to verify youself through a TAN. In my browser experiments I verified through TAN of course some time ago and a cookie was set in my browser. When i now try to open http://172.17.1.11:8083/?automate=true\&config=dkb.jso in the browser this cookie will be sent as well.

Locally i couldn't get it working with curl aswell and then tested my theory about the cookie. I looked up the PHPSESSID cookie in my browser and than ran the following: curl --cookie 'PHPSESSID=MyPhpSessId' http://172.17.1.11:8083/?automate=true\&config=dkb.jso

This then finally works.

So we need to implement easy way to authenticate ourselves via TAN and then get the cookie. Then we can use that with curl to do the querying headless. This also ties into the email notification issue I created some time ago https://github.com/bnw/firefly-iii-fints-importer/issues/75

All in all the headless mode somehow is still hackish - I hope I'll have time soon to think about implementing a better way. For now its retrieving the cookie manually. Sorry :/

bnw commented 1 year ago

I guess this is because the bank wants you to verify youself through a TAN. In my browser experiments I verified through TAN of course some time ago and a cookie was set in my browser. When i now try to open http://172.17.1.11:8083/?automate=true\&config=dkb.jso in the browser this cookie will be sent as well.

@Bur0k I think your analysis is not 100% correct:

Your bank will occasionally issue 2FA challenges (e.g. TAN). There is currently no way to answer them through curl, you have to click through the browser forms instead.
Once completed, banks typically do not issue another 2FA challenge for several months. During this time, it is possible to use the curl method. The fact that no re-authentication is necessary is, I believe, not based on anything stored in the session, but only based on the product key (https://github.com/bnw/firefly-iii-fints-importer/blob/master/app/FinTsFactory.php#L23) (a rather insecure design choice, imo).
The problem in @chises case is that the 2FA device was not chosen. The API requires it, even though no 2FA challenge needed to be solved.

-> I have added the possibility to also configure the 2FA device via the config (see the current config file example). @chises Add this, and the error should be fixed. :)

Bur0k commented 1 year ago

@Bur0k I think your analysis is not 100% correct:

Indeed it is incorrect. I think i made the simplest mistake and forgot the quotes around the url and never looked deep into the output.

-> I have added the possibility to also configure the 2FA device via the config (see the current config file example). @chises Add this, and the error should be fixed. :)

Thanks for the analysis and quick fix :)

Your bank will occasionally issue 2FA challenges (e.g. TAN). There is currently no way to answer them through curl, you have to click through the browser forms instead

I should really start working on https://github.com/bnw/firefly-iii-fints-importer/issues/75 in order to get an email notification at least

chises commented 1 year ago

@Bur0k Thank you very much, I will test it today!

Update: seeems to work. Addes a cronjob and will check it tomorrow :) Thank you!

Update 2: It is working, transactions are imported via cron with curl. 👍

bnw / firefly-iii-fints-importer

Headless usage not working #108