Only syncing when running a test event (AWS Serverless)

[Fckthesystem]

Hello,

Been using the AWS server less method for 6 months flawlessly, for which I'm grateful! Suddenly I stopped seeing the transactions in Ynab which normally were synced immediately. Revoked the codes and set up everything from scratch but no luck. It only syncs now when running a test event. Any advice on how to get it to work automatically again?

[nickstrijbos]

Hey @Fckthesystem I still need to fix the new template based on the latests releases from @wesselt. There is alsno a newer version available which I uploaded but that one still hangs partially. Also I am working on a Dockerized version which should also help with quicker setup time instead of relying on the cloud.

[Fckthesystem]

@nickstrijbos Thank you for your reply. So what do you suggest? Do I have to wait or is there something I can do? No clue why it suddenly doesn't automatically sync anymore.

[wesselt]

Hi, that may be the pagination problem. How does the solution from this issue work for you? https://github.com/wesselt/bunq2ynab/issues/53

[Fckthesystem]

Hi, that may be the pagination problem. How does the solution from this issue work for you? #53

Hi,

Thank you for replying. Just changed it, redeployed it and ran a test event. All my recent transactions got synced. After that I did a new transaction in Bunq to see if it works, but it doesn't come through in Ynab unfortunately.

[wesselt]

The immediate sync is a callback from bunq. So perhaps there's a problem with that. What do you see in the logs?

[Fckthesystem]

The immediate sync is a callback from bunq. So perhaps there's a problem with that. What do you see in the logs?

It is working now! The pagination issue was the culprit in the end. Only thing that changed compared to the last 6 months, is that the sync isn't instant anymore but takes a couple of hours at least. But as you said that could be because of Bunq.

Thank you for your help, it is much appreciated!👍

[wesselt]

Good to hear it's working!

When a mutation occurs bunq immediately sends a call to AWS. So sync should be immediate. What can you see in the logging?

[Fckthesystem]

logs-insights-results.xlsx Syncing takes easily 8+ hours. I've attached the log from the last day. I see some errors in the log but I've got no clue..

[wesselt]

Aha:

Exception: Use underscore instead of dash when getting config value callback-marker

Open lambda_function.py and replace the underscore with a dash as in this commit: https://github.com/wesselt/bunq2ynab/commit/4bc8af34fab5ca94ed5e6a223de20815236e4987

[Fckthesystem]

from this:

to this:

Is this what you meant?

[wesselt]

Hmm no, I mean to change the call to config, not the call to add_callback :) What happens when you change:

config.get("callback-marker")

To:

config.get("callback_marker")

As in this commit https://github.com/wesselt/bunq2ynab/commit/4bc8af34fab5ca94ed5e6a223de20815236e4987

[Fckthesystem]

Ok thanks, changed it as said. This doesn't resolve the delayed sync though. Very strange, that it suddenly went from instant to 8+ hours sync time.

[wesselt]

Yeah, it should be instant :) What do the logs show after you changed the config.get call?

[Fckthesystem]

No logs to be found of the last 6 hours, even though I used Bunq.

[wesselt]

Yeah, it looks like it fails to set up instant sync with bunq. That happens at the start and hopefully every day (not sure how the AWS part works.)

What do the logs say right when you start the app?

[Fckthesystem]

logs-insights-results (2).xlsx Here are the logs after a test event I just ran. Still an error that keeps occuring everytime. I don't think that the error is blocking the instant sync though..

[wesselt]

@Fckthesystem There's an exception in add_callback(). That's the function that tries to set up instant sync with bunq.

The exception is Exception: URL should end with end-of-url marker. It looks like you have config.get("callback-marker") instead of config.get("callback_marker")at line 19 in lambda_function.py. What happens when you change the dash to an underscore?

[Fckthesystem]

I changed that a while ago, as you advised. While looking at the code I got this message:

Not sure what happened. Was this you?

Here is the code as it's been for the past couple of weeks. Maybe you notice something out of place:

[wesselt]

Thanks for your feedback! Wasn't me, only you can edit code in your own Amazon Lambda environment. Not sure why you got that message.

Hmmm.... what happens when you replace or "bunqynab_autosync" with or "bunqynab_lambda"? That worked for someone here: https://github.com/wesselt/bunq2ynab/issues/49

[Fckthesystem]

Thanks, Just edited it and ran a test event. No errors this time.

Still no instant sync though :(

[wesselt]

What logs are there around the time you do a transaction in bunq? (Pay 1 cent from one account to another, it's ok if only one of them is synched.)

[Fckthesystem]

@wesselt There are no logs to be found right after doing the transaction.. Only when running a test event or waiting for hours it will show up in the logs and in Ynab.

[wesselt]

@Fckthesystem I wonder what port you're using. What does the URL of your lambda look like? Does it contain a port number, like https://server:port/a/b/c ?

[ChrisMisker]

Hi! I'm using version 1.4.4 (?) on the AWS version, and when I do the test, I get this error:

2024-05-30 15:11:32,342 | INFO | lambda_function.py:13 | add_callbacks | SSM callback = "https://iza606sxw2.execute-api.eu-west-3.amazonaws.com/bunq2ynab-lambda"
2024-05-30 15:11:32,342 | INFO | lambda_function.py:16 | add_callbacks | Adding callbacks...
2024-05-30 15:11:32,342 | ERROR | lambda_function.py:62 | lambda_handler | Exception occurred
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 56, in lambda_handler
add_callbacks(sync)
File "/var/task/lambda_function.py", line 18, in add_callbacks
callback_marker = config.get("callback-marker") or "bunq2ynab_autosync"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/task/lib/config.py", line 96, in get
raise Exception(f"Use underscore instead of dash when "
Exception: Use underscore instead of dash when getting config value callback-marker

How can I fix this?

[wesselt]

@ChrisMisker what do you think of the fix here https://github.com/wesselt/bunq2ynab/issues/49#issuecomment-1963491305

[ChrisMisker]

Thanks @wesselt , I fixed it. I changed the code of lambda_functions.py, line 18 to: callback_marker = config.get("callback_marker") or "bunq2ynab_autosync"

and added

,
  "callback_marker": "bunq2ynab-lambda"

to the configuration. Now it works.