auto_sync.py not working

Hammerpawn commented 2 years ago

Hey, in the past I ran bunq2ynab through AWS Lambda, but I'm trying to set it up on my Raspberry Pi. The bunq2ynab.py file works fine, but for some reason the auto_sync.py is throwing an error and doesn't work. These are the logs I get:

2022-02-12 18:28:18,625 | INFO | auto_sync.py:88 | setup_callback | Host has a private IP, trying upnp port mapping... 2022-02-12 18:28:18,626 | INFO | network.py:80 | portmap_search | Searching for upnp gateway... 2022-02-12 18:28:27,739 | INFO | network.py:38 | get_public_ip | Retrieving public IP from http://ip.42.pl/raw... 2022-02-12 18:28:27,853 | INFO | auto_sync.py:100 | setup_callback | Listening on port 64540... 2022-02-12 18:28:27,854 | INFO | network.py:105 | portmap_add | Adding upnp port mapping... 2022-02-12 18:28:32,882 | ERROR | network.py:113 | portmap_add | Failed to map port: UnknownError 2022-02-12 18:28:32,883 | ERROR | auto_sync.py:108 | setup_callback | Failed to map port, not registering callback. 2022-02-12 18:28:32,884 | INFO | auto_sync.py:171 | | Starting periodic synchronization... 2022-02-12 18:28:32,884 | INFO | auto_sync.py:40 | synchronize | Starting sync at 2022-02-12 18:28:32 2022-02-12 18:28:32,885 | ERROR | auto_sync.py:45 | synchronize | Error during synching: Synchronize called before populate 2022-02-12 18:28:32,887 | ERROR | auto_sync.py:46 | synchronize | Traceback (most recent call last): File "/home/pi/development/bunq2ynab/auto_sync.py", line 41, in synchronize sync_obj.synchronize() File "/home/pi/development/bunq2ynab/lib/sync.py", line 203, in synchronize raise Exception("Synchronize called before populate") Exception: Synchronize called before populate

2022-02-12 18:28:32,888 | WARNING | auto_sync.py:177 | | No callback, waiting for 60 minutes...

Any idea what's going on here and how I can fix it?

Hammerpawn commented 2 years ago

Never mind, reinstalling miniupnpc did the trick for some reason.

Hammerpawn commented 2 years ago

Okay, apparently it didn't fix the error. I did get it working, but 9 out of 10 times I'm still getting this error.

wesselt commented 2 years ago

It looks like opening a port using UPNP fails, and then it fails entirely. I've pushed an update that synchs once an hour if port setup fails. How does the latest version work?

Failed to map port: UnknownError

You could also investigate the web interface of the device that provides internet connectivity. Check if there are any UPNP related settings.

beninhos commented 1 year ago

I'am encountering the same. It fails to map a port via UPNP (tries a bunch and then goes into 60-minute intervals. I have it enabled on the router, and it has worked in the past if i remember correct. I'am on Amplifi HD with the latest firmware.

The service bunq2ynab is running from Ubuntu 22.04

wesselt commented 1 year ago

What do your logs show? How do other programs that use upnp portmap work? (bittorrent clients like qbittorrent or transmission)

beninhos commented 1 year ago

I do not have any other programs that use UPnP port mapping. However, I have manually mapped some ports on my router for a Plex Docker instance

Log's below.

systemd[1]: Started Autosync BUNQ2YNAB script.
python3[862479]: INFO | sync.py:60 | populate | Retrieving bunq accounts...
python3[862479]: INFO | bunq.py:102 | get_session_token | Requesting session token...
python3[862479]: INFO | sync.py:62 | populate | Retrieving ynab accounts...
python3[862479]: INFO | auto_sync.py:87 | setup_callback | Host has a private IP, trying upnp port mapping...
python3[862479]: INFO | network.py:80 | portmap_search | Searching for upnp gateway...
python3[862479]: INFO | network.py:93 | get_portmap_external_ip | Retrieved external IP xx.xxx.xx.xxx from upnp gateway...
python3[862479]: INFO | auto_sync.py:99 | setup_callback | Listening on port 60464...
python3[862479]: INFO | network.py:105 | portmap_add | Adding upnp port mapping...
python3[862479]: INFO | network.py:115 | portmap_add | Port 60464 is already mapped, trying next port...
python3[862479]: INFO | network.py:115 | portmap_add | Port 60465 is already mapped, trying next port...
python3[862479]: INFO | network.py:115 | portmap_add | Port 60466 is already mapped, trying next port...
python3[862479]: INFO | network.py:115 | portmap_add | Port 60467 is already mapped, trying next port...
python3[862479]: INFO | network.py:115 | portmap_add | Port 60468 is already mapped, trying next port...
python3[862479]: INFO | network.py:115 | portmap_add | Port 60469 is already mapped, trying next port...
python3[862479]: | WARNING | auto_sync.py:185 | <module> | No callback, waiting for 60 minutes..

wesselt commented 1 year ago

The message "is already mapped" means the router told the program it was unable to map the port because of a "ConflictInMappingEntry". This usually means another program has already port mapped 60564 to 60469. The bunq2ynab program will start at a random port, so it's curious if this always happens.

I wonder if another host is really claiming all these ports. What does your router say about portmapping? Or is the router configured to forward all internet traffic to one particular host?

beninhos commented 1 year ago

Yes, it has been running before and probably mapped those ports on the host. My router does specify to where the traffics need to go.

When i do sudo lsof -i i see a bunch of ports opend towards my latop. dunno why they are like that.

snippet: python3 2566 root 115u IPv4 1044151 0t0 TCP jaguar:21064->10.0.1.51:51640 (ESTABLISHED) and python3 862479 beninho 3u IPv4 3266826 0t0 TCP *:60464 (LISTEN)

wesselt commented 1 year ago

The first snippet is for a python3 process running as root connecting to 10.0.1.51. That's a private IP, probably a host on your local network. You could look up the full command line with ps -a --pid 2566.

The second looks like auto_sync listening on port 60464. The listening and forwarded port can be different. It binds locally first, then tries to forward the same port, but if that is blocked, it tries different ones. So auto_sync.py can end up listening to 3000 locally, while opening 3001 on the router and forwarding that to 3000.

What's the output of the list_callbacks.py script?

beninhos commented 1 year ago

hmm.. ps -a --pid 2566 gives me

PID TTY          TIME CMD
  48866 pts/0    00:00:00 ps

python3 list_callbacks.py gives Callbacks for user xxx: No callbacks Callbacks for account 827xxx (xxx): No callbacks Callbacks for account 811xxx (xxx): No callbacks Callbacks for account 827xxx (xxx): No callbacks

No leads so far..

wesselt commented 1 year ago

What other apps do you use that manage to map a port? What happens when you try Submission or qBittorrent?

beninhos commented 1 year ago

I do not use any apps that manage ports. I usually configure the ports in the apps i use and if needed, open a port to them manually on the router. Like when i need ssh i would route port xxx to 22 on my server ip. I do not use a firewall, since i do not expose anything to the internet. Only ssh.

wesselt commented 1 year ago

In that case, the --port argument for auto_sync.py is just what you're looking for. It allows you to specify a listening port, and auto_sync.py will not attempt upnp mapping. You can then configure that port on the router.

How does that work for you?

beninhos commented 1 year ago

like this? in the auto_sync.py i replaced the --port with a random port and i open it on the router.

config.parser.add_argument("8090", type=int, help="TCP port number to listen to. Default is a random port.") config.load() -edit- i tried adding it to parameters like --port = 8090

gives me:

Apr 13 20:34:54 jaguar systemd[1]: Started Autosync BUNQ2YNAB script.
Apr 13 20:34:59 jaguar python3[1074]:   File "/pool/apps/bunq2ynab/auto_sync.py", line 15
Apr 13 20:34:59 jaguar python3[1074]:     --port = 8090
Apr 13 20:34:59 jaguar python3[1074]:     ^^^^^^
Apr 13 20:34:59 jaguar python3[1074]: SyntaxError: cannot assign to expression here. Maybe you meant '==' instead of '='?

tried changing to == and get the same error somehow.

wesselt commented 1 year ago

No need to modify the code. Instead of starting the program like python auto_sync.py, you'd start it like python auto_sync.py --port 1234. (Without the =.)

Alternatively, you can add it in the config.json file:

{ "api_token": "xxx", "personal_access_token": "xxx", "port": "1234", ...

beninhos commented 1 year ago

ah yes. I see now. I have it running trough systemd. I just add it to the config file.

-edit- the thought of it needing to be in the config.json file popped op, but did not stay in mind.. error is gone. and oh wow. this is lightning fast.

wesselt / bunq2ynab

auto_sync.py not working #37