tomitan100 / org.openhab.binding.bom

OH2 binding for Australian weather forecast and images from Bureau of Meteorology (BOM)
4 stars 3 forks source link

FTP proxy issue - it half works #5

Closed electricbrain-code closed 3 years ago

electricbrain-code commented 3 years ago

I have configured the BOM binding for the forecast, which appears to work.

However, the image stuff can't connect to the same server ?

Essentially OpenHAB is running in a Private Virtual Network in Portainer (Docker Swarm). All access in and out is strictly controlled and is accomplished via plain old Apache mod_proxy. One for inbound (public) and one for outbound (to keep untrusted OpenHAB modules from snooping the network here). Everything is Rasberry Pi 4 (10 of them). OpenHAB is the official container. Apache is the official container, MySQL is the official MariaDB container. etc (phpMyAdmin and so on)

EXTRA_JAVA_OPTS has these guys in it: "-Dhttp.proxyHost=httpdproxy -Dhttp.proxyPort=80 -Dftp.proxyHost=httpdproxy -Dftp.proxyPort=80" which, according to Oracle Java 8's manual, suggests everything will use it because this is the standard way to do things.

So it works for the forecast but not the image stuff ? For some reason it's attempting to directly lookup the site (a function the proxy should perform AFAIK).

Here is a bit of the log. The 1st bit shows the error, then further down it shows access succeeds for the forecast bit:

Thanks in advance for any help.

020-10-16 09:57:01.835 [ERROR] [binding.bom.internal.BomImageHandler] - Unable to connect to FTP server ftp.bom.gov.au java.net.UnknownHostException: ftp.bom.gov.au: Temporary failure in name resolution at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_265] at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[?:1.8.0_265] at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[?:1.8.0_265] at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[?:1.8.0_265] at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_265] at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_265] at java.net.InetAddress.getByName(InetAddress.java:1077) ~[?:1.8.0_265] at org.apache.commons.net.SocketClient.connect(SocketClient.java:202) ~[?:?] at org.apache.commons.net.SocketClient.connect(SocketClient.java:306) ~[?:?] at org.openhab.binding.bom.internal.BomImageHandler.connect(BomImageHandler.java:505) ~[?:?] at org.openhab.binding.bom.internal.BomImageHandler.refreshImage(BomImageHandler.java:157) ~[?:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_265] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_265] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_265] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_265] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265] 2020-10-16 09:57:01.845 [ERROR] [binding.bom.internal.BomImageHandler] - Unable to connect to ftp://ftp.bom.gov.au 2020-10-16 09:58:11.178 [INFO ] [nhab.binding.bom.internal.BomHandler] - Cancelling observation refresh job 2020-10-16 09:58:11.181 [INFO ] [nhab.binding.bom.internal.BomHandler] - Cancelling forecast refresh job 2020-10-16 09:58:11.394 [INFO ] [nhab.binding.bom.internal.BomHandler] - Processing precis forecast from FTP path: ftp://ftp.bom.gov.au/anon/gen/fwo/IDV10753.xml, area ID: VIC_PT042 2020-10-16 09:58:13.288 [INFO ] [nhab.binding.bom.internal.BomHandler] - Successfully processed precis forecast data. 2020-10-16 09:58:13.289 [INFO ] [nhab.binding.bom.internal.BomHandler] - Processing city/town/district forecast from FTP path: ftp://ftp.bom.gov.au/anon/gen/fwo/IDV10450.xml, area ID: VIC_PT042 2020-10-16 09:58:14.623 [INFO ] [nhab.binding.bom.internal.BomHandler] - Successfully processed city/down/district forecast data. 2020-10-16 09:58:17.930 [INFO ] [nhab.binding.bom.internal.BomHandler] - Processing precis forecast from FTP path: ftp://ftp.bom.gov.au/anon/gen/fwo/IDV10753.xml, area ID: VIC_PT042 2020-10-16 09:58:19.548 [INFO ] [nhab.binding.bom.internal.BomHandler] - Successfully processed precis forecast data. 2020-10-16 09:58:19.550 [INFO ] [nhab.binding.bom.internal.BomHandler] - Processing city/town/district forecast from FTP path: ftp://ftp.bom.gov.au/anon/gen/fwo/IDV10450.xml, area ID: VIC_PT042 2020-10-16 09:58:20.665 [INFO ] [nhab.binding.bom.internal.BomHandler] - Successfully processed city/down/district forecast data. 20

electricbrain-code commented 3 years ago

It seems that Apache's FTPclient has a function which verifies the remote call-back connection is from the computer that was called. That function might me doing the lookup. The answer might be to disable remote verification when a proxy is in use, since the local network segment may not have access to an appropriate DNS server. It is on by default.

Added from Apache's website that documents the FTP client code. setRemoteVerificationEnabled

public void setRemoteVerificationEnabled(boolean enable)

Enable or disable verification that the remote host taking part of a data connection is the same as the host to which the control connection is attached. The default is for verification to be enabled. You may set this value at any time, whether the FTPClient is currently connected or not.

Parameters:
    enable - True to enable verification, false to disable verification.

isRemoteVerificationEnabled

public boolean isRemoteVerificationEnabled()

Return whether or not verification of the remote host participating in data connections is enabled. The default behavior is for verification to be enabled.

Returns:
    True if verification is enabled, false if not.
tomitan100 commented 3 years ago

Hi,

I have updated the code and the binary. Let me know how you go.

Regards, Tom

electricbrain-code commented 3 years ago

Hi Tom, Firstly, thankyou so much for the prompt action on this.

So currently I have the system generating images (although I'm still working through getting them on the screen). It still needs the workaround outlined below - which previously did not work..

What works is (my Portainer stack follows): 1) OpenHAB official 2) ftp-proxy (an old thing that looks like an ftp server on one end and talks out the other to wherever) 3) The very latest BOM Image thing with the ftp verification fix. 4) Somewhat unrelated - MySQL 5) Somewhat unrelated - phpMyAdmin

BOM Image still looks up the IP address for ftp.bom.gov.au even though it is going via a proxy (lookup not available on that segment). So I installed the trusty old ftp-proxy and told it to connect to ftp.bom.gov.au. Then I pointed BOM image to ftp-proxy. Bingo it worked.

Previously it did not work because Portainer/Docker Swarm had that container listening on 10.0.x.24 but replying from 10.0.x.25 (don't ask me why or how - it's some docker swarm thing - also FYI x is random each time the stack is launched). So now the new binding version has turned off the part that makes sure the reply comes from the same address as the request.

The practical upshot is I have an image (which sits in a different place in the official docker image /openhan/config/html)

root@ddbe887e32c2:/openhab/conf/html# ls -lsah total 904K 4.0K drwxr-xr-x 2 openhab openhab 4.0K Oct 22 01:22 . 4.0K drwxr-xr-x 13 openhab openhab 4.0K Sep 20 16:33 .. 888K -rw-r--r-- 1 openhab openhab 887K Oct 22 02:42 IDR023.gif 4.0K -rw-r--r-- 1 openhab openhab 451 Sep 20 16:33 index.html 4.0K -rw-r--r-- 1 openhab openhab 282 Sep 20 16:33 readme.txt root@ddbe887e32c2:/openhab/conf/html#