search

KomMonitor / kommonitor-importer

GNU General Public License v3.0
0 stars 3 forks source link

Request timeout behind proxy #23

Open SebaDro opened 1 year ago

SebaDro commented 1 year ago

Description Requesting resources in the internet fails, if the internet access is only possible via proxy. E.g. requesting the geocoder HSBO instance fails in those cases. Defining a proxy via HTTPS_PROXY env variable or -Dhttp.proxyHost and -Dhttp.proxyHost jvm options does not help. It seems as the utilizied http library, which is Spring's RestTemplate for requesting the geocoder, seems to ignore those settings.

How to fix Define proxy settings programatically. RestTemplate can be configured using SimpleClientHttpRequestFactory. With this, we can provide different beans of the RestTemplate for requesting resources with and without proxy. We should also take into account that we currently use two different libs for http request. Besides the RestTemplate usage e.g., as part of the AbstractTableConverter we also have a HttpHelper class, which uses Apache Http Client. This should be harmonized.

SebaDro commented 1 year ago

In addition, we should think about replacing RestTemplate with Spring WebClient. RestTemplate will become deprecated in future Spring versions and WebClient provides a non-blocking client.

SebaDro commented 1 year ago

It might be also helpful to configure the HtppClient to decide for which hosts to use proxy and for which not. See: https://docs.spring.io/spring-boot/docs/current/reference/html/howto.html#howto.http-clients.rest-template-proxy-configuration

SebaDro commented 1 year ago

With version 2.0.4 requesting the geocoder via http proxy is working. ed6412c adds support for defining a http proxy manually within the application.yml: https://github.com/KomMonitor/kommonitor-importer/blob/61c592a7c02ebafbac18bd2d8070d50a602546ee/kommonitor-importer-app/src/main/resources/application.yml#L59-L62 You can also set the http proxy via env variables HTTP_PROXY and PROXY_PORT.

However, I leave this issue open because importing other resources from the internet may still fail, since the HttpRetriever makes use of the HttpHelper, which is based on Apache HttpClient and needs a separate handling for setting a http proxy (see: https://hc.apache.org/httpcomponents-client-4.5.x/current/tutorial/html/connmgmt.html#d5e485).

Nevertheless, refactoring the importer to make use of only a single http framework should be considered for maintainability reasons.

@cDanowski: What do you think about it?