docker image not working on host: 503 service unavailable (proxy problem?)

[o2idev]

Dear validator team :),

it's really a pita ;) and frustrating to get the validator docker image running for us :-/

Short description

We finally managed to run the docker image, but calling it via http://localhost:8080/validator or http://172.17.0.1:8080/validator from the host, we get a 503 service unavailable :-( We are aware of #323 which did not help us. The only thing seeming worth mentioning: we have to use a web proxy to connect to the internet.

What is our general usage scenario?

We built a Ubuntu 16.04 LTS virtual machine (template) usable by various German states. It provides a software for mapping German road-centric data (OKSTRA standard) to INSPIRE data ("O2I Tool"). The O2I tool uses the ETF validator (WAR in a Tomcat container) for the final step.

It broke with the HTTPS forward for code lists :-( ...

Before the public XSD URLs were mapped from HTTP to HTTPS (#311) everything was fine. But since then the VM installed validator reports errors, because it can't access the code lists (via HTTP). As mentioned here , your provided workaround seems to be some fancy proxying mechanism utilizing Apache 2 and Squid 3 in the docker image.

The first question for us was, why you would not fix it in the validator WAR code in the first place? For us it is a black box and should work without having us to deal with internal problems?! Especially if we are now forced to "intercept" HTTP calls and manipulate them, so we can use the WAR functionality :-/ Or maybe fixing it ourself in the WAR code by intercepting the HTTPS forward response (302 fwd) and simply using this connection (HttpsUrlConnection) instead. Anyways we have to try to stick to our release schedule and the docker solution seemed good enough to give it a try...

Using the docker image does not work:

The expectation would be now, that we simply run the docker image and it works with http://localhost:8080/validator as before, but it didn't as mentioned in the beginning of the ticket.

What did we do till now?

• installed docker via apt-get
• setting up the client and daemon HTTP(S) proxies (needed for internet access)
• (set up an github account with access token for the docker image download)
• ran the v2020.2 (and also v2020.1) validator docker image as mentioned there: docker run --name inspire-validator -d -p 8080:8080 -v ~/etf:/etf docker.pkg.github.com/inspire-eu-validation/community/inspire-validator:2020.2

The 503 error:

• when calling http://localhost:8080/validator inside our host VM
• looking at the host and container logs and other commands everything seems to run fine (apache logs, squid logs, etf log, jetty logs, docker container ls, docker stats)
• changing ~/etf/config/etf-config.properties // etf.webapp.base.url = http://localhost:8080/validator seems to have no influence
• trying with http://172.17.0.1:8080/validator or /validator/v2/status returns the same 503 ** as mentioned in #323 by executing ip a to get the bridged IP:

  3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:c8:98:38:a4 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever inet6 fe80::42:c8ff:fe98:38a4/64 scope link valid_lft forever preferred_lft forever

• request

  GET /validator/v2/status HTTP/1.1 Host: 172.17.0.1:8080 User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8 Accept-Language: de,en-US;q=0.7,en;q=0.3 Accept-Encoding: gzip, deflate Connection: keep-alive Upgrade-Insecure-Requests: 1

• response:

  HTTP/1.1 503 Service Unavailable Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html;charset=iso-8859-1 Content-Length: 345 Server: Jetty(9.3.6.v20151106)

• (we had to add the NO_PROXY for 172.17.0.1 to the host browser (FF))

Please help us with this (or maybe even better - fix the issue in the WAR so we do not have to use the docker image and all the weired HTTP(S) handling at all).

Thanks a lot for your work and support Andreas :)

[jenriquesoriano]

Dear @o2idev,

thanks for your feedback. We have tried to reproduce the issue by deploying the dockerfile from the v2020.2 release and it has worked correctly.

We are aware that the different logs are not raising issues in your deployment, but it would be very useful if you can provide us with them, so we can analyze these logs in-depth.

Thus, as we have not been able to reproduce the reported behaviour, could you please provide us with these log files (apache logs, squid logs, etf log, jetty logs, docker container ls, docker stats)?

Best regards,

[o2idev]

export.zip Hey Jenrique,

thanks for your feedback. Attached you'll find the logs. Here's the output of the mentioned commands:

sudo docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0fb4f28fc41d docker.pkg.github.com/inspire-eu-validation/community/inspire-validator:2020.2 "/docker-entrypoint.…" 4 days ago Up 19 minutes 0.0.0.0:8080->8080/tcp inspire-validator

sudo docker stats CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 0fb4f28fc41d inspire-validator 0.19% 1.537GiB / 31.42GiB 4.89% 9.61kB / 4.59kB 258MB / 49.2kB 89

Please take into account that we have an internet proxy (as mentioned above) running that may have to do with this.

Thanks a lot for your help :) Andreas

[jenriquesoriano]

Dear @o2idev,

thanks for your feedback. As you mentioned, the logs do not reveal any relevant issue in the components of the infrastructure.

Nevertheless, as you are getting a 503 error inside the VM, it is possible that an issue might have occurred when running the dockerbuild of version v2020.2. Could you check if when running the dockerbuild (docker build -t etf-... ) it shows some kind of error and, if so, send it to us?

Best,

[o2idev]

Hi,

I have no experience with docker, so I don't know what's the problem with this :-/

> sudo docker build -t "docker.pkg.github.com/inspire-eu-validation/community/inspire-validator:2020.1.2"
"docker build" requires exactly 1 argument.
See 'docker build --help'.

Usage:  docker build [OPTIONS] PATH | URL | -
Build an image from a Dockerfile

[jenriquesoriano]

Dear @o2idev ,

thanks for your feedback. Let's try an alternative approach, downloading the .zip file and executing it locally. Please follow these steps: 1.- Download .zip file from v2020.2 version at: https://github.com/inspire-eu-validation/community/releases/download/v2020.2/inspire-validator-2020.2.zip 2.- Unzip the file in a local directory 3.- Build the dockerfile, accessing to the directory where the dockerfile is located and executing: docker build -t etf-webapp 4.- Run docker: sudo docker run -p 8080:8080 etf-webapp

If the issue persists, please send the logs from steps 3 and 4.

In order to isolate the issue, I may suggest to do it in a local environment to get familiar with the docker instance before moving to the VM.

Best regards,

[o2idev]

output.txt I do not know, what's the problem with your command:

/tmp/#12788-etfv-zip$ ls -ltra
insgesamt 336912
drwxrwxr-x  2 user user      4096 Mär 27 13:03 res
-rw-r--r--  1 user user      3315 Mai 25 17:44 Dockerfile
-rw-r--r--  1 user user 172428403 Jul  2 15:28 validator.war
-rw-r--r--  1 user user      3577 Jul 14 10:56 README.md
-rw-rw-r--  1 user user 172539245 Jul 25 13:54 inspire-validator-2020.2.zip
drwxrwxrwt 23 root root      4096 Jul 25 13:58 ..
drwxrwxr-x  3 user user      4096 Jul 25 13:59 .

user@...:/tmp/#12788-etfv-zip$ sudo docker build -t etf-webapp
"docker build" requires exactly 1 argument.
See 'docker build --help'.

Usage:  docker build [OPTIONS] PATH | URL | -

Build an image from a Dockerfile

user@...:/tmp/#12788-etfv-zip$ sudo docker --version
Docker version 19.03.12, build 48a66213fe

but I tried this and it seems to fail at step 30/38 (no logs other than console output found):

user@...:/tmp/#12788-etfv-zip$ sudo docker build - < Dockerfile 
Sending build context to Docker daemon   5.12kB
Step 1/38 : FROM jetty:9.3.6
9.3.6: Pulling from library/jetty
Image docker.io/library/jetty:9.3.6 uses outdated schema1 manifest format. Please upgrade to a schema2 image for better future compatibility. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
03e1855d4f31: Pull complete 
...
Step 29/38 : RUN mv /docker-entrypoint.bash /docker-entrypoint-jetty.bash
 ---> Running in 848a81833c9a
Removing intermediate container 848a81833c9a
 ---> 024a7873fec4
Step 30/38 : COPY res/docker-entrypoint.sh /
COPY failed: stat /var/lib/docker/tmp/docker-builder527712404/res/docker-entrypoint.sh: no such file or directory

the tmp dir seems to be the wrong one since the mentioned file seems to be in the res/ folder I guess:

root@...:/var/lib/docker/tmp# ls -ltra /var/lib/docker/tmp
insgesamt 8
drwx--x--x 14 root root 4096 Jul 21 20:51 ..
drwx------  2 root root 4096 Jul 25 14:07 .

root@...:/tmp/#12788-etfv-zip# ls res/
docker-entrypoint.sh  proxy.conf  squid.conf

re-executing did not help

I'll update when I go on with it or you point me in the right direction :-/

ok: this error ticket put me in the right direction which argument was likely missing in your provided suggestion (current dir '.' at the end): sudo docker build -t etf-webapp .

so now it fails with several problems (find full output.txt attached):

root@...:/tmp/#12788-etfv-zip# sudo docker build -t etf-webapp .
Sending build context to Docker daemon  345.3MB
Step 1/38 : FROM jetty:9.3.6
 ---> 8ca75721d978
Step 2/38 : MAINTAINER Carlos Palma <carlospalma at guadaltel.com>
 ---> Using cache
 ---> dd149fe926ed

[...]

Step 30/38 : COPY res/docker-entrypoint.sh /
 ---> a55080b9ccd2
Step 31/38 : RUN apt-get update; true
 ---> Running in 0c0087140044
Err http://httpredir.debian.org jessie InRelease

Err http://security.debian.org jessie/updates InRelease

Err http://security.debian.org jessie/updates Release.gpg
  Temporary failure resolving 'security.debian.org'
Err http://httpredir.debian.org jessie-updates InRelease

Err http://httpredir.debian.org jessie-backports InRelease

Err http://httpredir.debian.org jessie Release.gpg
  Temporary failure resolving 'httpredir.debian.org'
Err http://httpredir.debian.org jessie-updates Release.gpg
  Temporary failure resolving 'httpredir.debian.org'
Err http://httpredir.debian.org jessie-backports Release.gpg
  Temporary failure resolving 'httpredir.debian.org'
Reading package lists...
W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie/InRelease  

W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-updates/InRelease  

W: Failed to fetch http://security.debian.org/dists/jessie/updates/InRelease  

W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-backports/InRelease  

W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie/Release.gpg  Temporary failure resolving 'httpredir.debian.org'

W: Failed to fetch http://security.debian.org/dists/jessie/updates/Release.gpg  Temporary failure resolving 'security.debian.org'

W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-updates/Release.gpg  Temporary failure resolving 'httpredir.debian.org'

W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-backports/Release.gpg  Temporary failure resolving 'httpredir.debian.org'

W: Some index files failed to download. They have been ignored, or old ones used instead.
Removing intermediate container 0c0087140044
 ---> 4d56d1293176
Step 32/38 : RUN apt-get install -y squid3
 ---> Running in 915220bf24fb
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package squid3
The command '/bin/sh -c apt-get install -y squid3' returned a non-zero code: 100

I think the apt-get download problems can be ignored for now (and may be HTTP proxy related), since they are working in the VMs Firefox with working proxy for e.g. http://httpredir.debian.org/).

re-executing it shows the same apt-get install -y squid3 related problem:

[...]

Step 31/38 : RUN apt-get update; true
 ---> Using cache
 ---> 4d56d1293176
Step 32/38 : RUN apt-get install -y squid3
 ---> Running in f6a6d06877ab
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package squid3
The command '/bin/sh -c apt-get install -y squid3' returned a non-zero code: 100

inside the VM host it knows the squid3 package and it could be installed:

root@...:/tmp/#12788-etfv-zip# apt-cache search squid3
squid-common - Full featured Web Proxy cache (HTTP proxy) - common files
squid3 - Dummy transitional package.
squid-cgi - Vollausgestatteter Webproxycache (HTTP-Proxy) - CGI-Kontrollanwendung
squidclient - Vollausgestatteter Webproxycache (HTTP-Proxy) - Kontrollwerkzeug

root@...:/tmp/#12788-etfv-zip# sudo apt-get install squid3
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.       
Statusinformationen werden eingelesen.... Fertig
Die folgenden Pakete wurden automatisch installiert und werden nicht mehr benötigt:
  libgdal1i libprotobuf-c1 sysstat
Verwenden Sie »sudo apt autoremove«, um sie zu entfernen.
Die folgenden zusätzlichen Pakete werden installiert:
  libecap3 squid squid-common squid-langpack
Vorgeschlagene Pakete:
  squidclient squid-cgi squid-purge smbclient winbindd
Die folgenden NEUEN Pakete werden installiert:
  libecap3 squid squid-common squid-langpack squid3
0 aktualisiert, 5 neu installiert, 0 zu entfernen und 0 nicht aktualisiert.
Es müssen 2.690 kB an Archiven heruntergeladen werden.
Nach dieser Operation werden 11,0 MB Plattenplatz zusätzlich benutzt.

it also works with the claimed command:

root@list-lnxd-o2i:/tmp/#12788-etfv-zip# /bin/sh -c "apt-get install -y squid3"
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.       
Statusinformationen werden eingelesen.... Fertig
Die folgenden Pakete wurden automatisch installiert und werden nicht mehr benötigt:
  libgdal1i libprotobuf-c1 sysstat
Verwenden Sie »sudo apt autoremove«, um sie zu entfernen.
Die folgenden zusätzlichen Pakete werden installiert:
  libecap3 squid squid-common squid-langpack
Vorgeschlagene Pakete:
  squidclient squid-cgi squid-purge smbclient winbindd
Die folgenden NEUEN Pakete werden installiert:
  libecap3 squid squid-common squid-langpack squid3
0 aktualisiert, 5 neu installiert, 0 zu entfernen und 0 nicht aktualisiert.
Es müssen 2.690 kB an Archiven heruntergeladen werden.
Nach dieser Operation werden 11,0 MB Plattenplatz zusätzlich benutzt.
Holen:1 http://de.archive.ubuntu.com/ubuntu xenial/main amd64 libecap3 amd64 1.0.1-3ubuntu3 [16,7 kB]
Holen:2 http://de.archive.ubuntu.com/ubuntu xenial/main amd64 squid-langpack all 20150704-1 [145 kB]
Holen:3 http://de.archive.ubuntu.com/ubuntu xenial-updates/main amd64 squid-common all 3.5.12-1ubuntu7.11 [175 kB]
Holen:4 http://de.archive.ubuntu.com/ubuntu xenial-updates/main amd64 squid amd64 3.5.12-1ubuntu7.11 [2.320 kB]
Holen:5 http://de.archive.ubuntu.com/ubuntu xenial-updates/main amd64 squid3 all 3.5.12-1ubuntu7.11 [32,3 kB]
Es wurden 2.690 kB in 4 s geholt (570 kB/s).
Vormals nicht ausgewähltes Paket libecap3:amd64 wird gewählt.
(Lese Datenbank ... 192461 Dateien und Verzeichnisse sind derzeit installiert.)
Vorbereitung zum Entpacken von .../libecap3_1.0.1-3ubuntu3_amd64.deb ...
Entpacken von libecap3:amd64 (1.0.1-3ubuntu3) ...
Vormals nicht ausgewähltes Paket squid-langpack wird gewählt.
Vorbereitung zum Entpacken von .../squid-langpack_20150704-1_all.deb ...
Entpacken von squid-langpack (20150704-1) ...
Vormals nicht ausgewähltes Paket squid-common wird gewählt.
Vorbereitung zum Entpacken von .../squid-common_3.5.12-1ubuntu7.11_all.deb ...
Entpacken von squid-common (3.5.12-1ubuntu7.11) ...
Vormals nicht ausgewähltes Paket squid wird gewählt.
Vorbereitung zum Entpacken von .../squid_3.5.12-1ubuntu7.11_amd64.deb ...
Entpacken von squid (3.5.12-1ubuntu7.11) ...
Trigger für libc-bin (2.23-0ubuntu11.2) werden verarbeitet ...
Trigger für man-db (2.7.5-1) werden verarbeitet ...
Trigger für systemd (229-4ubuntu21.28) werden verarbeitet ...
Trigger für ureadahead (0.100.0-19.1) werden verarbeitet ...
Trigger für ufw (0.35-0ubuntu2) werden verarbeitet ...
libecap3:amd64 (1.0.1-3ubuntu3) wird eingerichtet ...
squid-langpack (20150704-1) wird eingerichtet ...
squid-common (3.5.12-1ubuntu7.11) wird eingerichtet ...
squid (3.5.12-1ubuntu7.11) wird eingerichtet ...
Skipping profile in /etc/apparmor.d/disable: usr.sbin.squid
Trigger für libc-bin (2.23-0ubuntu11.2) werden verarbeitet ...
Trigger für systemd (229-4ubuntu21.28) werden verarbeitet ...
Trigger für ureadahead (0.100.0-19.1) werden verarbeitet ...
Trigger für ufw (0.35-0ubuntu2) werden verarbeitet ...
Vormals nicht ausgewähltes Paket squid3 wird gewählt.
(Lese Datenbank ... 194745 Dateien und Verzeichnisse sind derzeit installiert.)
Vorbereitung zum Entpacken von .../squid3_3.5.12-1ubuntu7.11_all.deb ...
Entpacken von squid3 (3.5.12-1ubuntu7.11) ...
squid3 (3.5.12-1ubuntu7.11) wird eingerichtet ...

So I guess it could be HTTP(S) proxy related that it must be set up inside the docker container so apt-get works in it properly? How to do it then?

(The VM is no problem to use. Locally I have a Win10 machine and this is already quite different.)

Thx

[o2idev]

Please have a look at this because we have to release our product and this is a critical blocker :-( (best would be to allow HTTP access to codelists as discussed in #334)

[jenriquesoriano]

Dear @o2idev

we have replicated the process you have posted and we have been able to deploy the infrastructure (thanks for the "." issue). As far as we can understand from your comments, it seems to be a problem of the proxy that you are using for apt-get. If it is the case, you should configure docker to use your proxy, so these particularities do not affect the docker deployment. Please follow this link for further details on how to configure docker to use a proxy. https://docs.docker.com/network/proxy/

Other useful links: https://elegantinfrastructure.com/docker/ultimate-guide-to-docker-http-proxy-configuration/ https://www.opensourceforu.com/2019/03/setting-up-a-proxy-for-docker-containers-in-linux-machines/ https://www.thegeekdiary.com/how-to-configure-docker-to-use-proxy/

Hope it helps.

Best regards,

[carlospzurita]

Dear @o2idev

Did our feedback resulted useful for you? Have you progressed in the setup of the instance? Please let us know if there is anything else we can do, or any other information that we may provide.

[o2idev]

Dear @carlospzurita,

thanks for your commitment. We delayed working on this for now. It was decided that for now we advise our customers to use the online validator although we know its much more manual work and it has problems with large files (we have 10 to 50 GB) ones. It's sad, but we have to go on. Maybe I'll have time to test this within the next weeks or it's fixed on your end which is anyways the preferred and clean way as I see it.

Having had a quick look at the provided links, I guess there may be some things worth a try.

Thanks again Andreas :)

[carlospzurita]

Dear @o2idev

Did you have the occasion to revisit this issue? It seems that the problematic part in your side is the proxy configuration for the Docker daemon. We have been successfully building the Docker image in different environments, and using the released image in GitHub. If you could provide more output from new attempts, we can solve this.

Moving the solution into the WAR file is rather problematic. This situation is very INSPIRE specific, and the ETF that is running underneath it is being kept the most agnostic as possible. In order to have this solved at code level, we would need to include a solution to let all redirection pass through, which is less than ideal from the perspective of security.

[o2idev]

Dear @o2idev

we have replicated the process you have posted and we have been able to deploy the infrastructure (thanks for the "." issue). As far as we can understand from your comments, it seems to be a problem of the proxy that you are using for apt-get. If it is the case, you should configure docker to use your proxy, so these particularities do not affect the docker deployment. Please follow this link for further details on how to configure docker to use a proxy. https://docs.docker.com/network/proxy/

Other useful links: https://elegantinfrastructure.com/docker/ultimate-guide-to-docker-http-proxy-configuration/ https://www.opensourceforu.com/2019/03/setting-up-a-proxy-for-docker-containers-in-linux-machines/ https://www.thegeekdiary.com/how-to-configure-docker-to-use-proxy/

Hope it helps.

Best regards,

Hi Carlos,

we now had some time to get back on this, used the proxy for the build and run environments (which we did not before), managed to build (and run as before) it now (with errors), but the 503 issue persits. We had a look at all the links and details. (The proxy worked providing the IPs, but not with DNS names which is understandable and just mentioned as side note for others.)

More details: build cmd: (see attached build logs for all details)

> docker build --build-arg http_proxy=http://<my-proxy-ip>:8080  --build-arg https_proxy=http://<my-proxy-ip>:8080  --build-arg no_proxy=127.0.0.1,localhost,*.<my-domain>  -t etf-webapp .

Sending build context to Docker daemon  172.7MB
Step 1/38 : FROM jetty:9.3.6
 ---> 8ca75721d978

...

Step 31/38 : RUN apt-get update; true
 ---> Running in ede8ffa25ed1
Ign http://httpredir.debian.org jessie InRelease
Get:1 http://security.debian.org jessie/updates InRelease [44.9 kB]
Get:2 http://httpredir.debian.org jessie-updates InRelease [16.3 kB]
Ign http://httpredir.debian.org jessie-backports InRelease
Get:3 http://httpredir.debian.org jessie Release.gpg [1652 B]
Ign http://httpredir.debian.org jessie-backports Release.gpg
Get:4 http://httpredir.debian.org jessie Release [77.3 kB]
Ign http://httpredir.debian.org jessie-backports Release
Err http://httpredir.debian.org jessie-backports/main amd64 Packages

Err http://httpredir.debian.org jessie-backports/main amd64 Packages

Err http://httpredir.debian.org jessie-backports/main amd64 Packages

Err http://httpredir.debian.org jessie-backports/main amd64 Packages

Err http://httpredir.debian.org jessie-backports/main amd64 Packages
  404  Not Found
Get:5 http://security.debian.org jessie/updates/main amd64 Packages [992 kB]
Get:6 http://httpredir.debian.org jessie-updates/main amd64 Packages [20 B]
Get:7 http://httpredir.debian.org jessie/main amd64 Packages [9098 kB]
W: There is no public key available for the following key IDs:
AA8E81B4331F7F50
W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-backports/main/binary-amd64/Packages  404  Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.
Fetched 10.2 MB in 20s (488 kB/s)
Removing intermediate container ede8ffa25ed1
 ---> b1be2ddda727
Step 32/38 : RUN apt-get install -y squid3
 ---> Running in 2b8bd95c69b4
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  bsd-mailx cron exim4 exim4-base exim4-config exim4-daemon-light
  init-system-helpers libalgorithm-c3-perl libarchive-extract-perl libbsd0
  libcgi-fast-perl libcgi-pm-perl libclass-c3-perl libclass-c3-xs-perl
  libcpan-meta-perl libdata-optlist-perl libdata-section-perl libecap2
  libfcgi-perl libgdbm3 liblockfile-bin liblockfile1 liblog-message-perl
  liblog-message-simple-perl libltdl7 libmnl0 libmodule-build-perl
  libmodule-pluggable-perl libmodule-signature-perl libmro-compat-perl
  libnetfilter-conntrack3 libnfnetlink0 libpackage-constants-perl
  libparams-util-perl libpod-latex-perl libpod-readme-perl libpopt0
  libregexp-common-perl libsoftware-license-perl libsub-exporter-perl
  libsub-install-perl libterm-ui-perl libtext-soundex-perl
  libtext-template-perl libxml2 logrotate perl perl-base perl-modules psmisc
  rename sgml-base squid-langpack squid3-common xml-core
Suggested packages:
  anacron checksecurity mail-reader eximon4 exim4-doc-html exim4-doc-info file
  spf-tools-perl swaks perl-doc libterm-readline-gnu-perl
  libterm-readline-perl-perl make libb-lint-perl libcpanplus-dist-build-perl
  libcpanplus-perl libfile-checktree-perl libobject-accessor-perl
  sgml-base-doc squidclient squid-cgi squid-purge resolvconf smbclient ufw
  winbindd debhelper
Recommended packages:
  mailx libarchive-tar-perl
The following NEW packages will be installed:
  bsd-mailx cron exim4 exim4-base exim4-config exim4-daemon-light
  init-system-helpers libalgorithm-c3-perl libarchive-extract-perl libbsd0
  libcgi-fast-perl libcgi-pm-perl libclass-c3-perl libclass-c3-xs-perl
  libcpan-meta-perl libdata-optlist-perl libdata-section-perl libecap2
  libfcgi-perl libgdbm3 liblockfile-bin liblockfile1 liblog-message-perl
  liblog-message-simple-perl libltdl7 libmnl0 libmodule-build-perl
  libmodule-pluggable-perl libmodule-signature-perl libmro-compat-perl
  libnetfilter-conntrack3 libnfnetlink0 libpackage-constants-perl
  libparams-util-perl libpod-latex-perl libpod-readme-perl libpopt0
  libregexp-common-perl libsoftware-license-perl libsub-exporter-perl
  libsub-install-perl libterm-ui-perl libtext-soundex-perl
  libtext-template-perl libxml2 logrotate perl perl-modules psmisc rename
  sgml-base squid-langpack squid3 squid3-common xml-core
The following packages will be upgraded:
  perl-base
1 upgraded, 55 newly installed, 0 to remove and 92 not upgraded.
Need to get 14.0 MB of archives.
After this operation, 55.1 MB of additional disk space will be used.

...

Step 38/38 : CMD ["java","-jar","/usr/local/jetty/start.jar"]
 ---> Running in edad0f4a52ba
Removing intermediate container edad0f4a52ba
 ---> b91ef742ebb3
Successfully built b91ef742ebb3
Successfully tagged etf-webapp:latest

with this run.sh script it did not work:

## run.sh

proxy_url=http://<myproxy-ip>:8080
proxy_params="--env http_proxy=$proxy_url  --env https_proxy=$proxy_url  --env no_proxy=127.0.0.1,localhost,*.<mydomain>"
etfv_docker_img=docker.pkg.github.com/inspire-eu-validation/community/inspire-validator:2020.2
docker stop inspire-validator
docker rm inspire-validator
docker run  $proxy_params  --name inspire-validator -d -p 8080:8080 -v ~/etf:/etf  $etfv_docker_img
wget http://localhost:8080/validator

> ./run.sh

inspire-validator
inspire-validator
--2020-08-27 17:28:56--  http://localhost:8080/validator
Auflösen des Hostnamen »localhost (localhost)«... 127.0.0.1
Verbindungsaufbau zu localhost (localhost)|127.0.0.1|:8080... verbunden.
HTTP-Anforderung gesendet, warte auf Antwort... 302 Found
Platz: http://localhost:8080/validator/ [folge]
--2020-08-27 17:29:27--  http://localhost:8080/validator/
Wiederverwendung der bestehenden Verbindung zu localhost:8080.
HTTP-Anforderung gesendet, warte auf Antwort... 503 Service Unavailable
2020-08-27 17:29:27 FEHLER 503: Service Unavailable.

Thanks for any hint or help! Andreas

build-with-proxy-ip.log

[carlospzurita]

Dear @o2idev

We have been checking your script and build command. One thing to be noted is that you are not using the same image being built that the one executed on run.sh. Anyway, we have executed ourselves the run script with the proxy parameters set to our own proxy_url, with the image docker.pkg.github.com/inspire-eu-validation/community/inspire-validator:2020.2, and we have been able to start the container without any issue.

After executing the docker run, can you please execute the command docker logs inspire-validator and share the output? To have a more clear picture. Also, if it's possible, please share with us the file /etf/logs/etf.log inside the volume of the container.

Also, if it's possible, you can send us a virtualization of your environment, to deploy it under a proxy and check the execution.

[o2idev]

Hi Carlos,

it was not our intent to run the built image, but we'll have a look into it. Attached are the logs (likely unimportant ... the system time there is CET-2 (=UTC) and thus 2 hours before our CET: 12:46 UTC vs. 14:46 CET).

docker,logs,inspire-validator.log etf,log,etf.log

Thanks for your help Andreas :)

[o2idev]

and we are currently evaluating if we are allowed to ship you our VM.

[o2idev]

For now it was decided to try other ways and that providing the VM is not desired.

[carlospzurita]

Do you happen to have the full Docker log? It appears that is truncated. However, what we can extract from the etf.log file is that the error seems to be happening on the installation of the library FunctX

2020-09-01 12:45:35.286 [main] INFO  d.i.e.d.d.b.BsxDataStorage - Installing FunctX XQuery Function Library from https://files.basex.
org/modules/expath/functx-1.0.xar
2020-09-01 12:46:15.470 [main] INFO  d.i.e.w.c.EtfConfigController - Bye

Can you verify that the URL https://files.basex.org/modules/expath/functx-1.0.xar is reachable from inside the Docker container, and through your proxy?

[o2idev]

Now here we have the full output including stderr output because we used

sudo docker logs inspire-validator > /tmp/docker,logs,inspire-validator.log

before, but have to use

sudo docker logs inspire-validator > /tmp/docker,logs,inspire-validator.log 2>$1

docker,logs,inspire-validator.log

it confirms your suspicion:

Sep 01, 2020 12:46:15 PM org.springframework.web.context.support.XmlWebApplicationContext refresh
WARNING: Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataStorageService': Invocation of init method failed; nested exception is de.interactive_instruments.exceptions.InitializationException: FunctX XQuery Function Library installation failed. If a proxy server is used, set the Java Virtual Machine parameter 'http.proxyHost'. Otherwise download functx-1.0.xar manually, extract the file (it is a ZIP file) and copy it to the BaseXRepo 'repo' folder /etf/ds/db/repo [bxerr:BXRE0001] Package 'https://files.basex.org/modules/expath/functx-1.0.xar' does not exist.

the containers env property HTTPS_PROXY_HOST (which is also used by wget) could be a problem indicator which is set to none but should be localhost like HTTP_PROXY_HOST:

trying to dowload the (HTTPS) https://.../functx-1.0.xar or https://www.google.com:

> root@bdeb6472f6c8:/var/lib/jetty# wget https://files.basex.org/modules/expath/functx-1.0.xar
--2020-09-02 13:37:16--  https://files.basex.org/modules/expath/functx-1.0.xar
Resolving files.basex.org (files.basex.org)... failed: Name or service not known.
wget: unable to resolve host address ‘files.basex.org’

root@bdeb6472f6c8:/var/lib/jetty# wget https://www.google.com
--2020-09-02 13:38:09--  https://www.google.com/
Resolving www.google.com (www.google.com)... ^C

(same problem with google.com => aborted wait by user)

trying to dowload the (HTTP) http://.../functx-1.0.xar and http://www.google.com we see our (like the etf-validators) 503 service unavailable response:

wget http://files.basex.org/modules/expath/functx-1.0.xar
--2020-09-02 13:55:36--  http://files.basex.org/modules/expath/functx-1.0.xar
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:3128... connected.
Proxy request sent, awaiting response... 503 Service Unavailable
2020-09-02 13:56:06 ERROR 503: Service Unavailable.

wget http://www.google.com
--2020-09-02 13:40:38--  http://www.google.com/
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:3128... connected.
Proxy request sent, awaiting response... 503 Service Unavailable
2020-09-02 13:41:08 ERROR 503: Service Unavailable.

and here we can see why:

set | grep -i proxy

HTTPS_PROXY_HOST=none
HTTPS_PROXY_PASSWORD=none
HTTPS_PROXY_PORT=3129
HTTPS_PROXY_USERNAME=none

HTTP_PROXY_HOST=localhost
HTTP_PROXY_PASSWORD=none
HTTP_PROXY_PORT=3128
HTTP_PROXY_USERNAME=none

now trying to fix it by simple assigning the localhost does not seem to fix it:

root@bdeb6472f6c8:/var/lib/jetty# export HTTPS_PROXY_HOST=localhost
root@bdeb6472f6c8:/var/lib/jetty# set | grep -i proxy

HTTPS_PROXY_HOST=localhost
HTTPS_PROXY_PASSWORD=none
HTTPS_PROXY_PORT=3129
HTTPS_PROXY_USERNAME=none

HTTP_PROXY_HOST=localhost
HTTP_PROXY_PASSWORD=none
HTTP_PROXY_PORT=3128
HTTP_PROXY_USERNAME=none
_=HTTPS_PROXY_HOST

root@bdeb6472f6c8:/var/lib/jetty# wget https://files.basex.org/modules/expath/functx-1.0.xar
--2020-09-02 14:01:09--  https://files.basex.org/modules/expath/functx-1.0.xar
Resolving files.basex.org (files.basex.org)... failed: Name or service not known.
wget: unable to resolve host address ‘files.basex.org’

directly querying the proxies results in this (again we have the 503 service unavailable here for HTTP):

root@bdeb6472f6c8:/var/lib/jetty# wget http://localhost:3128
--2020-09-02 14:02:26--  http://localhost:3128/
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:3128... connected.
Proxy request sent, awaiting response... 503 Service Unavailable
2020-09-02 14:02:26 ERROR 503: Service Unavailable.

root@bdeb6472f6c8:/var/lib/jetty# wget https://localhost:3129
--2020-09-02 14:02:37--  https://localhost:3129/
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:3129... failed: Connection refused.
Connecting to localhost (localhost)|::1|:3129... failed: Cannot assign requested address.
Retrying.

--2020-09-02 14:02:38--  (try: 2)  https://localhost:3129/
...

(resetting the HTTPS_PROXY_HOST to none gives the same result)

[carlospzurita]

The HTTPS_PROXY_HOST and HTTP_PROXY_HOST environment variables are only used internally by the ETF, and is referring to localhost as it is inside the container. They are passed as Java options for starting the server. These are not the regular variables that are used by your Docker client to communicate with the external network, that are named HTTPS_PROXY and HTTP_PROXY.

The library functx is requested via HTTPS from the ETF (the HTTP one has a redirection). In order to be able to reach it, the container must be started using the same approach as for the HTTP proxy. Seeing your run parameters, you are using the same host and port for HTTP and HTTPS, and this may be causing issues.

[o2idev]

In order to be able to reach it, the container must be started using the same approach as for the HTTP proxy. Seeing your run parameters, you are using the same host and port for HTTP and HTTPS, and this may be causing issues.

Sorry, could you be a bit more specific as what exactly we should do? Our HTTP and HTTPS requests are proxied by the same company service/proxy and that is a given must for us (and does not cause any issues anywhere else). If that should be the problem, then your docker workaround construct is broken (for us).

Thanks for any more hints on how to solve it

[carlospzurita]

What we were suggesting is that, looking at run.sh script that you provided before, there may be some issue setting the same exact port for HTTP and HTTPS requests, and that may be causing some problems, because usually proxy services have different ports set up for them.

proxy_url=http://<myproxy-ip>:8080
proxy_params="--env http_proxy=$proxy_url  --env https_proxy=$proxy_url  --env no_proxy=127.0.0.1,localhost,*.<mydomain>"

Checking the log file from the Docker container, it seems that you can't connect to any HTTPS location, as you can see here

+ wget -q https://github.com/inspire-eu-validation/ets-repository/archive/staging.zip -O projects.zip
2020/09/01 12:44:41 kid1| Making directories in /var/spool/squid3/0A
2020/09/01 12:44:41 kid1| Making directories in /var/spool/squid3/0B
2020/09/01 12:44:41 kid1| Making directories in /var/spool/squid3/0C
2020/09/01 12:44:41 kid1| Making directories in /var/spool/squid3/0D
2020/09/01 12:44:41 kid1| Making directories in /var/spool/squid3/0E
2020/09/01 12:44:41 kid1| Making directories in /var/spool/squid3/0F
+ mkdir -p /etf/projects/inspire-ets-repository
+ unzip -o projects.zip -d /etf/projects/inspire-ets-repository
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
Archive:  projects.zip
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of projects.zip or
        projects.zip.zip, and cannot find projects.zip.ZIP, period.
+ rm master.zip

This was the same situation that was happening in your previous comment https://github.com/inspire-eu-validation/community/issues/351#issuecomment-663849277 executing the Docker build, where the step of downloading packages was failing due to HTTP requests unable go through your network, until the HTTP proxy was correctly configured.

As a way to test this, you can run any other common Docker image: for example the cURL official image, and executing a cURL from insdie the container. Using the run script modified, you can check the output for this and look the response code. If everything is okay, you should be getting a 200 OK code

proxy_url=http://localhost:8080
proxy_params="--env http_proxy=$proxy_url  --env https_proxy=$proxy_url  --env no_proxy=127.0.0.1,localhost,*.mydomain"
etfv_docker_img=curlimages/curl
docker stop inspire-validator
docker rm inspire-validator
docker run -it $proxy_params  --name inspire-validator -p 8080:8080 $etfv_docker_img curl -v https://files.basex.org/modules/expath/functx-1.0.xar

At the moment, this problems do not seem related to any internal issue on the specific Docker image generated for the INSPIRE validator, and more on the Docker client configuration on your host machine. With the proposed test we can confirm the suspicions, or investigate deeper in this issue.

[o2idev]

Thanks a lot for the details and explanations. So with the curl image everything seems fine (just replaced our proxy ip with <myproxyip> and domain with <mydomain> below). I guess this means that it is more likely a problem on your docker image side?

> sudo sh run.sh
inspire-validator
inspire-validator
--2020-09-09 10:19:17--  http://localhost:8080/validator
Auflösen des Hostnamen »localhost (localhost)«... 127.0.0.1
Verbindungsaufbau zu localhost (localhost)|127.0.0.1|:8080... fehlgeschlagen: Verbindungsaufbau abgelehnt.
Unable to find image 'curlimages/curl:latest' locally
latest: Pulling from curlimages/curl
aad63a933944: Pull complete 
f874071515b5: Pull complete 
0455a44b58fe: Pull complete 
03e51947833e: Pull complete 
7e0b43b45efc: Pull complete 
fde5427d273e: Pull complete 
d0ddcc96de0e: Pull complete 
93ee9c225daa: Pull complete 
3d382a82c39c: Pull complete 
Digest: sha256:bd5bbd35f89b867c1dccbc84b8be52f3f74dea20b46c5fe0db3780e040afcb6f
Status: Downloaded newer image for curlimages/curl:latest
* Uses proxy env variable no_proxy == '127.0.0.1,localhost,*.<mydomain>'
* Uses proxy env variable https_proxy == 'http://<myproxyip>:8080'
*   Trying <myproxyip>:8080...
* Connected to <myproxyip> (<myproxyip>) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to files.basex.org:443
> CONNECT files.basex.org:443 HTTP/1.1
> Host: files.basex.org:443
> User-Agent: curl/7.72.0-DEV
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /cacert.pem
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=files.basex.org
*  start date: Aug 30 21:53:38 2020 GMT
*  expire date: Nov 28 21:53:38 2020 GMT
*  subjectAltName: host "files.basex.org" matched cert's "files.basex.org"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x555c24a829c0)
> GET /modules/expath/functx-1.0.xar HTTP/2
> Host: files.basex.org
> user-agent: curl/7.72.0-DEV
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200 
< date: Wed, 09 Sep 2020 08:19:04 GMT
< server: Apache
< last-modified: Wed, 18 Jul 2012 11:56:17 GMT
< etag: "663d-4c51959b91a40"
< accept-ranges: bytes
< content-length: 26173
< content-type: application/vnd.xara
< 
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* stopped the pause stream!
* Connection #0 to host <myproxyip> left intact

[carlospzurita]

After seeing this, it seems indeed the problem is specific to the validator container communicating with your proxy setup. Given that we can't access a VM from your current environment, can you at least share with us the stack in your proxy? What kind of servers are involved, how many of them are there, how are they configured... In order to replicate in our own premises your network and work on the solution for this.

Also, did you attempted to use the Docker image in a different environment? We have deployed this very same image in different scenarios (AWS ElasticBeanstalk, docker-swarm clusters...), and we didn't encountered any issue, so we may have overlooked some problems that may be present in other situations such as yours developing this solution.

[o2idev]

Thanks for the clarification. We are currently evaluating, which information we could share, but we would not like to provide it publicly here. Please send us some contact address to o2i-tool |AT| list.smwa.sachsen.de.

ah - no - we did not test the image somewhere else.

[carlospzurita]

Dear @o2idev

We have sent you the contact information for this, please check it out and provide us whichever information that you can provide.

Also, it would very useful to know of alternative deployments, so if it's possible for you please share with us your experience using the validator deployment in other settings outside of your proxy configuration.

[carlospzurita]

Dear @o2idev

Did you get our mail with the contact information about this issue? We would like to replicate your environment as much as we can in order to help you. Also, did you have the opportunity to check this comment with an illustrative diagram of the networking infrastructure? It may have useful information for your issue.

Best.

[o2idev]

Hey Carlos,

thanks for the reminder. Unfortunately we can't spend any more time and resources on the subject for budget and criticality reasons. We and our customers so far treat the issue as an ignorable warning (although technically it is an error). As long as this is fine with us it'll do. :-/

Thanks for the diagram anyways which is good to know in case, we have to get back to it. Maybe you can find a more practical and straight forward solution till then, to really fix the, in our view, broken approach here, where you are using a swiss knife* where a simple one would suffice.

Kind regards Andreas :)

*: as one could see in the reviews it admittedly has it advantages ;)

[carlospzurita]

Dear @o2idev

We are closing this issue for now, as you have stopped the deployment of the validator. We will keep improving the solution to make the validator deployment accessible for everyone. Feel free to reopen it whenever you can return to work on this.

Best.

[o2idev]

That's fine and thanks for all your work and time so far! :)