Exception: Connection failed. (docker instance v2023.3)

[opeeters]

We have a local installation of the INSPIRE Reference Validator (v2023.3 ) with direct access to the internet, installed as a docker container (cf https://github.com/INSPIRE-MIF/helpdesk-validator/releases/tag/v2023.3). Every now and again this instance cannot find certain resources during the validation and failing the test of INSPIRE resources that are valid (see: log_231103_fail.txt).

After a couple of weeks the issue disappears on it's own and all works as expected. The same resource (also passing the reference installation at https://inspire.ec.europa.eu/validator) is correctly evaluated as being valid.

Let's take this resource as an example:

03.11.2023 08:29:15 - "Checking URL: 'http://inspire.ec.europa.eu/metadata-codelist/PriorityDataset/PriorityDataset.bg.atom'"
03.11.2023 08:29:15 - "Exception: Connection failed. URL: http://inspire.ec.europa.eu/metadata-codelist/PriorityDataset/PriorityDataset.bg.atom"
03.11.2023 08:29:15 - "Test Assertion 'M.2: Priority datasets keyword': FAILED - 2 ms"

When you try reaching http://inspire.ec.europa.eu/metadata-codelist/PriorityDataset/PriorityDataset.bg.atom with curt -L (following redirects) the resource is retrieved (also from within the container). With curl -O I get a 301 Moved Permanently. This is the only lead I have at the moment, but I'm not sure yet whether this might be different when the validator works as expected since our instance has been misbehaving all week.

[DanielMartinPerezdeLeon]

Dear @opeeters ,

Please note that the URL you are trying to connect to in the example resource is 'http://inspire.ec.europa.eu/metadata-codelist/PriorityDataset/PriorityDataset.bg.atom'. Since the URLs for retrieving resources have been updated to use only the HTTPS protocol, that URL redirects you to 'https://inspire.ec.europa.eu/metadata-codelist/PriorityDataset/PriorityDataset.bg.atom'.

Java does not allow redirections from HTTP to HTTPS due to security constraints, and therefore, the ETF cannot resolve this when validating an endpoint with an HTTP to HTTPS redirection.

It is recommended, whenever possible, to use HTTPS URLs instead of HTTP in your resources.

To address this redirection issue, the INSPIRE Reference Validator employs a combination of Apache and the Squid proxy service within the Docker instance to automatically redirect all these URLs to their HTTPS versions.

Considering the description of the issue, it's possible that Squid or Apache have suddenly stopped working or is unable to start.

We kindly request that you check the availability of Squid and Apache (to ensure it is up and running) and then attempt to run the validation process again.

If this problem persists or if you have any further questions, please do not hesitate to contact us.

Regards,

[opeeters]

Ok, thanks a million @DanielMartinPerezdeLeon This does indeed explain the issue!

I did:

docker exec -it etfwebapp service squid status

Which revealed that squid was not running.

I did:

docker exec -it etfwebapp service squid start

Checking squid status and the validator worked again as expected, finding also the HTTP resource thanks to the internal redirect.

Strange that squid sometimes just seems to die. I will try to monitor this. I did do multiple docker restarts before, but this did not solve the issue.. So squid apparently also sometimes does not come up during restarts.

Kind regards!

[opeeters]

For the record, the status of apache can be checked with:

docker exec -it etfwebapp service apache2 status

[DanielMartinPerezdeLeon]

Dear @opeeters ,

You are welcome, we are pleased that we could assist with the problem. Indeed, we also encounter problems with Squid from time to time. To resolve this, we usually employ a cron job to restart the Squid service once a day.

Please do not hesitate to contact us regarding this issue or any future ones.

Regards,

[fabiovinci]

Dear @opeeters,

since the issue seems solved, I will convert it into a discussion so that the solution is visible to other users.