Issue with Docker ETF behind the proxy

[MartinTuchyna]

Dear Validator Helpdesk Team

We need to deploy containerized etf on host which is not directly connected to internet i.e. it uses forward proxy (corporate squid instance) to access internet and simultaneosly etf web UI/API would be accessed by client from internet via reverse proxy (nginx/haproxy instance). We find using etf docker container in this deployment scenario very difficult mainly because of its complicated architecture/structure (considering all the stuff running within single container i.e apache, squid, nginx).

Neverthless few months ago we came with working solution, which was working until latest etf docker image release 2022.03 (relased in September), working solution is attached to this email and is comprised of several files:

• docker-compose.yaml
• proxy.conf (overriden Apache condfiguration /etc/apache2/sites-available/proxy.conf) - the only modification here is to tell apache to use corporate forward proxy for outgoing http traffic
• squid.conf (overriden Squid configuration /etc/squid3/squid.conf) - force squid to use our corporate forward proxy (also squid) for all the outgoing http[s] traffic except traffic destined to "inspire.ec.europa.eu" domain cause that is handled by internal apache (modifications are located at the very end of file)
• config.js (overriden JS file in validator web UI app /etf/validator/js/config.js) - overide default front facing URL to let JS construct proper self referencing urls (we are behind reverse proxy)

As was said this solution was working until latest release of etf docker image 2022.03, so please check our configuraion and let us know if something has changed in latest docker image that could have "breaking" effect with regard to our configuration ...

etf.zip

[fernandocuevasgdtel]

Dear @MartinTuchyna,

we have analyzed the data provided and it seems that in the config.js file the TimeUpMessage var is missing. It's a string that you can leave empty but it needs to be there. The absence of it may lead to a crash in the test reports page, and it could be the cause of your error. Please make sure you are using alligned versions of the UI source code and the configuration files.

In case it doesn't works, could you share with us which is the breaking effect that it has?

Any additional information that you could provide us with is very helpful to try to provide you with feedback for this issue.

Regards,

[jenriquesoriano]

Dear @joachimnielandt ,

did you have the chance to check whether if you are using alligned versions of the UI source code and the configuration files? Please let us know if it worked or if you found additional issues.

Thanks!

[joachimnielandt]

Dear @jenriquesoriano , in my current setup I am using two configuration files on top of the base image:

• etf-config.properties
• config.js

I have just updated to 2023.2 and compared the default files with my own: they are aligned. I'll attach them here, for reference. Question: the serverToken was supplied in the default setup - should this be set to a fixed / private value?

etc-config.properties

#########################################
# ETF CONFIGURATION PROPERTY FILE
#########################################

# Version of the property path, used for backward compatibility.
# (do not modify unless you know what you are doing)
etf.config.properties.version = 2

#########################################
## General properties
#########################################

# Modify the base URL which points to the deployed web application.
# The URL is used to reference this ETF instance from the Test Reports.
# Example: http://yourserver/etf-webapp
etf.webapp.base.url = https://metadata.dev-vlaanderen.be/inspirevalidator

# Restrict cross-origin HTTP requests for browsers, see
# https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS
# for more information.
# Default: taken from etf.webapp.base.url
etf.api.allow.origin = *

# An optional branding text which is shown in the header
etf.branding.text = Metadata Vlaanderen - INSPIRE Reference Validator

# Max file upload size for test data in bytes.
# By default the max upload size is estimated based on presumable free memory
# during the start of the application.
# Hint: 524288000 bytes are 500 MB
# Default: auto
etf.max.upload.size = 52428800

# Maximum size of the Test Object data in bytes.
# As compressed files can be uploaded and the maximum file upload size
# is different from the size of the extracted test files, this property
# ensures that the specified limit is not exceeded.
# Default: 5368709120 (5 GB)
# etf.testobject.max.size = 5368709120

# Expiration time for temporary Test Objects. After the time has elapsed,
# the test objects and test data are deleted.
# To deactivate this feature and keep all Test Objects, the property
# must be changed to the value 0.
# Default: 480 (8 hours)
# etf.testobject.temporary.lifetime.expiration = 480

# Expiration time for Test Reports. After the time has elapsed,
# the reports are deleted.
# To deactivate this feature and keep all Test Reports, the property
# must be changed to the value 0.
# Default: 11520 (8 days)
etf.testreports.lifetime.expiration = 4320

# Allow access to private networks. Checked when
# a new webservice test object is created. "false" is recommended in public
# deployments.
# Default: false
# etf.testobject.allow.privatenet.access = false

# Recreate the internal BaseX configuration file (.basex) on startup. "true" is
# recommended.
# Default: true
# etf.bsx.recreate.config = true

# Possible values default/simplified . A simplified workflow enables the user
# to start a test run without creating a test object.
# Default: simplified
# etf.workflows = simplified

# Maximum number of tests which can run in parallel.
# Default: auto (the number of CPU cores of this machine)
# etf.testruns.threads.max = auto

# Size of the task pool queue
# Default: auto (three times the parameter etf.testruns.threads.max)
# etf.testruns.queued.max = auto

##################################################################################
## Contact, legal notice, privacy statement
##################################################################################
#
# Please note: you can add multiple lines by using the \ character at the end
# of the line.
#

# Contact
# etf.meta.contact.text = <p>Operator A</p> \
#                    <p>X Avenue</p> \
#                    <p>Fort Lauderdale</p> \
#                    <p>Broward County</p> \
#                    <p>Florida, USA</p> \
#                    <br>\
#                    <p>Email: nielsen@a.com</p>

# Disclaimer shown under Legal notice
# etf.meta.legalnotice.disclaimer.text = Disclaimer ...

# Copyright notice only shown under Legal notice if Disclaimer is set!
# etf.meta.legalnotice.copyrightnotice.text = Copyright notice ...

# Privacy statement
# etf.meta.privacystatement.text = Privacy statement ...

#########################################
## Directory properties
#########################################

# For all directories, paths can be set that are either absolute or
# relative to the etf.dir. If the etf.dir is not specified it is
# set to ../ relative to this configuration file.

# Optional base path
# Default: ../
# etf.dir=/data/etf

# Directory for the test projects
# Default: projects
# etf.projects.dir = projects

# Directory which contains the reports styles
# Default: reportstyles
# etf.reportstyles.dir = reportstyles

# Directory which contains the test drivers
# Default: td
# etf.testdrivers.dir = td

# Directory that the application uses for text data tests
# Default: testdata
# etf.testdata.dir = testdata

# Directory for the internal data source
# Default: ds
# etf.datasource.dir = ds

# Directory that the application uses to store uploaded files
# Default: http_uploads
# etf.testdata.upload.dir = http_uploads

# Directory that the application uses to backup files
# Default: bak
# etf.backup.dir = bak

#TEAM Engine credentials of your organization in order to properly use TEAM Engine remote calls
etf.testdrivers.teamengine.url = http://cite.opengeospatial.org/teamengine
etf.testdrivers.teamengine.username = abc
etf.testdrivers.teamengine.password = abc

config.js

var captchaEnabled = false;
var betaBanner = false;
var labelStaging = false;
var environment = "PROD";
// PROD
var serverURL = "https://metadata.dev-vlaanderen.be/inspirevalidator/v2/";
var serverRealURL = "https://metadata.dev-vlaanderen.be/inspirevalidator/v2/";
var serverDirectURL = "https://metadata.dev-vlaanderen.be/inspirevalidator/v2/";
var serverCaptchaURL = "https://metadata.dev-vlaanderen.be/inspirevalidator/captcha/verify;
var serverToken = "BQIAJulY249NyeiT6gsfc9AUTCa8Nkgi72J9jIVP";
var swaggerURL = "https://metadata.dev-vlaanderen.be/inspirevalidator/swagger-ui.html";
var timeUpMessage="";