I'm trying to run ml-hub with ml-workspace but having problems starting the spawner. The hub is up and running, logging in users (ldap) is working fine but it fails when i start the spawner.
The setup is pretty much standard except that
i'm using podman instead of docker (tried both running as root and rootless)
overriding the jupyterhub_config.py to be able to modify the function that renames the container (not working in podman)
Technical
Log output
Starting ML Hub
No certificate was provided for SSL/HTTPS.
Generate self-signed certificate for SSL/HTTPS.
Start SSH Daemon service
Start JupyterHub
Warning: If you want to use the SSH feature, you have to start the hub with ssl enabled.
Start nginx
[I 2021-11-02 06:10:54.127 JupyterHub app:2120] Using Authenticator: ldapauthenticator.ldapauthenticator.LDAPAuth enticator-1.3.2
[I 2021-11-02 06:10:54.128 JupyterHub app:2120] Using Spawner: mlhubspawner.mlhubspawner.MLHubDockerSpawner
[I 2021-11-02 06:10:54.136 JupyterHub app:1257] Loading cookie_secret from /data/jupyterhub_cookie_secret
[D 2021-11-02 06:10:54.148 JupyterHub app:1424] Connecting to db: sqlite:////data/jupyterhub.sqlite
[D 2021-11-02 06:10:54.184 JupyterHub orm:749] database schema version found: 4dc2d5a8c53c
[I 2021-11-02 06:10:54.198 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2021-11-02 06:10:54.253 JupyterHub app:1563] Not using whitelist. Any authenticated user will be allowed.
[D 2021-11-02 06:10:54.368 JupyterHub app:1910] Loading state for xxx001 from db
[D 2021-11-02 06:10:54.369 JupyterHub app:1910] Loading state for admin from db
[D 2021-11-02 06:10:54.369 JupyterHub app:1926] Loaded users:
xxx001 admin
admin admin
[I 2021-11-02 06:10:54.382 JupyterHub app:2337] Hub API listening on http://0.0.0.0:8081/hub/
[I 2021-11-02 06:10:54.383 JupyterHub app:2339] Private Hub API connect url http://188f0675ac42:8081/hub/
[W 2021-11-02 06:10:54.400 JupyterHub proxy:642] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
[I 2021-11-02 06:10:54.400 JupyterHub proxy:645] Starting proxy @ http://:8000
[D 2021-11-02 06:10:54.400 JupyterHub proxy:646] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '80 00', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://188f0675ac42:8081/hub/error']
[D 2021-11-02 06:10:54.426 JupyterHub proxy:561] Writing proxy pid file: jupyterhub-proxy.pid
06:10:54.850 [ConfigProxy] info: Proxying http://*:8000 to (no default)
06:10:54.852 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2021-11-02 06:10:55.331 JupyterHub proxy:681] Proxy started and appears to be up
[I 2021-11-02 06:10:55.331 JupyterHub app:2362] Starting managed service cleanup-service at http://127.0.0.1:9000
[I 2021-11-02 06:10:55.332 JupyterHub service:316] Starting service 'cleanup-service': ['/usr/bin/python3', '/res ources/cleanup-service.py']
[I 2021-11-02 06:10:55.339 JupyterHub service:121] Spawning /usr/bin/python3 /resources/cleanup-service.py
[D 2021-11-02 06:10:55.351 JupyterHub spawner:1084] Polling subprocess every 30s
WARNING:tornado.access:404 GET /services/cleanup-service/ (127.0.0.1) 0.55ms
[D 2021-11-02 06:10:57.353 JupyterHub utils:218] Server at http://127.0.0.1:9000/services/cleanup-service/ respon ded with 404
[D 2021-11-02 06:10:57.353 JupyterHub proxy:314] Fetching routes to check
[D 2021-11-02 06:10:57.354 JupyterHub proxy:765] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2021-11-02 06:10:57.361 JupyterHub proxy:319] Checking routes
[I 2021-11-02 06:10:57.361 JupyterHub proxy:399] Adding default route for Hub: / => http://188f0675ac42:8081
[W 2021-11-02 06:10:57.362 JupyterHub proxy:373] Adding missing route for cleanup-service (Server(url=http://127. 0.0.1:9000/services/cleanup-service/, bind_url=http://127.0.0.1:9000/services/cleanup-service/))
[D 2021-11-02 06:10:57.363 JupyterHub proxy:765] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
[I 2021-11-02 06:10:57.364 JupyterHub proxy:242] Adding service cleanup-service to proxy /services/cleanup-servic e/ => http://127.0.0.1:9000
06:10:57.372 [ConfigProxy] info: 200 GET /api/routes
[D 2021-11-02 06:10:57.373 JupyterHub proxy:765] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/services/c leanup-service
06:10:57.375 [ConfigProxy] info: Adding route / -> http://188f0675ac42:8081
06:10:57.376 [ConfigProxy] info: Route added / -> http://188f0675ac42:8081
06:10:57.376 [ConfigProxy] info: 201 POST /api/routes/
06:10:57.377 [ConfigProxy] info: Adding route /services/cleanup-service -> http://127.0.0.1:9000
06:10:57.377 [ConfigProxy] info: Route added /services/cleanup-service -> http://127.0.0.1:9000
06:10:57.377 [ConfigProxy] info: 201 POST /api/routes/services/cleanup-service
[I 2021-11-02 06:10:57.378 JupyterHub app:2422] JupyterHub is now running at http://:8000
[I 2021-11-02 06:11:16.425 JupyterHub log:174] 302 GET / -> /hub/ (@::ffff:10.162.2.18) 1.59ms
[D 2021-11-02 06:11:16.454 JupyterHub base:289] Refreshing auth for xxx001
[I 2021-11-02 06:11:16.455 JupyterHub log:174] 302 GET /hub/ -> /hub/home (xxx001@::ffff:10.162.2.18) 13.83ms
[D 2021-11-02 06:11:16.522 JupyterHub user:240] Creating <class 'mlhubspawner.mlhubspawner.MLHubDockerSpawner'> f or xxx001:
[I 2021-11-02 06:11:17.062 JupyterHub log:174] 200 GET /hub/home (xxx001@::ffff:10.162.2.18) 577.46ms
[D 2021-11-02 06:11:17.206 JupyterHub log:174] 200 GET /hub/static/js/home.js?v=20211102061054 (@::ffff:10.162.2. 18) 1.56ms
[I 2021-11-02 06:11:17.238 JupyterHub log:174] 200 GET /hub/api/users (xxx001@::ffff:10.162.2.18) 29.34ms
[D 2021-11-02 06:11:17.243 JupyterHub log:174] 200 GET /hub/static/js/jhapi.js?v=20211102061054 (@::ffff:10.162.2 .18) 1.67ms
[D 2021-11-02 06:11:17.244 JupyterHub log:174] 200 GET /hub/static/js/utils.js?v=20211102061054 (@::ffff:10.162.2 .18) 0.67ms
[D 2021-11-02 06:11:17.246 JupyterHub log:174] 200 GET /hub/static/components/moment/moment.js?v=20211102061054 ( @::ffff:10.162.2.18) 6.70ms
[D 2021-11-02 06:11:20.565 JupyterHub log:174] 304 GET /hub/home (xxx001@::ffff:10.162.2.18) 64.74ms
[I 2021-11-02 06:11:20.697 JupyterHub log:174] 200 GET /hub/api/users (xxx001@::ffff:10.162.2.18) 41.45ms
[I 2021-11-02 06:11:23.437 JupyterHub login:43] User logged out: xxx001
[I 2021-11-02 06:11:23.480 JupyterHub log:174] 302 GET /hub/logout -> /hub/login (xxx001@::ffff:10.162.2.18) 54.6 4ms
[I 2021-11-02 06:11:23.515 JupyterHub log:174] 200 GET /hub/login (@::ffff:10.162.2.18) 6.75ms
[D 2021-11-02 06:11:33.389 JupyterHub ldapauthenticator:379] Attempting to bind xxx001with CN=xxx001,OU=Users,OU =People,OU=SE,OU=***,DC=******,DC=***,DC=biz
[D 2021-11-02 06:11:34.079 JupyterHub ldapauthenticator:392] Status of user bind xxx001with CN=xxx001,OU=Users,O U=People,OU=SE,OU=***,DC=******,DC=***,DC=biz : True
[D 2021-11-02 06:11:34.084 JupyterHub base:482] Setting cookie for xxx001: jupyterhub-services
[D 2021-11-02 06:11:34.084 JupyterHub base:478] Setting cookie jupyterhub-services: {'httponly': True, 'path': '/ services'}
[D 2021-11-02 06:11:34.085 JupyterHub base:478] Setting cookie jupyterhub-session-id: {'httponly': True}
[D 2021-11-02 06:11:34.085 JupyterHub base:482] Setting cookie for xxx001: jupyterhub-hub-login
[D 2021-11-02 06:11:34.085 JupyterHub base:478] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': ' /hub/'}
[I 2021-11-02 06:11:34.085 JupyterHub base:663] User logged in: xxx001
[I 2021-11-02 06:11:34.086 JupyterHub log:174] 302 POST /hub/login?next= -> /hub/home (xxx001@::ffff:10.162.2.18) 698.01ms
[D 2021-11-02 06:11:34.471 JupyterHub log:174] 304 GET /hub/home (xxx001@::ffff:10.162.2.18) 364.68ms
[I 2021-11-02 06:11:34.587 JupyterHub log:174] 200 GET /hub/api/users (xxx001@::ffff:10.162.2.18) 18.62ms
[D 2021-11-02 06:11:51.696 JupyterHub pages:165] Triggering spawn with default options for xxx001
[D 2021-11-02 06:11:51.696 JupyterHub base:780] Initiating spawn for xxx001
[D 2021-11-02 06:11:51.697 JupyterHub base:787] 0/100 concurrent spawns
[D 2021-11-02 06:11:51.697 JupyterHub base:792] 0 active servers
[D 2021-11-02 06:11:51.718 JupyterHub user:542] Calling Spawner.start for xxx001
[I 2021-11-02 06:11:51.933 JupyterHub mlhubspawner:283] Create network mlhub-xxx001 with subnet 172.33.1.0/24
[E 2021-11-02 06:11:51.949 JupyterHub mlhubspawner:298] Could not connect mlhub to the network and, thus, cannot create the container.
[D 2021-11-02 06:11:52.012 JupyterHub dockerspawner:811] Getting container 'ws-xxx001-mlhub'
[I 2021-11-02 06:11:52.016 JupyterHub dockerspawner:818] Container 'ws-xxx001-mlhub' is gone
[I 2021-11-02 06:11:52.056 JupyterHub mlhubspawner:264] Network mlhub-xxx001 already exists
[E 2021-11-02 06:11:52.060 JupyterHub mlhubspawner:298] Could not connect mlhub to the network and, thus, cannot create the container.
[D 2021-11-02 06:11:52.089 JupyterHub dockerspawner:907] Starting host with config: {'binds': {}, 'links': {}, 'network_mode': 'mlhub-xxx001'}
[I 2021-11-02 06:11:52.161 JupyterHub dockerspawner:1028] Created container ws-xxx001-mlhub (id: 609b2b5) from image localhost/dlab_mlworkspace
[I 2021-11-02 06:11:52.161 JupyterHub dockerspawner:1051] Starting container ws-xxx001-mlhub (id: 609b2b5)
[D 2021-11-02 06:11:52.680 JupyterHub spawner:1084] Polling subprocess every 30s
[I 2021-11-02 06:11:52.726 JupyterHub log:174] 302 GET /hub/spawn/xxx001 -> /hub/spawn-pending/xxx001 (xxx001@::ffff:10.162.2.18) 1035.21ms
[I 2021-11-02 06:11:52.831 JupyterHub pages:303] xxx001 is pending spawn
[I 2021-11-02 06:11:52.836 JupyterHub log:174] 200 GET /hub/spawn-pending/xxx001 (xxx001@::ffff:10.162.2.18) 74.55ms
[W 2021-11-02 06:11:56.881 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
[W 2021-11-02 06:11:57.031 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
[D 2021-11-02 06:11:57.385 JupyterHub app:1812] Managed service cleanup-service running at http://127.0.0.1:9000
[W 2021-11-02 06:12:01.478 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
[D 2021-11-02 06:12:01.701 JupyterHub dockerspawner:811] Getting container 'ws-xxx001-mlhub'
[D 2021-11-02 06:12:01.715 JupyterHub dockerspawner:796] Container 609b2b5 status: {'Dead': False,
'Error': '',
'ExitCode': 0,
'FinishedAt': '0001-01-01T00:00:00Z',
'Health': {'FailingStreak': 0, 'Log': None, 'Status': ''},
'OOMKilled': False,
'Paused': False,
'Pid': 17099,
'Restarting': False,
'Running': True,
'StartedAt': '2021-11-02T06:11:52.563936822Z',
'Status': 'running'}
[W 2021-11-02 06:12:01.715 JupyterHub base:932] User xxx001 is slow to become responsive (timeout=10)
[D 2021-11-02 06:12:01.715 JupyterHub base:937] Expecting server for xxx001 at: http://172.33.1.2:8080/user/xxx001/
[W 2021-11-02 06:12:06.486 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
[W 2021-11-02 06:12:11.493 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
[W 2021-11-02 06:12:16.498 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
[W 2021-11-02 06:12:21.502 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
Config
"""
Basic configuration file for jupyterhub.
"""
import os
import re
import signal
import socket
import sys
import docker.errors
import json
from traitlets.log import get_logger
logger = get_logger()
from mlhubspawner import utils
from subprocess import call
c = get_config()
c.Application.log_level = 'DEBUG'
c.Spawner.debug = True
# Override the Jupyterhub `normalize_username` function to remove problematic characters from the username - independent from the used authenticator.
# E.g. when the username is "lastname, firstname" and the comma and whitespace are not removed, they are encoded by the browser, which can lead to brok$
# especially for the tools-part.
# Everybody who starts the hub can override this behavior the same way we do in a mounted `jupyterhub_user_config.py` (Docker local) or via the `hub.ex$
from jupyterhub.auth import Authenticator
original_normalize_username = Authenticator.normalize_username
def custom_normalize_username(self, username):
username = original_normalize_username(self, username)
more_than_one_forbidden_char = False
for forbidden_username_char in [" ", ",", ";", ".", "-", "@", "_"]:
# Replace special characters with a non-special character. Cannot just be empty, like "", because then it could happen that two distinct user n$
# Example: "foo, bar" and "fo, obar" would both become "foobar".
replace_char = "0"
# If there is more than one special character, just replace one of them. Otherwise, "foo, bar" would become "foo00bar" instead of "foo0bar"
if more_than_one_forbidden_char == True:
replace_char = ""
temp_username = username
username = username.replace(forbidden_username_char, replace_char, 1)
if username != temp_username:
more_than_one_forbidden_char = True
return username
Authenticator.normalize_username = custom_normalize_username
original_check_whitelist = Authenticator.check_whitelist
def dynamic_check_whitelist(self, username, authentication=None):
dynamic_whitelist_file = "/resources/users/dynamic_whitelist.txt"
if os.getenv("DYNAMIC_WHITELIST_ENABLED", "false") == "true":
# TODO: create the file and warn the user that the user has to go into the hub pod and modify it there
if not os.path.exists(dynamic_whitelist_file):
logger.error("The dynamic white list has to be mounted to '{}'. Use standard JupyterHub whitelist behavior.".format(dynamic_whitelist_file))
else:
with open(dynamic_whitelist_file, "r") as f:
#whitelisted_users = f.readlines()
# rstrip() will remove trailing whitespaces or newline characters
whitelisted_users = [line.rstrip() for line in f]
return username.lower() in whitelisted_users
return original_check_whitelist(self, username, authentication)
Authenticator.check_whitelist = dynamic_check_whitelist
### Helper Functions ###
def get_or_init(config: object, config_type: type) -> object:
if not isinstance(config, config_type):
return config_type()
return config
def combine_config_dicts(*configs) -> dict:
combined_config = {}
for config in configs:
if not isinstance(config, dict):
config = {}
combined_config.update(config)
return combined_config
### END HELPER FUNCTIONS###
ENV_NAME_HUB_NAME = 'HUB_NAME'
ENV_HUB_NAME = os.environ[ENV_NAME_HUB_NAME]
ENV_EXECUTION_MODE = os.environ[utils.ENV_NAME_EXECUTION_MODE]
# User containers will access hub by container name on the Docker network
c.JupyterHub.hub_ip = '0.0.0.0' #'research-hub
c.JupyterHub.port = 8000
# Persist hub data on volume mounted inside container
# TODO: should really be persisted?
data_dir = os.environ.get('DATA_VOLUME_CONTAINER', '/data')
if not os.path.exists(data_dir):
os.makedirs(data_dir)
c.JupyterHub.cookie_secret_file = os.path.join(data_dir, 'jupyterhub_cookie_secret')
c.JupyterHub.db_url = os.path.join(data_dir, 'jupyterhub.sqlite')
c.JupyterHub.admin_access = True
# prevents directly opening your workspace after login
c.JupyterHub.redirect_to_server=False
c.JupyterHub.allow_named_servers = True
c.Spawner.port = int(os.getenv("DEFAULT_WORKSPACE_PORT", 8080))
# Set default environment variables used by our ml-workspace container
default_env = {"AUTHENTICATE_VIA_JUPYTER": "true", "SHUTDOWN_INACTIVE_KERNELS": "true"}
# Workaround to prevent api problems
#c.Spawner.will_resume = True
# --- Spawner-specific ----
c.JupyterHub.spawner_class = 'mlhubspawner.MLHubDockerSpawner' # override in your config if you want to have a different spawner. If it is the or inher$
#c.Spawner.image = "mltooling/ml-workspace:0.8.7"
#c.Spawner.workspace_images = [c.Spawner.image, "mltooling/ml-workspace-gpu:0.8.7", "mltooling/ml-workspace-r:0.8.7", "mltooling/ml-workspace-spark:0.8$
c.Spawner.image = 'localhost/dlab_mlworkspace'
c.Spawner.workspace_images = [c.Spawner.image]
c.Spawner.notebook_dir = '/workspace'
# Connect containers to this Docker network
c.Spawner.use_internal_ip = True
c.Spawner.prefix = 'ws'
c.Spawner.name_template = c.Spawner.prefix + '-{username}-' + ENV_HUB_NAME + '{servername}' # override in your config when you want to have a different$
# Don't remove containers once they are stopped - persist state
c.Spawner.remove_containers = False
c.Spawner.start_timeout = 600 # should remove errors related to pulling Docker images (see https://github.com/jupyterhub/dockerspawner/issues/293)
c.Spawner.http_timeout = 120
# --- Authenticator ---
c.Authenticator.admin_users = {"xxx001"} # override in your config when needed, for example if you use a different authenticator (e.g. set Github usern$
# Forbid user names that could collide with a named server (check ) to prevent security & routing problems
c.Authenticator.username_pattern = '^((?!-hub).)*$'
NATIVE_AUTHENTICATOR_CLASS = 'nativeauthenticator.NativeAuthenticator'
#c.JupyterHub.authenticator_class = NATIVE_AUTHENTICATOR_CLASS # override in your config if you want to use a different authenticator
c.JupyterHub.authenticator_class = 'ldapauthenticator.LDAPAuthenticator'
c.LDAPAuthenticator.server_address = 'ldap://***.***.**'
c.LDAPAuthenticator.lookup_dn = False
c.LDAPAuthenticator.bind_dn_template = [
'CN={username},OU=Users,OU=People,OU=SE,OU=***,DC=******,DC=***,DC=biz',
'CN={username},OU=Impersonal Accounts,OU=Admin,OU=Central,OU=***,DC=******,DC=***,DC=biz',
]
# --- Load user config ---
# Allow passing an additional config upon mlhub container startup.
# Enables the user to override all configurations occurring above the load_subconfig command; be careful to not break anything ;)
# An empty config file already exists in case the user does not mount another config file.
# The extra config could look like:
# jupyterhub_user_config.py
# > c = get_config()
# > c.DockerSpawner.extra_create_kwargs.update({'labels': {'foo': 'bar'}})
# See https://traitlets.readthedocs.io/en/stable/config.html#configuration-files-inheritance
load_subconfig("{}/jupyterhub_user_config.py".format(os.getenv("_RESOURCES_PATH")))
c.Spawner.environment = get_or_init(c.Spawner.environment, dict)
c.Spawner.environment.update(default_env)
service_environment = {
ENV_NAME_HUB_NAME: ENV_HUB_NAME,
utils.ENV_NAME_EXECUTION_MODE: ENV_EXECUTION_MODE,
utils.ENV_NAME_CLEANUP_INTERVAL_SECONDS: os.getenv(utils.ENV_NAME_CLEANUP_INTERVAL_SECONDS),
}
# shm_size can only be set for Docker, not Kubernetes (see https://stackoverflow.com/questions/43373463/how-to-increase-shm-size-of-a-kubernetes-contai$
#c.Spawner.extra_host_config = { 'shm_size': '256m' }
client_kwargs = {**get_or_init(c.Spawner.client_kwargs, dict)} # {**get_or_init(c.DockerSpawner.client_kwargs, dict), **get_or_init(c.MLHubDockerSpawne$
tls_config = {**get_or_init(c.Spawner.tls_config, dict)} # {**get_or_init(c.DockerSpawner.tls_config, dict), **get_or_init(c.MLHubDockerSpawner.tls_con$
#docker_client = utils.init_docker_client(client_kwargs, tls_config)
#try:
# container = docker_client.containers.list(filters={"id": socket.gethostname()})[0]
# if container.name.lower() != ENV_HUB_NAME.lower():
# container.rename(ENV_HUB_NAME.lower())
#except docker.errors.APIError as e:
# logger.error("Could not correctly start MLHub container. " + str(e))
# os.kill(os.getpid(), signal.SIGTERM)
# For cleanup-service
service_environment.update({"DOCKER_CLIENT_KWARGS": json.dumps(client_kwargs), "DOCKER_TLS_CONFIG": json.dumps(tls_config)})
service_host = "127.0.0.1"
# Consider the case where the user-config contains c.DockerSpawner.environment instead of c.Spawner.environment
# c.DockerSpawner.environment = get_or_init(c.DockerSpawner.environment, dict)
# c.Spawner.environment.update(c.DockerSpawner.environment)
#c.MLHubDockerSpawner.hub_name = ENV_HUB_NAME
# Add nativeauthenticator-specific templates
if c.JupyterHub.authenticator_class == NATIVE_AUTHENTICATOR_CLASS:
import nativeauthenticator
# if template_paths is not set yet in user_config, it is of type traitlets.config.loader.LazyConfigValue; in other words, it was not initialized yet
c.JupyterHub.template_paths = get_or_init(c.JupyterHub.template_paths, list)
# if not isinstance(c.JupyterHub.template_paths, list):
# c.JupyterHub.template_paths = []
c.JupyterHub.template_paths.append("{}/templates/".format(os.path.dirname(nativeauthenticator.__file__)))
# TODO: add env variable to readme
if (os.getenv("IS_CLEANUP_SERVICE_ENABLED", "true") == "true"):
c.JupyterHub.services = [
{
'name': 'cleanup-service',
'admin': True,
'url': 'http://{}:9000'.format(service_host),
'environment': service_environment,
'command': [sys.executable, '/resources/cleanup-service.py']
}
]
Hub version mltooling/ml-workspace-minimal:0.13.2, mltooling/ml-hub:1.0.0
I'm trying to run ml-hub with ml-workspace but having problems starting the spawner. The hub is up and running, logging in users (ldap) is working fine but it fails when i start the spawner.
The setup is pretty much standard except that
Log output
Config