'ERROR TLS requested but server does not support TLS ' while trying to use orchestrator without TLS

[sprutner]

Hi there, new Orchestrator user. When I am try to use discover on a MySQL Master, I am getting the following output:

./orchestrator -c discover -i 10.1.1.151:3306
2018-07-16 22:11:32 DEBUG Hostname unresolved yet: 10.1.1.151
2018-07-16 22:11:32 DEBUG Cache hostname resolve 10.1.1.151 as 10.1.1.151
2018-07-16 22:11:32 DEBUG Connected to orchestrator backend: sqlite on /usr/local/orchestrator/orchestrator.sqlite3
2018-07-16 22:11:32 DEBUG Initializing orchestrator
2018-07-16 22:11:32 INFO Connecting to backend orchestrator-mysql:3306: maxConnections: 128, maxIdleConns: 32
2018-07-16 22:11:32 DEBUG WriteResolvedHostname: resolved 10.1.1.151 to 10.1.1.151
2018-07-16 22:11:32 ERROR TLS requested but server does not support TLS
2018-07-16 22:11:32 ERROR ReadTopologyInstance(10.1.1.151:3306) show variables like 'maxscale%': TLS requested but server does not support TLS
2018-07-16 22:11:32 ERROR ReadTopologyInstance(10.1.1.151:3306) show global status like 'Uptime': TLS requested but server does not support TLS
2018-07-16 22:11:33 FATAL TLS requested but server does not support TLS
/usr/local/orchestrator #

I can log in no problem with the TopologyCredentials specified.

My orchestrator.config.json seems like it isn't going to try to attempt TLS. What am I missing here?

{
  "Debug": true,
  "EnableSyslog": false,
  "ListenAddress": ":3000",
  "MySQLTopologyCredentialsConfigFile": "/usr/local/orchestrator/topology-creds",
  "MySQLTopologySSLPrivateKeyFile": "",
  "MySQLTopologySSLCertFile": "",
  "MySQLTopologySSLCAFile": "",
  "MySQLTopologySSLSkipVerify": true,
  "MySQLTopologyUseMutualTLS": false,
  "MySQLOrchestratorHost": "orchestrator-mysql",
  "MySQLOrchestratorPort": 3306,
  "MySQLOrchestratorDatabase": "orchestrator",
  "MySQLOrchestratorCredentialsConfigFile": "/usr/local/orchestrator/basic-creds",
  "MySQLOrchestratorSSLPrivateKeyFile": "",
  "MySQLOrchestratorSSLCertFile": "",
  "MySQLOrchestratorSSLCAFile": "",
  "MySQLOrchestratorSSLSkipVerify": true,
  "MySQLOrchestratorUseMutualTLS": false,
  "MySQLConnectTimeoutSeconds": 1,
  "DefaultInstancePort": 3306,
  "DiscoverByShowSlaveHosts": true,
  "InstancePollSeconds": 5,
  "UnseenInstanceForgetHours": 240,
  "SnapshotTopologiesIntervalHours": 0,
  "InstanceBulkOperationsWaitTimeoutSeconds": 10,
  "HostnameResolveMethod": "default",
  "MySQLHostnameResolveMethod": "@@hostname",
  "SkipBinlogServerUnresolveCheck": true,
  "ExpiryHostnameResolvesMinutes": 60,
  "RejectHostnameResolvePattern": "",
  "ReasonableReplicationLagSeconds": 10,
  "ProblemIgnoreHostnameFilters": [],
  "VerifyReplicationFilters": false,
  "ReasonableMaintenanceReplicationLagSeconds": 20,
  "CandidateInstanceExpireMinutes": 60,
  "AuditLogFile": "",
  "AuditToSyslog": false,
  "RemoveTextFromHostnameDisplay": ".mydomain.com:3306",
  "ReadOnly": false,
  "AuthenticationMethod": "",
  "HTTPAuthUser": "",
  "HTTPAuthPassword": "",
  "AuthUserHeader": "",
  "PowerAuthUsers": [
    "*"
  ],
  "ClusterNameToAlias": {
    "127.0.0.1": "test suite"
  },
  "SlaveLagQuery": "",
  "DetectClusterAliasQuery": "SELECT SUBSTRING_INDEX(@@hostname, '.', 1)",
  "DetectClusterDomainQuery": "",
  "DetectInstanceAliasQuery": "",
  "DetectPromotionRuleQuery": "",
  "DataCenterPattern": "[.]([^.]+)[.][^.]+[.]mydomain[.]com",
  "PhysicalEnvironmentPattern": "[.]([^.]+[.][^.]+)[.]mydomain[.]com",
  "PromotionIgnoreHostnameFilters": [],
  "DetectSemiSyncEnforcedQuery": "",
  "ServeAgentsHttp": false,
  "AgentsServerPort": ":3001",
  "AgentsUseSSL": false,
  "AgentsUseMutualTLS": false,
  "AgentSSLSkipVerify": false,
  "AgentSSLPrivateKeyFile": "",
  "AgentSSLCertFile": "",
  "AgentSSLCAFile": "",
  "AgentSSLValidOUs": [],
  "UseSSL": false,
  "UseMutualTLS": false,
  "SSLSkipVerify": false,
  "SSLPrivateKeyFile": "",
  "SSLCertFile": "",
  "SSLCAFile": "",
  "SSLValidOUs": [],
  "URLPrefix": "",
  "StatusEndpoint": "/api/status",
  "StatusSimpleHealth": true,
  "StatusOUVerify": false,
  "AgentPollMinutes": 60,
  "UnseenAgentForgetHours": 6,
  "StaleSeedFailMinutes": 60,
  "SeedAcceptableBytesDiff": 8192,
  "PseudoGTIDPattern": "",
  "PseudoGTIDPatternIsFixedSubstring": false,
  "PseudoGTIDMonotonicHint": "asc:",
  "DetectPseudoGTIDQuery": "",
  "BinlogEventsChunkSize": 10000,
  "SkipBinlogEventsContaining": [],
  "ReduceReplicationAnalysisCount": true,
  "FailureDetectionPeriodBlockMinutes": 60,
  "RecoveryPeriodBlockSeconds": 3600,
  "RecoveryIgnoreHostnameFilters": [],
  "RecoverMasterClusterFilters": [
    "_master_pattern_"
  ],
  "RecoverIntermediateMasterClusterFilters": [
    "_intermediate_master_pattern_"
  ],
  "OnFailureDetectionProcesses": [
    "echo 'Detected {failureType} on {failureCluster}. Affected replicas: {countSlaves}' >> /tmp/recovery.log"
  ],
  "PreGracefulTakeoverProcesses": [
    "echo 'Planned takeover about to take place on {failureCluster}. Master will switch to read_only' >> /tmp/recovery.log"
  ],
  "PreFailoverProcesses": [
    "echo 'Will recover from {failureType} on {failureCluster}' >> /tmp/recovery.log"
  ],
  "PostFailoverProcesses": [
    "echo '(for all types) Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Successor: {successorHost}:{successorPort}' >> /tmp/recovery.log"
  ],
  "PostUnsuccessfulFailoverProcesses": [],
  "PostMasterFailoverProcesses": [
    "echo 'Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Promoted: {successorHost}:{successorPort}' >> /tmp/recovery.log"
  ],
  "PostIntermediateMasterFailoverProcesses": [
    "echo 'Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Successor: {successorHost}:{successorPort}' >> /tmp/recovery.log"
  ],
  "PostGracefulTakeoverProcesses": [
    "echo 'Planned takeover complete' >> /tmp/recovery.log"
  ],
  "CoMasterRecoveryMustPromoteOtherCoMaster": true,
  "DetachLostSlavesAfterMasterFailover": true,
  "ApplyMySQLPromotionAfterMasterFailover": false,
  "MasterFailoverDetachSlaveMasterHost": false,
  "MasterFailoverLostInstancesDowntimeMinutes": 0,
  "PostponeSlaveRecoveryOnLagMinutes": 0,
  "OSCIgnoreHostnameFilters": [],
  "GraphiteAddr": "",
  "GraphitePath": "",
  "GraphiteConvertHostnameDotsToUnderscores": true,
  "ConsulAddress": "",
  "ConsulAclToken": ""
}

[shlomi-noach]

Something is extremely strange with the log output.

It mentions connecting both to a sqlite backend and a MySQL backend:

2018-07-16 22:11:32 DEBUG Connected to orchestrator backend: sqlite on /usr/local/orchestrator/orchestrator.sqlite3
2018-07-16 22:11:32 DEBUG Initializing orchestrator
2018-07-16 22:11:32 INFO Connecting to backend orchestrator-mysql:3306: maxConnections: 128, maxIdleConns: 32

But I don't see a BackendDB nor SQLite3DataFile configuration in your above. Is it possible you have a rogue orchestrator.conf.json file in /etc or in /usr/local/orchestrator?

[sprutner]

I removed a rogue config file the Docker entrypoint was creating, and moved my config file to /etc/orchestrator.conf.json but I am still getting the same issue--minus the sqlite backend stuff:

## Startup: 

2018-07-17 18:30:13 DEBUG Connected to orchestrator backend: orchestrator:?@tcp(orchestrator-mysql:3306)/orchestrator?timeout=1s
2018-07-17 18:30:13 DEBUG Orchestrator pool SetMaxOpenConns: 128
2018-07-17 18:30:13 DEBUG Initializing orchestrator
2018-07-17 18:30:13 DEBUG Migrating database schema
2018-07-17 18:30:18 DEBUG Migrated database schema to version [3.0.11]
2018-07-17 18:30:18 INFO Connecting to backend orchestrator-mysql:3306: maxConnections: 128, maxIdleConns: 32
2018-07-17 18:30:18 INFO Starting Discovery
2018-07-17 18:30:18 INFO Registering endpoints
2018-07-17 18:30:18 INFO continuous discovery: setting up
2018-07-17 18:30:18 INFO continuous discovery: starting
2018-07-17 18:30:18 DEBUG Queue.startMonitoring(DEFAULT)
2018-07-17 18:30:18 INFO Starting HTTP listener on :3000
2018-07-17 18:30:19 INFO Not elected as active node; active node: ; polling
2018-07-17 18:30:21 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
2018-07-17 18:30:22 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
2018-07-17 18:30:23 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
2018-07-17 18:30:24 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
[martini] Started GET / for 100.100.0.0:24876
[martini] Completed 302 Found in 5.213259ms
[martini] Started GET /web/clusters for 100.100.0.0:24878
[martini] Completed 200 OK in 3.876265ms
2018-07-17 18:30:25 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
[martini] Started GET / for 100.100.0.0:24888
[martini] Completed 302 Found in 847.505µs
[martini] Started GET /web/clusters for 100.100.0.0:24890
[martini] Completed 200 OK in 1.670137ms
2018-07-17 18:30:26 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
2018-07-17 18:30:27 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
2018-07-17 18:30:28 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
2018-07-17 18:30:29 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
2018-07-17 18:30:30 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
2018-07-17 18:30:31 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
2018-07-17 18:30:32 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
[martini] Started GET / for 100.100.0.0:24922

<<<<### Omitted health checks ### >>>>

[martini] Completed 302 Found in 658.018µs
[martini] Started GET /web/clusters for 100.100.0.0:26284
[martini] Completed 200 OK in 1.296347ms

# Running a discover on my master from the GUI

[martini] Started GET /api/discover/10.1.1.151/3306 for 100.116.0.0:37639
2018-07-17 18:33:42 DEBUG Hostname unresolved yet: 10.1.1.151
2018-07-17 18:33:42 DEBUG Cache hostname resolve 10.1.1.151 as 10.1.1.151
2018-07-17 18:33:42 DEBUG WriteResolvedHostname: resolved 10.1.1.151 to 10.1.1.151
2018-07-17 18:33:42 ERROR TLS requested but server does not support TLS
2018-07-17 18:33:42 ERROR ReadTopologyInstance(10.1.1.151:3306) show variables like 'maxscale%': TLS requested but server does not support TLS
2018-07-17 18:33:43 ERROR ReadTopologyInstance(10.1.1.151:3306) show global status like 'Uptime': TLS requested but server does not support TLS
[martini] Completed 500 Internal Server Error in 1.484724555s

[pznamensky]

The same issue. I'm using SQLite as orchestrator backend DB. (orchestrator.conf.json) And when I'm trying to discover a new instance I'm getting following error:

# orchestrator --config /etc/orchestrator.conf.json --debug http 
2018-08-30 17:35:41 INFO starting orchestrator, version: 3.0.12, git commit: d318760701a5649867606f455148e28f4353d288
2018-08-30 17:35:41 INFO Read config: /etc/orchestrator.conf.json
2018-08-30 17:35:41 DEBUG Connected to orchestrator backend: sqlite on /usr/local/orchestrator/orchestrator.sqlite3
2018-08-30 17:35:41 DEBUG Initializing orchestrator
2018-08-30 17:35:41 INFO Connecting to backend :3306: maxConnections: 128, maxIdleConns: 32
2018-08-30 17:35:41 INFO Starting Discovery
2018-08-30 17:35:41 INFO Registering endpoints
2018-08-30 17:35:41 INFO continuous discovery: setting up
2018-08-30 17:35:41 INFO Starting HTTP listener on :3000
2018-08-30 17:35:41 INFO continuous discovery: starting
2018-08-30 17:35:41 DEBUG Queue.startMonitoring(DEFAULT)
2018-08-30 17:35:43 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
2018-08-30 17:35:44 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
2018-08-30 17:35:45 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
2018-08-30 17:35:46 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
2018-08-30 17:35:47 DEBUG Waiting for 15 seconds to pass before running failure detection/recovery
[martini] Started GET /api/discover/some-mysql-srv/3306 for [ip]:44856

^^^
Trying to discover a new instance from the web interface

2018-08-30 17:35:47 ERROR TLS requested but server does not support TLS
2018-08-30 17:35:47 ERROR ReadTopologyInstance(some-mysql-srv:3306) show variables like 'maxscale%': TLS requested but server does not support TLS
[martini] Completed 500 Internal Server Error in 15.480666ms

[cezmunsta]

@sprutner @pznamensky perhaps MySQLTopologyUseMixedTLSis set to true and you are getting an "Access denied for user" error (https://github.com/github/orchestrator/commit/42d7d8942b44257a1c68d7f25fcd58ca1629d65b)? Do you have SSL enabled at all for any of the instances that orchestrator is monitoring?

[pznamensky]

@cezmunsta you're right. I just added MySQLTopologyUseMixedTLS: false and error Access denied appeared. After changing credentials everything started working fine. Thank you!

[shlomi-noach]

hah, I love it. Just seen the error myself and google led me to this issue.

[sapisuper]

MySQLTopologyUseMixedTLS: false

this variable doesn't exist in json config file. must input manually

[ilhaan]

I needed this in a docker container and made the following change:

diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
index 05045ba1..8fc9627c 100755
--- a/docker/entrypoint.sh
+++ b/docker/entrypoint.sh
@@ -10,7 +10,8 @@ cat <<EOF > /etc/orchestrator.conf.json
   "MySQLOrchestratorPort": ${ORC_DB_PORT:-3306},
   "MySQLOrchestratorDatabase": "${ORC_DB_NAME:-orchestrator}",
   "MySQLOrchestratorUser": "${ORC_USER:-orc_server_user}",
-  "MySQLOrchestratorPassword": "${ORC_PASSWORD:-orc_server_password}"
+  "MySQLOrchestratorPassword": "${ORC_PASSWORD:-orc_server_password}",
+  "MySQLTopologyUseMixedTLS": "${ORC_TOPOLOGY_MIXED_TLS:-false}"
 }
 EOF
 fi

However, that resulted in the following error when starting the container:

2020-10-07 21:51:06 FATAL Cannot read config file: /etc/orchestrator.conf.json json: cannot unmarshal string into Go struct field Configuration.MySQLTopologyUseMixedTLS of type bool

My workaround has been to do the following and rebuild the docker container:

diff --git a/docker/Dockerfile b/docker/Dockerfile
index 530f7a37..b218cb81 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -36,7 +36,7 @@ EXPOSE 3000

 COPY --from=build /usr/local/orchestrator /usr/local/orchestrator
 COPY --from=build /usr/bin/orchestrator-client /usr/bin/orchestrator-client
-COPY --from=build /etc/orchestrator.conf.json /etc/orchestrator.conf.json
+#COPY --from=build /etc/orchestrator.conf.json /etc/orchestrator.conf.json

 WORKDIR /usr/local/orchestrator
 ADD docker/entrypoint.sh /entrypoint.sh
diff --git a/go/config/config.go b/go/config/config.go
index 85bb1b66..fa9779f7 100644
--- a/go/config/config.go
+++ b/go/config/config.go
@@ -304,7 +304,7 @@ func newConfiguration() *Configuration {
                MySQLOrchestratorMaxPoolConnections:        128, // limit concurrent conns to backend DB
                MySQLOrchestratorPort:                      3306,
                MySQLTopologyUseMutualTLS:                  false,
-               MySQLTopologyUseMixedTLS:                   true,
+               MySQLTopologyUseMixedTLS:                   false,
                MySQLOrchestratorUseMutualTLS:              false,
                MySQLConnectTimeoutSeconds:                 2,
                MySQLOrchestratorReadTimeoutSeconds:        30,

Note that the change to Dockerfile shown above is required to get configs in env variables to be used instead of the /etc/orchestrator.conf.json that is copied into the container during build (refer to this)

[shlomi-noach]

However, that resulted in the following error when starting the container:

Unquote the boolean value. replace:

"MySQLTopologyUseMixedTLS": "${ORC_TOPOLOGY_MIXED_TLS:-false}"

with:

"MySQLTopologyUseMixedTLS": ${ORC_TOPOLOGY_MIXED_TLS:-false}

[ilhaan]

Ah I should have noticed the setting for Debug in the same file. Thanks @shlomi-noach!

What about the need to comment out COPY --from=build /etc/orchestrator.conf.json /etc/orchestrator.conf.json from Dockerfile? If that file is being copied during build then the if statement here (if [ ! -e /etc/orchestrator.conf.json ]) is always going to skip creating the file and prevent using env conf vars as mentioned here