Unable to Start with Multi-Tenant Enabled...

[Hummdis]

We have several HANA servers, all multi-tenant systems. When we try to start the hanadb_exporter application with the multi-tenant option set to true, it fails to start with an authentication issue. However, if we start with it set to false, then it starts up without issue.

We have confirmed that the same user exists in all of the tenants and the permissions match that of the DB commands that are in the docs.

Our config:

{
  "listen_address": "0.0.0.0",
  "exposition_port": 7825,
  "multi_tenant": true,
  "timeout": 30,
  "hana": {
    "host": "<fqdn>",
    "port": 32013,
    "user": "<db user>",
    "password": "<db pass>",
    "ssl": false,
    "ssl_validate_cert": false
  },
  "logging": {
    "config_file": "/opt/hanadb_exporter/logging_config.ini",
    "log_file": "/var/log/hanadb_exporter.log"
  }
}

Obviously we're using actual values for the host, user, and password fields. They've been removed for security.

The SAP instance is 20, so the 32013 port is correct. We've used the FQDN, hostname, and localhost as options to the host option, but there's no change.

When we try to start it, we get:

# hanadb_exporter -c /opt/hanadb_exporter/config.json -m /opt/hanadb_exporter/metrics.json 
2023-07-11 01:28:37,673 ERROR hanadb_exporter Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/shaptools/hdb_connector/connectors/dbapi_connector.py", line 55, in connect
    **self.__properties
hdbcli.dbapi.Error: (10, 'authentication failed')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/hanadb_exporter", line 9, in <module>
    main.run()
  File "/usr/lib/python3.6/site-packages/hanadb_exporter/main.py", line 134, in run
    timeout=config.get('timeout', 600))
  File "/usr/lib/python3.6/site-packages/hanadb_exporter/db_manager.py", line 130, in start
    self._connect_tenants(host, connection_data)
  File "/usr/lib/python3.6/site-packages/hanadb_exporter/db_manager.py", line 64, in _connect_tenants
    host, tenant_port, **connection_data)
  File "/usr/lib/python3.6/site-packages/shaptools/hdb_connector/connectors/dbapi_connector.py", line 58, in connect
    raise base_connector.ConnectionError('connection failed: {}'.format(err))
shaptools.hdb_connector.connectors.base_connector.ConnectionError: connection failed: (10, 'authentication failed')

We are using the dbapi method, not the pyhdb.

I feel like there's something we're missing, but it's unclear what that is. Any help would be appreciated.

Thanks!

[yeoldegrove]

@Hummdis I did a quick check from my side:

• installing a new HANA

• adding an additional tenant DB

• adding 3 users (same username and password) for SYSTEMDB and both tenant DBs as described in the docs

su - prdadm

hdbsql -u SYSTEM -p secretpassword -d SYSTEMDB
CREATE USER HANADB_EXPORTER_USER PASSWORD secretpassword NO FORCE_FIRST_PASSWORD_CHANGE;
CREATE ROLE HANADB_EXPORTER_ROLE;
GRANT MONITORING TO HANADB_EXPORTER_ROLE;
GRANT HANADB_EXPORTER_ROLE TO HANADB_EXPORTER_USER;

hdbsql -u SYSTEM -p secretpassword -d PRD
CREATE USER HANADB_EXPORTER_USER PASSWORD secretpassword NO FORCE_FIRST_PASSWORD_CHANGE;
CREATE ROLE HANADB_EXPORTER_ROLE;
GRANT MONITORING TO HANADB_EXPORTER_ROLE;

hdbsql -u SYSTEM -p secretpassword -d SYSTEMDB
CREATE DATABASE PR2 SYSTEM USER PASSWORD secretpassword;

hdbsql -u SYSTEM -p secretpassword -d PR2
CREATE USER HANADB_EXPORTER_USER PASSWORD secretpassword NO FORCE_FIRST_PASSWORD_CHANGE;
CREATE ROLE HANADB_EXPORTER_ROLE;
GRANT MONITORING TO HANADB_EXPORTER_ROLE;

• using the exporter config provided by you

I could not find any issues in my test setup :(

I used localhost:30013 as my instance number is 00. Also using dbapi.

[Hummdis]

Thanks for the work, @yeoldegrove . I'm no the Basis/SAP person, but I'll get a meeting scheduled with them and confirm the user is properly setup. I'll report back when I have details.

[Hummdis]

Hi @yeoldegrove , So, I've been working on this (thank you for your patience) and we got past the issue on a few systems once it was verified that the user was setup correctly in all tenants. However, we've located another problem that prohibits the HANA DB Exporter from running and that problem appears to be the fact that it require all tenants to be online. We have some systems where the tenant is inactive/unused and as a result it's unable to communicate with the tenant, so it fails with "connection refused."

shaptools.hdb_connector.connectors.base_connector.ConnectionError: connection failed: (-10709, "Connection failed (RTE:[89006] System call 'connect' failed, rc=111:Connection refused {10.0.0.210:30241} (localhost:30241))")

So, the only place that it appears the multi-tenant ability can be used is on systems where every tenant is online.

Is the only other option to monitor the the SYSTEMDB and all tenants is to run a separate process for each tenant? It just seems counterproductive to me given that it has the capability to talk to all tenants in a single process. Perhaps it should check for only the started tenants and just connect to those?