search

TOSIT-IO / tdp-collection

Ansible collection to deploy the components of TDP
Apache License 2.0
21 stars 19 forks source link

Ranger KMS setup.sh fail for shema setup #492

Closed rpignolet closed 2 years ago

rpignolet commented 2 years ago

Deployed with getting started, here the log:

TASK [tosit.tdp.kms : Run setup.sh] ************************************************************
fatal: [master-03]: FAILED! => changed=true 
  ansible_facts:
    discovered_interpreter_python: /usr/bin/python
  cmd: |-
    export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
    ./setup.sh
  delta: '0:00:01.965807'
  end: '2022-10-06 11:42:59.087449'
  msg: non-zero return code
  rc: 1
  start: '2022-10-06 11:42:57.121642'
  stderr: |-
    2022-10-06 11:42:57,877  [I] DB FLAVOR :POSTGRES
    2022-10-06 11:42:57,878  [I] --------- Verifying Ranger DB connection ---------
    2022-10-06 11:42:57,878  [I] Checking connection
    2022-10-06 11:42:57,878  [JISQL] /usr/lib/jvm/jre-1.8.0-openjdk/bin/java  -cp /usr/share/java/postgresql-jdbc.jar:/opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://master-03.tdp:5432/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
    2022-10-06 11:42:58,291  [I] connection success
    2022-10-06 11:42:58,292  [I] --------- Verifying Ranger DB tables ---------
    2022-10-06 11:42:58,292  [I] Verifying table ranger_masterkey in database rangerkms
    2022-10-06 11:42:58,292  [JISQL] /usr/lib/jvm/jre-1.8.0-openjdk/bin/java  -cp /usr/share/java/postgresql-jdbc.jar:/opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://master-03.tdp:5432/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
    2022-10-06 11:42:58,674  [I] Table ranger_masterkey does not exist in database rangerkms
    2022-10-06 11:42:58,674  [I] --------- Importing Ranger Core DB Schema ---------
    2022-10-06 11:42:58,674  [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
    2022-10-06 11:42:58,674  [JISQL] /usr/lib/jvm/jre-1.8.0-openjdk/bin/java  -cp /usr/share/java/postgresql-jdbc.jar:/opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://master-03.tdp:5432/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/db/postgres/kms_core_db_postgres.sql
    SQLException : SQL state: 23505 org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
      Detail: Key (typname, typnamespace)=(ranger_masterkey, 16391) already exists. ErrorCode: 0
    SQLException : SQL state: 23505 org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
      Detail: Key (typname, typnamespace)=(ranger_masterkey, 16391) already exists. ErrorCode: 0
    2022-10-06 11:42:59,079  [E] kms_core_db_postgres.sql DB schema import failed!
  stderr_lines: <omitted>
  stdout: |-
    2022-10-06 11:42:57,665   --------- Running Ranger KMS Application Install Script ---------
    2022-10-06 11:42:57,667  [I] uname=Linux
    2022-10-06 11:42:57,670  [I] hostname=master-03
    2022-10-06 11:42:57,676  [I] DB_FLAVOR=POSTGRES
    2022-10-06 11:42:57,679  [I] KMS Master key password validated.
    2022-10-06 11:42:57,683  [I] Checking distribution name..
    2022-10-06 11:42:57,692  [I] Found distribution : CentOS
    2022-10-06 11:42:57,693  [I] check if command /usr/lib/jvm/jre-1.8.0-openjdk/bin/java exists
    2022-10-06 11:42:57,694  [I] '/usr/lib/jvm/jre-1.8.0-openjdk/bin/java' command found
    2022-10-06 11:42:57,756  [I] Checking POSTGRES CONNECTOR FILE : /usr/share/java/postgresql-jdbc.jar
    2022-10-06 11:42:57,757  [I] POSTGRES CONNECTOR FILE : /usr/share/java/postgresql-jdbc.jar file found
    2022-10-06 11:42:57,758  [I] Setting up UNIX user : keyadmin and group: hadoop
    2022-10-06 11:42:57,767  [I] the keyadmin user already exists and belongs to group hadoop
    2022-10-06 11:42:57,769  [I] Setting up UNIX user : keyadmin and group: hadoop DONE
    2022-10-06 11:42:57,771  [I] Setting up installation files and directory
    2022-10-06 11:42:57,773  [I] Copying /opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/ews/webapp/WEB-INF/classes/conf.dist /opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/ews/webapp/WEB-INF/classes/conf
    2022-10-06 11:42:57,788  [I] Setting up init.d
    2022-10-06 11:42:57,792  [I] Creating script S88ranger-kms/K90ranger-kms in /etc/rc2.d directory ....
    2022-10-06 11:42:57,797  [I] Creating script S88ranger-kms/K90ranger-kms in /etc/rc3.d directory ....
    2022-10-06 11:42:57,804  [I] Creating script S88ranger-kms/K90ranger-kms in /etc/rc.d/rc2.d directory ....
    2022-10-06 11:42:57,808  [I] Creating script S88ranger-kms/K90ranger-kms in /etc/rc.d/rc3.d directory ....
    2022-10-06 11:42:57,818  [I] Setting up installation files and directory DONE
    2022-10-06 11:42:57,831  [I] /opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/ews/webapp folder found
    2022-10-06 11:42:57,833  [I] db/postgres/kms_core_db_postgres.sql file found
    2022-10-06 11:42:57,834  Creating /opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/ews/lib
    2022-10-06 11:42:57,840  [I] Copying POSTGRES Connector to /opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/ews/lib
    2022-10-06 11:42:57,843  [I] Copying POSTGRES Connector to /opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/ews/lib DONE
    2022-10-06 11:42:57,844  [I] check if command python2 exists
    2022-10-06 11:42:57,845  [I] 'python2' command found
    2022-10-06 11:42:57,851  [I] Setup mode is set to SeparateDBA. Not Running DBA steps. Please run dba_script.py before running setup..!
    2022-10-06 11:42:59,086  [E] DB schema setup failed! Please contact Administrator.
  stdout_lines: <omitted>
changed: [master-02]
kpgtek commented 2 years ago

I can reproduce the issue after dropping tables and sequences of rangerkms database with their names in lower case, and then run the playbook ranger_kms_config.yml. It seems the error message shows that there are few references not removed from the "postgres" catalog database (the index pg_type_typname_nsp_index in the catalog table pg_type).

If I use the following SQL commands provided by the RangerKMS (script kms_core_db_postgres.sql) to delete tables and sequences, it do not reproduce the issue.

DROP TABLE IF EXISTS ranger_masterkey CASCADE;
DROP SEQUENCE IF EXISTS RANGER_MASTERKEY_SEQ;
DROP TABLE IF EXISTS ranger_keystore CASCADE;
DROP SEQUENCE IF EXISTS RANGER_KEYSTORE_SEQ;

The problem is when you drops rangerkms database, references are not properly removed from pg_catalog. I tried an vacuum --full to rangerkms and postgres databases, it seems this could fixe the issue.

But I needs to investigate more.

kpgtek commented 2 years ago

In fact, I have two SQL error messages:

ERROR: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
      Detail: Key (typname, typnamespace)=(ranger_masterkey, xxxx) already exists. ErrorCode: 0

and

ERROR: relation "ranger_masterkey" already exists ErrorCode: 0

In most case, I have duplicate key value violates unique constraint.

In my topology, I have:

[ranger_kms:children]
master2
master3

According to the following Ansible traces and timestamps of the DB schema import from master2 and master3 nodes, it seems that this is a parallelization issue, two master nodes try to import the same DB schema simultaneously. That is why in most case, the error occures in master3, and rangerkms tables are correctly created at the end despite the failure.

On master2 node:

STDERR:

2022-10-12 11:19:05,082  [I] DB FLAVOR :POSTGRES
2022-10-12 11:19:05,082  [I] --------- Verifying Ranger DB connection ---------
2022-10-12 11:19:05,082  [I] Checking connection
2022-10-12 11:19:05,082  [JISQL] /usr/lib/jvm/jre-1.8.0-openjdk/bin/java  -cp /usr/share/java/postgresql-jdbc.jar:/opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://ldap-01:5432/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2022-10-12 11:19:05,570  [I] connection success
2022-10-12 11:19:05,570  [I] --------- Verifying Ranger DB tables ---------
2022-10-12 11:19:05,570  [I] Verifying table ranger_masterkey in database rangerkms
2022-10-12 11:19:05,570  [JISQL] /usr/lib/jvm/jre-1.8.0-openjdk/bin/java  -cp /usr/share/java/postgresql-jdbc.jar:/opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://ldap-01:5432/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2022-10-12 11:19:06,062  [I] Table ranger_masterkey does not exist in database rangerkms
2022-10-12 11:19:06,062  [I] --------- Importing Ranger Core DB Schema ---------
2022-10-12 11:19:06,062  [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2022-10-12 11:19:06,062  [JISQL] /usr/lib/jvm/jre-1.8.0-openjdk/bin/java  -cp /usr/share/java/postgresql-jdbc.jar:/opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://ldap-01:5432/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/db/postgres/kms_core_db_postgres.sql
2022-10-12 11:19:06,583  [I] kms_core_db_postgres.sql DB schema imported successfully
2022-10-12 11:19:10,234  [I] DB FLAVOR :POSTGRES
2022-10-12 11:19:10,234  [I] --------- Verifying Ranger DB connection ---------
2022-10-12 11:19:10,234  [I] Checking connection
2022-10-12 11:19:10,234  [JISQL] /usr/lib/jvm/jre-1.8.0-openjdk/bin/java  -cp /usr/share/java/postgresql-jdbc.jar:/opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://ldap-01:5432/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2022-10-12 11:19:10,704  [I] connection success
./setup.sh: line 431: /usr/lib/jvm/jre-1.8.0-openjdk/bin/jar: No such file or directory

On master3 node:

STDERR:

2022-10-12 11:19:05,107  [I] DB FLAVOR :POSTGRES
2022-10-12 11:19:05,107  [I] --------- Verifying Ranger DB connection ---------
2022-10-12 11:19:05,107  [I] Checking connection
2022-10-12 11:19:05,108  [JISQL] /usr/lib/jvm/jre-1.8.0-openjdk/bin/java  -cp /usr/share/java/postgresql-jdbc.jar:/opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://ldap-01:5432/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2022-10-12 11:19:05,603  [I] connection success
2022-10-12 11:19:05,603  [I] --------- Verifying Ranger DB tables ---------
2022-10-12 11:19:05,603  [I] Verifying table ranger_masterkey in database rangerkms
2022-10-12 11:19:05,603  [JISQL] /usr/lib/jvm/jre-1.8.0-openjdk/bin/java  -cp /usr/share/java/postgresql-jdbc.jar:/opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://ldap-01:5432/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2022-10-12 11:19:06,089  [I] Table ranger_masterkey does not exist in database rangerkms
2022-10-12 11:19:06,089  [I] --------- Importing Ranger Core DB Schema ---------
2022-10-12 11:19:06,089  [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2022-10-12 11:19:06,089  [JISQL] /usr/lib/jvm/jre-1.8.0-openjdk/bin/java  -cp /usr/share/java/postgresql-jdbc.jar:/opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://ldap-01:5432/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /opt/tdp/ranger-2.0.1-TDP-0.1.0-SNAPSHOT-kms/db/postgres/kms_core_db_postgres.sql
SQLException : SQL state: 23505 org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
  Detail: Key (typname, typnamespace)=(ranger_masterkey, 2200) already exists. ErrorCode: 0
SQLException : SQL state: 23505 org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
  Detail: Key (typname, typnamespace)=(ranger_masterkey, 2200) already exists. ErrorCode: 0
2022-10-12 11:19:06,571  [E] kms_core_db_postgres.sql DB schema import failed!
rpignolet commented 2 years ago

Nice catch @kpgtek ! Indeed we run the setup.sh in parallel for Ranger KMS and Ranger Admin but we only have 1 Ranger Admin and 2 Ranger KMS. We should run setup.sh in sequence, maybe throttle: 1 can do the job https://docs.ansible.com/ansible/2.9/user_guide/playbooks_strategies.html#using-keywords-to-control-execution.

kpgtek commented 2 years ago

By controlling the playbook execution with throttle: 1, it seems that fixes the issue. I put this in the setup.sh running task.