search

sap-linuxlab / terraform.templates_for_sap

End to End Automation for SAP - Terraform Templates for deployment of various SAP solution scenarios, for every Cloud and Hypervisor
Apache License 2.0
9 stars 4 forks source link

SAP SWPM failure during connection to SAP HANA #3

Closed rgentyala closed 2 years ago

rgentyala commented 2 years ago

This error was seen during SAP S/4HANA OneHost install with SAP Maintenance Planner.

I used the following Terraform variables:

ibmcloud_api_key                  = "myapikey"

resource_tags                     = "mytag2"

resource_prefix                   = "myprefix1"

ibmcloud_resource_group           = "new"

ibmcloud_region                   = "us-east"

ibmcloud_iam_yesno                = "yes"

ibmcloud_vpc_subnet_name          = "new"

dns_root_domain                   = "mydomainlocal.com"

bastion_os_image                  = "rhel-8-4"

bastion_user                      = "myuser1"

bastion_ssh_port                  = "65500"

host_specification_plan           = "small_256gb"

host_os_image                     = "rhel-8-4-sap-hana"

disk_volume_capacity_software     = "200"

disk_volume_type_software         = "10iops-tier"

sap_software_download_directory   = "mysoftware"

sap_id_user                       = "Sxxxxxxxxxx"

sap_id_user_password              = "mysapidpw"

sap_hana_install_master_password  = "abcd@Abcde123"

sap_hana_install_sid              = "SA1"

sap_hana_install_instance_number  = "00"

sap_s4hana_install_sid            = "S4D"

sap_swpm_template_selected        = "sap_s4hana_2020_onehost_install"

sap_nwas_pas_instance_no          = "01"

sap_maintenance_planner_transaction_name = "MP_S4H_20xxxxxx_xxxxxx"

==========================================================================================

Below is the error:

│ INFO       2022-06-09 16:14:45.279 (root/sapinst) (startInstallation) [syuxcfile.cpp:216] id=syslib.filesystem.removedFile CSyFileImpl::removeEx(ISyFSErrorHandler *
│ pErrorHandler)
│ Removed file /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/HdbStdout.txt."
│ 
│ ERROR      2022-06-09 16:14:45.355 (root/sapinst)
│ (startInstallation) [iaxxbjsmod.cpp:83] id=modlib.jslib.caughtException errno=MUT-03025"
│ Caught ESAPinstException in module call: Validator of step
│ '|NW_ABAP_OneHost|ind|ind|ind|ind|0|0|NW_Onehost_System|ind|ind|ind|ind|onehost|0|NW_GetSidNoProfiles|ind|ind|ind|ind|getSid|0|NW_getDBInfo|ind|ind|ind|ind|db|0|NW_HDB_getDBInfo|ind|ind|ind|ind|hdb_dbinfo|0|getDBInfo'
│ reported an error:"
│ The database connection with database user SYSTEM cannot be set up. Check that the database is online and the password of user SYSTEM is correct.",
│ "Start SAPinst in interactive mode to solve this problem."
│ 
│ ERROR      2022-06-09 16:14:45.430 (root/sapinst) (startInstallation) [CSiStepExecute.cpp:1104]
│ id=controller.stepExecuted errno=FCO-00011 CSiStepExecute::execute()
│ The step getDBInfo with step key
│ |NW_ABAP_OneHost|ind|ind|ind|ind|0|0|NW_Onehost_System|ind|ind|ind|ind|onehost|0|NW_GetSidNoProfiles|ind|ind|ind|ind|getSid|0|NW_getDBInfo|ind|ind|ind|ind|db|0|NW_HDB_getDBInfo|ind|ind|ind|ind|hdb_dbinfo|0|getDBInfo
│ was executed with status ERROR (Last error reported by the step: Caught ESAPinstException in module call: Validator of step
│ '|NW_ABAP_OneHost|ind|ind|ind|ind|0|0|NW_Onehost_System|ind|ind|ind|ind|onehost|0|NW_GetSidNoProfiles|ind|ind|ind|ind|getSid|0|NW_getDBInfo|ind|ind|ind|ind|db|0|NW_HDB_getDBInfo|ind|ind|ind|ind|hdb_dbinfo|0|getDBInfo'
│ reported an error:
│ 
│ The database connection with database user SYSTEM cannot be set up. Check that the database is online and the password of user SYSTEM is correct.
│ "Start SAPinst in interactive mode to solve this problem)."
│ 
│ INFO       2022-06-09 16:14:45.561 (root/sapinst) (startInstallation) [syuxcpath.cpp:441]
│ id=syslib.filesystem.creatingFile CSyPath::createFile()
│ Creating file /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/INSTANA.XML.
│ 
│ INFO       2022-06-09
│ 16:14:45.631 (root/sapinst) (startInstallation) [syuxcpath.cpp:441] id=syslib.filesystem.creatingFile CSyPath::createFile()
│ Creating file
│ /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/instslana.xml.
│ 
│ 
│ a /tmp/sapinst_exe.221200.1654791257/dev_selfex.out
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/FormalPasswordCheck.1.log
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/FormalPasswordCheck.log
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/HdbCmdOut.log
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/LogAnalyzer.html
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/SECINSTTOOLS.log
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/control.xml
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/controllerKdbClient.dmp
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/instana.xsl
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/java.1.log
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/java.log
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/keydb.xml
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/sapinst.log
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/sapinst_dev.log
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/sapinst_loginquirer.log
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/start_dir.cd
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/stepKeydbClient.dmp
│ a /tmp/sapinst_instdir/S4HANA2020/CORE/HDB/INSTALL/STD/ABAP/syslib_priv.log
│ 
│ INFO       2022-06-09 16:14:47.651 (root/sapinst) (startInstallation) [syuxcdir.cpp:147]
│ id=syslib.filesystem.removedDirectory CSyDirectoryImpl::sap_remove(ISyFSErrorHandler * pErrorHandler)
│ Removed directory /root/.sapinst/s4h01/221271.
│ 
│
│ RUNNING HANDLER [community.sap_install.sap_general_preconfigure : Reboot the managed node] ***
│
│ RUNNING HANDLER [community.sap_install.sap_general_preconfigure : Let the role fail if a reboot is required] ***
│
│ RUNNING HANDLER [community.sap_install.sap_general_preconfigure : Show a warning message if a reboot is required] ***
│
│ RUNNING HANDLER [community.sap_install.sap_general_preconfigure : Remount /dev/shm] ***
│
│ RUNNING HANDLER [community.sap_install.sap_general_preconfigure : Check if /dev/shm is available] ***
│
│ RUNNING HANDLER [community.sap_install.sap_general_preconfigure : Show the result of df -h /dev/shm] ***
│
│ RUNNING HANDLER [community.sap_install.sap_hana_preconfigure : Run grubby for enabling TSX] ***
│
│ RUNNING HANDLER [community.sap_install.sap_hana_preconfigure : Reboot the managed node] ***
│
│ RUNNING HANDLER [community.sap_install.sap_hana_preconfigure : Let the role fail if a reboot is required] ***
│
│ RUNNING HANDLER [community.sap_install.sap_hana_preconfigure : Show a warning message if a reboot is required] ***
│
│ PLAY RECAP *********************************************************************
│ 10.241.0.4                 : ok=329  changed=90   unreachable=0    failed=1    skipped=183  rescued=0
│ ignored=5
│

========================================================================================= Maintainer: do not raise issues within an issue. Ignoring these for reasons below I have few other observations that needs a closer look at though it is not related to my present issue that I am reporting as these limitations cause restrictions.

  1. In IBM Cloud unless I mention new resource group, new subnet somehow script is failing. Maintainer: no data provided, cannot debug
  2. Restrictions applied on S/4HANA Host and Bastion with Public Key what if a customer want to have their own methods to use public keys for such stuff where they have control. Maintainer: Terraform Templates are free to amend, for click-run a design principle is to always leverage a created Bastion even when re-using an existing network, to ensure security and reduce unpredictable results
  3. Is there a process developed around this script that can help address not to destroy everything but what you really want. The split and use like step 1 Bastion, step 2 SAP Host and then how they work together once 2 resources are created. If I need to destroy, I can destroy either SAP or Bastion based on what I need to. Maintainer: please see existing Terraform documentation and blogs on Google, this can be achieved in two ways primarily (1) comment/remove from Terraform Template and terraform apply which will remove those resources or (2) use -target combined with terraform destroy see doc https://learn.hashicorp.com/tutorials/terraform/resource-targeting.
sean-freeman commented 2 years ago

Edited original issue text for readability of the issue, and to address best practice of raising GH Issues.

Terraform Template identified from text: sap_s4hana_single_node_install/ibmcloud_vs

Error identified from text: The database connection with database user SYSTEM cannot be set up. Check that the database is online and the password of user SYSTEM is correct

This error usually appears on SWPM Unattended Mode when the wrong Database Schema name for SAP (inside the SAP HANA MDC Tenant) has been provided, so a connection cannot be established to perform the DB Restore during System Copy. To see this error with an installation from SAP Maintenance Planner is unusual.

I will re-run with the same Terraform Input Variables provided above, to see if I have the same result.

rgentyala commented 2 years ago

Hello Sean - Thank you for helping me on my issue. I will wait for your confirmation, try to destroy my deployment and recreate the instances once again. If I can preserve Bastion host while destroying and perform only remaining part of the deployment which is S/4 deployment that really saves my time. I will wait for your confirmation.

sean-freeman commented 2 years ago

Prior to test run, errors observed in data entry

Sample data entry in Terraform Templates:

resource_tags                    = ["sap", "saps4hana"]
bastion_ssh_port                 = 50222
disk_volume_capacity_software    = 304
sap_software_download_directory  = "/software"

Values given in this GH Issue:

resource_tags                    = "mytag2"
bastion_ssh_port                 = "65500"
disk_volume_capacity_software    = "200"
sap_software_download_directory  = "mysoftware"

EDIT - Results from Terraform provisioning:

sean-freeman commented 2 years ago

Ansible calls - Keeping software directory variable as mysoftware only

i.e. Terraform variable sap_software_download_directory = "mysoftware"

module.run_ansible_sap_s4hana_install_maintplan["s4h01"].null_resource.ansible_exec[0] (local-exec):

TASK [community.sap_install.sap_install_media_detect : SAP Media Organize - SAP HANA - Extracting IMDB_CLIENT20_013_13-80002082.SAR] ***
fatal: [10.241.0.4]: FAILED! =>
{
  "changed": false,
  "cmd": "mysoftware/sap_hana/SAPCAR_1115-70006178.EXE -R mysoftware/sap_hana/tmp -xvf mysoftware/sap_hana/IMDB_CLIENT20_013_13-80002082.SAR -manifest SIGNATURE.SMF",
  "msg": "[Errno 2] No such file or directory: b'mysoftware/sap_hana/SAPCAR_1115-70006178.EXE': b'mysoftware/sap_hana/SAPCAR_1115-70006178.EXE'",
  "rc": 2,
  "stderr_lines": [],
  "stdout_lines": []
}

Ansible calls - Amending software directory variable but keeping under root /root/mysoftware

i.e. Terraform variable sap_software_download_directory = "/root/mysoftware"

module.run_ansible_sap_s4hana_install_maintplan["s4h01"].null_resource.ansible_exec[0] (local-exec):

TASK [community.sap_install.sap_hana_install : Install SAP HANA - SA1 00] ******
fatal: [10.241.0.4]: FAILED! =>
{
  "changed": true,
  "cmd": [
    "./hdblcm",
    "--configfile=/tmp/ansible.kdz652z5hanaconfig/configfile.cfg",
    "-b"
  ],
  "delta": "0:00:02.245110",
  "end": "2022-06-22 09:19:12.958743",
  "msg": "non-zero return code",
  "rc": 1,
  "start": "2022-06-22 09:19:10.713633",
  "stderr_lines": [
    "Running in batch mode",
    "  The following software installation files are missing execute permissions:",
    "    /root",
    "  Ensure that no permissions are changed by the umask when extracting or copying the folders."
  ],
  "stdout_lines": [
    "",
    "",
    "SAP HANA Lifecycle Management - SAP HANA Database 2.00.062.00.1650891137",
    "************************************************************************",
    "",
    "",
    "Scanning software locations...",
    "Detected components:",
    "    SAP HANA Database (2.00.062.00.1650891137) in /root/mysoftware/sap_hana/extracted/SAP_HANA_DATABASE/server",
    "    SAP HANA Database Client (2.13.13.1650313242) in /root/mysoftware/sap_hana/extracted/SAP_HANA_CLIENT/client",
    "Log file written to '/var/tmp/hdb_SA1_hdblcm_install_2022-06-22_09.19.10/hdblcm.log' on host 's4h01'."
  ]
}
sean-freeman commented 2 years ago

I performed another deployment to correct for the software download directory being an incomplete path; as above comment describes.

Unfortunately after re-deployment, I was unable to replicate the issue. The Ansible execution triggered by Terraform, took 1h17m47s to deploy SAP S/4HANA from SAP Maintenance Planner.

Approximate breakdown:

module.run_ansible_sap_s4hana_install_maintplan["rjs4h01"].null_resource.ansible_exec[0] (local-exec):
FAILED - RETRYING: [10.241.0.4]: SAP SWPM - Wait for sapinst process to exit, poll every 60 seconds (981 retries left).

module.run_ansible_sap_s4hana_install_maintplan["rjs4h01"].null_resource.ansible_exec[0]:
Still creating... [1h17m20s elapsed]
module.run_ansible_sap_s4hana_install_maintplan["rjs4h01"].null_resource.ansible_exec[0] (local-exec):
ok: [10.241.0.4]

TASK [community.sap_install.sap_swpm : SAP SWPM - Verify if sapinst process finished successfully] ***
changed: [10.241.0.4]

TASK [community.sap_install.sap_swpm : SAP SWPM - Find last installation location] ***
changed: [10.241.0.4]

TASK [community.sap_install.sap_swpm : SAP SWPM - Verify if SWPM success file exists] ***
ok: [10.241.0.4]

TASK [community.sap_install.sap_swpm : SAP SWPM - Get success file] ************
changed: [10.241.0.4]

TASK [community.sap_install.sap_swpm : SAP SWPM - Display installation finished from success file] ***
ok: [10.241.0.4] => {
    "msg": [
        "NW_ABAP_OneHost|ind|ind|ind|ind finished 22_06_2022_17_47_43"
    ]
}

TASK [community.sap_install.sap_swpm : SAP SWPM Post Install - Set s4dadm to no expire] ***
changed: [10.241.0.4]

TASK [community.sap_install.sap_swpm : SAP SWPM Post Install - Firewall Setup] ***
skipping: [10.241.0.4]

TASK [community.sap_install.sap_swpm : SAP SWPM Deployment - Finished] *********
ok: [10.241.0.4] => {
    "msg": [
        "   SAP SWPM deployment successfully completed ",
        " ",
        "   SAP Product        -       NW_ABAP_OneHost:S4HANA2020.CORE.HDB.ABAP  ",
        "   SID                -       S4D ",
        "   Primary Instance   -       01 ",
        "   Host               -       s4h01 ",
        "   FQDN               -       s4h01.mydomainlocal.com  ",
        "   IP                 -       10.241.0.4  ",
        "   Master Password    -       abcd@Abcde123 ",
        "   DDIC 000 Password  -       abcd@Abcde123 "
    ]
}
sean-freeman commented 2 years ago

As stated originally, error The database connection with database user SYSTEM cannot be set up. Check that the database is online and the password of user SYSTEM is correct is usually witnessed on SWPM Unattended Mode when the wrong Database Schema name for SAP (inside the SAP HANA MDC Tenant) has been provided, so a connection cannot be established to perform the DB Restore during System Copy.

To see this error with an installation from SAP Maintenance Planner is unusual. Given I cannot replicate this issue, I have to conclude this as an uncharacteristic error with SAP SWPM in this instance. Closing the issue.

rgentyala commented 2 years ago

I will correct variable definitions properly to reflect declaration standards, however it helps if we give default value and comments that can help improve the standard and these Hints either can be in readme.md or input.auto.tfvars.tf file to help.

I have a question regarding keeping mysoftware under /root that is not recommended. Can we have a way of keeping mysoftware in its own mount point?

I will try to work on your recommendations and come back to you with more comments.