createDomain encountered an unexpected runtime exception

[vikashsonu900]

Create domain failing for OSB domain.

        "Issue Log for createDomain version 4.2.0 running WebLogic version 12.2.1.4.0.240325 in offline mode:",
        "",
        "SEVERE Messages:",
        "",
        "        1. WLSDPLY-20035: createDomain encountered an unexpected runtime exception.  Please file an issue on GitHub and attach the log file and stdout. Exception: exceptions.AttributeError",
        "",
        "Total:   SEVERE :    1  WARNING :    0"
    ] 

Custom Typedef

{
  "name": "OSB",
  "description": "OSB type domain definitions",
  "versions": {
    "12.2.1.4": "OSB_12214"
  },
  "definitions": {
    "OSB_12214": {
      "baseTemplate": "Basic WebLogic Server Domain",
      "extensionTemplates": [
         "Oracle Service Bus",
         "Oracle Enterprise Scheduler Service Basic", "Oracle Enterprise Manager Plugin for ESS"
      ],
      "serverGroupsToTarget": [ "JRF-MAN-SVR", "WSMPM-MAN-SVR", "OSB-MGD-SVRS-COMBINED", "ESS-MGD-SVRS" ],
      "rcuSchemas": [ "STB", "WLS", "ESS", "SOAINFRA", "MDS", "IAU", "IAU_VIEWER", "IAU_APPEND", "OPSS", "UCSUMS" ],
      "dynamicClusterServerGroupsToTarget": [ "WSMPM-DYN-CLUSTER", "WSM-CACHE-DYN-CLUSTER" ]
    }
  }
}

[vikashsonu900]

createDomain.log

[robertpatrick]

@vikashsonu900 Try the attached installer and let us know if resolves the error.

weblogic-deploy.zip

[vikashsonu900]

thanks, trying now.

[vikashsonu900]

Getting below error with this.

        "",
        "Issue Log for createDomain version 4.2.1-SNAPSHOT running WebLogic version 12.2.1.4.0.240325 in offline mode:",
        "",
        "SEVERE Messages:",
        "",
        "        1. WLSDPLY-12409: createDomain failed to create the domain: OSB domain creation precheck failed. exceptions.AttributeError: set",
        "",
        "Total:   SEVERE :    1  WARNING :    0"
    ]
}

[robertpatrick]

@vikashsonu900 we will need to see the new log file...

[robertpatrick]

@vikashsonu900 Using a model that looks like this:

domainInfo:
    AdminUserName: weblogic
    AdminPassword: welcome1
    RCUDbInfo:
        rcu_db_conn_string: <redacted>.com:1521/orclpdb.<redacted>.com
        rcu_prefix: RHP
        rcu_admin_password: <redacted>
        rcu_schema_password: welcome1
topology:
    Name: mydomain
    ProductionModeEnabled: true
    Server:
        AdminServer:
            ListenPort: 7001
            ListenAddress: rpatrick-mac
        MyOSBServer:
            ListenPort: 8001
            ListenAddress: rpatrick-mac

And your typedef, I was able to successfully create an OSB domain.

JDK version is 1.8.0_351-b11
JAVA_HOME = /Library/Java/JavaVirtualMachines/jdk1.8.0_351.jdk/Contents/Home
WLST_EXT_CLASSPATH = /Users/rpatrick/tmp/osb/weblogic-deploy/lib/weblogic-deploy-core.jar
CLASSPATH = /Users/rpatrick/tmp/osb/weblogic-deploy/lib/weblogic-deploy-core.jar
WLST_PROPERTIES = -Dcom.oracle.cie.script.throwException=true -Djava.util.logging.config.class=oracle.weblogic.deploy.logging.WLSDeployLoggingConfig 
/opt/weblogic/soa12214/oracle_common/common/bin/wlst.sh /Users/rpatrick/tmp/osb/weblogic-deploy/lib/python/create.py -oracle_home /opt/weblogic/soa12214 -domain_parent /Users/rpatrick/tmp/osb -domain_type OSB -model_file model.yaml -run_rcu

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

####<Jul 17, 2024 7:20:41 AM> <INFO> <WebLogicDeployToolingVersion> <logVersionInfo> <WLSDPLY-01750> <The WebLogic Deploy Tooling createDomain version is 4.2.1-SNAPSHOT:main.927180f:Jul 17, 2024 12:18 UTC>
####<Jul 17, 2024 7:20:41 AM> <INFO> <WLSDeployLoggingConfig> <logLoggingDirectory> <WLSDPLY-01755> <The createDomain program will write its log to directory /Users/rpatrick/tmp/osb/weblogic-deploy/logs>
####<Jul 17, 2024 7:20:41 AM> <INFO> <create> <main> <WLSDPLY-20043> </Users/rpatrick/tmp/osb/weblogic-deploy/lib/python/create.py running with Java version 1.8.0_351 on operating system Mac OS X 14.5 (x86_64)>
####<Jul 17, 2024 7:20:42 AM> <INFO> <create> <main> <WLSDPLY-20044> </Users/rpatrick/tmp/osb/weblogic-deploy/lib/python/create.py is running on a Java version that supports the encryption algorithms used by WDT encryption>
####<Jul 17, 2024 7:20:42 AM> <INFO> <PathHelper> <__init__> <WLSDPLY-02100> <Setting local file system to use Posix-style paths>
####<Jul 17, 2024 7:20:42 AM> <INFO> <ModelContext> <__copy_from_args> <WLSDPLY-01050> <Local WebLogic version is 12.2.1.4.0.240325>

...

####<Jul 17, 2024 7:38:16 AM> <INFO> <DomainCreator> <__extend_domain_with_select_template> <WLSDPLY-12205> <Writing base domain mydomain to directory /Users/rpatrick/tmp/osb/mydomain>
####<Jul 17, 2024 7:49:51 AM> <INFO> <AbstractPolicyImpl> <initializeReadStore> <> <Property for read store in parallel: oracle.security.jps.az.runtime.readstore.threads = null>
####<Jul 17, 2024 8:22:21 AM> <INFO> <DomainCreator> <__extend_domain_with_select_template> <WLSDPLY-12206> <Closing templates for domain mydomain>
####<Jul 17, 2024 8:22:24 AM> <INFO> <Creator> <_create_named_mbeans> <WLSDPLY-12101> <Updating Server with the name AdminServer>
####<Jul 17, 2024 8:22:25 AM> <INFO> <Creator> <_create_named_mbeans> <WLSDPLY-12101> <Updating Server with the name MyOSBServer>
####<Jul 17, 2024 8:22:25 AM> <INFO> <DefaultAuthenticatorHelper> <create_default_init_file> <WLSDPLY-01900> <Updating default authenticator initialization file /Users/rpatrick/tmp/osb/mydomain/security/DefaultAuthenticatorInit.ldift>
####<Jul 17, 2024 8:22:25 AM> <INFO> <LibraryHelper> <install_domain_libraries> <WLSDPLY-12213> <The model did not specify any domain libraries to install>
####<Jul 17, 2024 8:22:25 AM> <INFO> <LibraryHelper> <extract_classpath_libraries> <WLSDPLY-12216> <The archive file was not provided so there are no classpath libraries to extract>
####<Jul 17, 2024 8:22:25 AM> <INFO> <LibraryHelper> <install_domain_scripts> <WLSDPLY-12241> <The model did not specify any domain scripts to install>

Issue Log for createDomain version 4.2.1-SNAPSHOT running WebLogic version 12.2.1.4.0.240325 in offline mode:

Total:   SEVERE :    0  WARNING :    0

createDomain.sh completed successfully (exit code = 0)

[vikashsonu900]

I don't understand the logic here, I have been trying to understand the changes.

1. https://github.com/oracle/weblogic-deploy-tooling/blame/f2bf13aea205119615c2fcc5c46d14e7b3e51d66/core/src/main/python/wlsdeploy/tool/util/rcu_helper.py#L887

There is no set method in this java class nor in its parents hierarchy. My java version is 1.8.0_411-b25 Reference doc https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html. Also put is not recommended.

2. Second, I see the point to extract the properties again from model file after defining properties user and password at https://github.com/oracle/weblogic-deploy-tooling/blob/f2bf13aea205119615c2fcc5c46d14e7b3e51d66/core/src/main/python/wlsdeploy/tool/util/rcu_helper.py#L281

but if you extract the values from model then properties will get or set for username like this

"('PROP: ', 'user', {'Value': 'ELHOSB_STB'})",
        "('PROP: ', 'oracle.net.CONNECT_TIMEOUT', {'Value': 10000})",
        "('PROP: ', 'SendStreamAsBlob', {'Value': true})",
        "('PROP: ', 'databasename', {'Value': 'LocalSvcTblDataSource'})",

Which is incorrect username and hence I will get logon denied

So in my opinion I think it should be something like this but I am not the product expert :-)

        new_password = dictionary_utils.get_element(model_driver_params_dict, PASSWORD_ENCRYPTED)
        if new_password is not None:
            new_password = self._aliases.decrypt_password(new_password)
            props.setProperty('password', new_password)

        for prop_key, prop_value in model_properties_dict.iteritems():
            if isinstance(prop_value, dict):
                print(prop_key, prop_value.get('Value'))
                if prop_value.get('Value') is not None:
                        props.setProperty(prop_key, str(prop_value.get('Value')))

            else:
                print(prop_key, prop_value)
                if prop_value is not None:
                        props.setProperty(prop_key, str(prop_value))

This works for me but now I am hitting below error, which I will see tomorrow.

     "",
        "Issue Log for createDomain version 4.2.1-SNAPSHOT running WebLogic version 12.2.1.4.0.240325 in offline mode:",
        "",
        "WARNING Messages:",
        "",
        "        1. WLSDPLY-09109: Unable to delete FileStore wlsbJMSServer, name does not exist",
        "",
        "Total:   SEVERE :    0  WARNING :    1"
    ]
}

[robertpatrick]

@vikashsonu900 OK, you are correct in the first point, thanks for pointing this out. However, that code only gets invoked if you are using a domain with RCU schemas AND your model has the RCU STB datasource defined in the resources:/JDBCSystemResource section of the model provided to createDomain AND that resources definition includes Driver Properties.

Do you have the STB data source defined in the resources section of the model?

[vikashsonu900]

This is really weird and and I am still confused how this set() method working on instance of Properties() as my environment is treating this like only and I was able to modify the codes as stated above and run it in WLST jython env. https://github.com/oracle/weblogic-deploy-tooling/blob/f2bf13aea205119615c2fcc5c46d14e7b3e51d66/core/src/main/python/wlsdeploy/tool/util/rcu_helper.py#L16 https://github.com/oracle/weblogic-deploy-tooling/blob/f2bf13aea205119615c2fcc5c46d14e7b3e51d66/core/src/main/python/wlsdeploy/tool/util/rcu_helper.py#L268

Also https://github.com/oracle/weblogic-deploy-tooling/blob/f2bf13aea205119615c2fcc5c46d14e7b3e51d66/core/src/main/python/wlsdeploy/tool/util/rcu_helper.py#L286 takes getConnection(String url, Properties info) instance as second parameters.

So not sure how set() method can work from WLST primitive here and how Properties from util is getting differenciated at runtime between WLST one JAVA one when we are also importing utils.Properties in code.

I am saying these details only to avoid having my own copy of codes and then again merging with new one in future.

[vikashsonu900]

ok, saw your comment now :-)

[vikashsonu900]

Yes, I have both of them.

                JDBCDriverParams:
                    URL: '{{ jdbc_hosts | ***:oracle.jdbc(redacted) }}'
                    PasswordEncrypted: '{{ db_pw_stb }}'
                    DriverName: oracle.jdbc.OracleDriver
                    Properties:
                        user:
                            Value: {{ oracle_user|upper }}_STB
                        oracle.net.CONNECT_TIMEOUT:
                            Value: 10000
                        SendStreamAsBlob:
                            Value: true
                        databasename:
                            Value: LocalSvcTblDataSource
                        servername:
                            Value: '{{ jdbc_hosts | join(',')}}'
                        portnumber:
                            Value: ***
                        logintimeout:
                        v$session.process:
                            Value: LocalSvcTblD
                        v$session.machine:
                            SysPropValue: weblogic.Name

[robertpatrick]

@vikashsonu900 See if this installer resolves the issue for you.

weblogic-deploy.zip

[vikashsonu900]

I tried this and have seen the code, but it does not address my 2nd concern. We cannot set user = {'Value': 'ELHOSB_STB'}) it has to be ELHOSB_STB. It needs to check the dictionary and extract value from dict in case it is dict.

As expected I am getting logon denied.

[robertpatrick]

@vikashsonu900 Try this installer to see if it resolves the problem.

weblogic-deploy.zip

[vikashsonu900]

Getting below which is self clear

Caused by: oracle.weblogic.deploy.create.CreateException: WLSDPLY-12265: During RCU pre-check, the resources section of
 the model contained JDBCSystemResource LocalSvcTblDataSource that defined a JDBCDriverParams Property 
v$session.machine that did not have the excepted key Value but had the key SysPropValue instead.  This key is not currently handled so please raise a GitHub issue if you require support for this key type.

[robertpatrick]

@vikashsonu900 so I assume removing this property from the model is ok?

[vikashsonu900]

I think this is what populates the session table in oracle db.

v$session.process:
     Value: LocalSvcTblD
 v$session.machine:
      SysPropValue: weblogic.Name

[robertpatrick]

@vikashsonu900 But why would v$session.machine be passed as a JDBC driver property? I don't think this should be necessary.

[vikashsonu900]

What we have followed when we introduced WDT was to discover a live working domain and variablize those discovered templates to create other environments. So when we discover a live env we get something like this and this is passed to createDomain in next phase. Something wrong in this approach?

        LocalSvcTblDataSource:
            Target: elhosb_cluster,AdminServer
            JdbcResource:
                JDBCConnectionPoolParams:
                    StatementCacheSize: 10
                    InitialCapacity: 2
                    CapacityIncrement: 1
                    TestConnectionsOnReserve: true
                    ConnectionCreationRetryFrequencySeconds: 10
                    MinCapacity: 2
                    TestTableName: SQL SELECT 1 FROM DUAL
                    TestFrequencySeconds: 0
                    SecondsToTrustAnIdlePoolConnection: 0
                    ShrinkFrequencySeconds: 900
                    MaxCapacity: 200
                JDBCDataSourceParams:
                    GlobalTransactionsProtocol: None
                    JNDIName: jdbc/LocalSvcTblDataSource
                JDBCDriverParams:
                    URL: jdbc:oracle:thin:@(DESCRIPTION=(CONNECT_TIMEOUT=15)(RETRY_COUNT=50)(RETRY_DELAY=3)(ADDRESS_LIST=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=--redacted--)(PORT=--redacted--)))(CONNECT_DATA=(SERVICE_NAME=--redacted--)))
                    PasswordEncrypted: --FIX ME--
                    DriverName: oracle.jdbc.OracleDriver
                    Properties:
                        user:
                            Value: ELHOSB_STB
                        oracle.net.CONNECT_TIMEOUT:
                            Value: '10000'
                        SendStreamAsBlob:
                            Value: 'true'
                        weblogic.jdbc.crossPartitionEnabled:
                            Value: 'true'
                        databasename:
                            Value: LocalSvcTblDataSource
                        servername:
                            Value: --redacted--
                        portnumber: {}
                        logintimeout: {}
                        v$session.process:
                            Value: LocalSvcTblD
                        v$session.machine:
                            SysPropValue: weblogic.Name 

[vikashsonu900]

From a dba perspective this value in session needs to be there to help troubleshoot from perspective of datasource's sources. An architecture of weblogic from beginning which I have seen.

[robertpatrick]

@vikashsonu900 I have been working with WLS since WLS 3.0 and this has nothing to do with WebLogic but...try this installer.

weblogic-deploy.zip

[robertpatrick]

You might also want to try to new -discover_password option to discover the data source passwords from the live environment and put them into the discovered model using WDT encryption. You will need to provide a WDT encryption passphrase to discoverDomain and then provide the same passphrase to createDomain.

[vikashsonu900]

I can give it a try but I think we are already getting encrypted one and masking them with --FIX --ME. In our project we variablize those part and have Ansible Jinja templating to fetch at runtime from Pass-Vault inside wdt templates and then run wdt commands on them. We mask the password on server also when it gets created via discover.

But I can try the new passphrase one which looks cool.

- name: Hide passwords from WDT template files
  ansible.builtin.replace:
    path: "{{ wdt_templates_home }}/{{ item }}"
    regexp: (.*Encrypted:\s+)(.*)
    replace: '\1 --FIXME--'
  loop: "{{ wdt_templates }}"
  tags:
    - cleanup 

[robertpatrick]

Fair enough, what about the new installer?

[vikashsonu900]

My OPPS schema has been corrupted, so I am creating fresh one but looks like it has passed those checks.

        "",
        "Issue Log for createDomain version 4.2.1-SNAPSHOT running WebLogic version 12.2.1.4.0.240325 in offline mode:",
        "",
        "WARNING Messages:",
        "",
        "        1. WLSDPLY-12271: During RCU pre-check, the resources section of the model contained JDBCSystemResource LocalSvcTblDataSource that defined a JDBCDriverParams Property v$session.machine that specified the key SysPropValue but the specified Java System Property \"weblogic.Name\" had no value so is it is being skipped.",
        "",
        "SEVERE Messages:",
        "",
        "        1. WLSDPLY-12409: createDomain failed to create the domain: writeDomain(/u01/elhosb/domain/elhosb_domain) failed: Error writing domain:",
        "64254: Error occurred in \"OPSS Processing\" phase execution",
        "64254: Encountered error: oracle.security.opss.tools.lifecycle.LifecycleException: Error during configuring DB security store. Exception oracle.security.opss.tools.lifecycle.LifecycleException: The schema ELHOSB_OPSS is already in use for security store(s).  Please create a new schema..",
        "64254: Check log for more detail.",
        "null",
        "",
        "Total:   SEVERE :    1  WARNING :    1"

[robertpatrick]

Maybe since WDT RCU pre-check always executes outside a WLS environment (and hence, weblogic.Name will not be set), we should reduce the severity of the warning message to info or below... Here is a new installer that does that. weblogic-deploy.zip

[vikashsonu900]

Yeah, was about to say that, as I got this.

        "Issue Log for createDomain version 4.2.1-SNAPSHOT running WebLogic version 12.2.1.4.0.240325 in offline mode:",
        "",
        "WARNING Messages:",
        "",
        "        1. WLSDPLY-12271: During RCU pre-check, the resources section of the model contained JDBCSystemResource LocalSvcTblDataSource that defined a JDBCDriverParams Property v$session.machine that specified the key SysPropValue but the specified Java System Property \"weblogic.Name\" had no value so is it is being skipped.",
        "",
        "Total:   SEVERE :    0  WARNING :    1"
    ]
}

NO MORE HOSTS LEFT **********************************************************************************************************************

PLAY RECAP ******************************************************************************************************************************
Server-name  : ok=9    changed=1    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Despite warning it is failing the playbook execution, might be return code needs to be tuned with the last binary I tried. I will try this new one uploaded now by you.

[robertpatrick]

The new installer should have reduced that to an info message so that you get a 0 exit code. Please verify.

[vikashsonu900]

On it, I just need to reset all.

[vikashsonu900]

Yes, it work great now.

        "####<Jul 19, 2024 1:22:06 PM> <INFO> <OdlHelper> <_update_config> <WLSDPLY-19708> <Applying ODLConfiguration ODLConfiguration-1 to server elhosb_ms1>",
        "",
        "Issue Log for createDomain version 4.2.1-SNAPSHOT running WebLogic version 12.2.1.4.0.240325 in offline mode:",
        "",
        "Total:   SEVERE :    0  WARNING :    0",
        "",
        "createDomain.sh completed successfully (exit code = 0)"
    ]
}

[vikashsonu900]

I think I should wait for new release then :-). Thanks a lot for quick responses.

[robertpatrick]

Give me a few days...