CDP environment creation failed

[mearchana]

Hi, I am getting the following error while executing /opt/cldr-runner/collections/ansible_collections/cloudera/cloud/plugins/modules/env.py from /opt/cldr-runner/collections/ansible_collections/cloudera/exe/roles/platform/tasks/setup_aws_env.yml

below is the error message * Monday 25 October 2021 17:08:40 +0000 (0:00:02.728) 0:01:33.504 **** ok: [localhost] => { "msg": { "changed": false, "exception": "Traceback (most recent call last):\n File \"/root/.ansible/tmp/ansible-tmp-1635181717.7778952-26659-167799228659899/AnsiballZ_env.py\", line 102, in \n _ansiballz_main()\n File \"/root/.ansible/tmp/ansible-tmp-1635181717.7778952-26659-167799228659899/AnsiballZ_env.py\", line 94, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/root/.ansible/tmp/ansible-tmp-1635181717.7778952-26659-167799228659899/AnsiballZ_env.py\", line 40, in invoke_module\n runpy.run_module(mod_name='ansible_collections.cloudera.cloud.plugins.modules.env', init_globals=None, run_name='main', alter_sys=True)\n File \"/usr/lib64/python3.8/runpy.py\", line 207, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.8/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib64/python3.8/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_cloudera.cloud.env_payload_69lmx119/ansible_cloudera.cloud.env_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/env.py\", line 1055, in \n File \"/tmp/ansible_cloudera.cloud.env_payload_69lmx119/ansible_cloudera.cloud.env_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/env.py\", line 1045, in main\n File \"/tmp/ansible_cloudera.cloud.env_payload_69lmx119/ansible_cloudera.cloud.env_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/env.py\", line 662, in init\n File \"/tmp/ansible_cloudera.cloud.env_payload_69lmx119/ansible_cloudera.cloud.env_payload.zip/ansible_collections/cloudera/cloud/plugins/module_utils/cdp_common.py\", line 42, in _impl\n File \"/tmp/ansible_cloudera.cloud.env_payload_69lmx119/ansible_cloudera.cloud.env_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/env.py\", line 687, in process\n File \"/tmp/ansible_cloudera.cloud.env_payload_69lmx119/ansible_cloudera.cloud.env_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/env.py\", line 926, in _reconcile_existing_state\nKeyError: 'logStorage'\n", "failed": true, "module_stderr": "Traceback (most recent call last):\n File \"/root/.ansible/tmp/ansible-tmp-1635181717.7778952-26659-167799228659899/AnsiballZ_env.py\", line 102, in \n _ansiballz_main()\n File \"/root/.ansible/tmp/ansible-tmp-1635181717.7778952-26659-167799228659899/AnsiballZ_env.py\", line 94, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/root/.ansible/tmp/ansible-tmp-1635181717.7778952-26659-167799228659899/AnsiballZ_env.py\", line 40, in invoke_module\n runpy.run_module(mod_name='ansible_collections.cloudera.cloud.plugins.modules.env', init_globals=None, run_name='main', alter_sys=True)\n File \"/usr/lib64/python3.8/runpy.py\", line 207, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.8/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib64/python3.8/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_cloudera.cloud.env_payload_69lmx119/ansible_cloudera.cloud.env_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/env.py\", line 1055, in \n File \"/tmp/ansible_cloudera.cloud.env_payload_69lmx119/ansible_cloudera.cloud.env_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/env.py\", line 1045, in main\n File \"/tmp/ansible_cloudera.cloud.env_payload_69lmx119/ansible_cloudera.cloud.env_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/env.py\", line 662, in init\n File \"/tmp/ansible_cloudera.cloud.env_payload_69lmx119/ansible_cloudera.cloud.env_payload.zip/ansible_collections/cloudera/cloud/plugins/module_utils/cdp_common.py\", line 42, in _impl\n File \"/tmp/ansible_cloudera.cloud.env_payload_69lmx119/ansible_cloudera.cloud.env_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/env.py\", line 687, in process\n File \"/tmp/ansible_cloudera.cloud.env_payload_69lmx119/ansible_cloudera.cloud.env_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/env.py\", line 926, in _reconcile_existing_state\nKeyError: 'logStorage'\n", "module_stdout": "{'environmentName': 'arcp-aw-env', 'crn': 'crn:cdp:environments:us-west-1:a0ec84c6-fee6-4e9c-acdc-68e1f49a5184:environment:714b0df3-e459-40ba-b722-837018456722', 'status': 'CREATE_FAILED', 'region': 'us-east-1', 'cloudPlatform': 'AWS', 'credentialName': 'arcp-aw-xaccount-cred', 'created': datetime.datetime(2021, 10, 14, 6, 53, 1, 412000, tzinfo=tzlocal())}\nsdf\nexisting\n{'environmentName': 'arcp-aw-env', 'crn': 'crn:cdp:environments:', 'status': 'CREATE_FAILED', 'region': 'us-east-1', 'cloudPlatform': 'AWS', 'credentialName': 'arcp-aw-xaccount-cred', 'created': datetime.datetime(2021, 10, 14, 6, 53, 1, 412000, tzinfo=tzlocal())}\narn:aws:iam:::instance-profile/arcp-logs-role\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1 } }

[Chaffelson]

Hi @mearchana , I see a KeyError with logStorage from line 926 in cloudera.cloud/plugings/modules/env.py, which is certainly unusual.

Can you please share the following:

• Version of cloudera-deploy you are using (it should be in the cmdline prompt)
• The definition.yml you are using, redacting any secrets
• The ansible-playbook command you used for execution, redacting any secrets

[mearchana]

HI @Chaffelson Cloudera-deploy version I got after I running quickstart script is 'cldr full-v1.4.2' I am using default definition.yml. /opt/cloudera-deploy/examples/sandbox/definition.yml Below is the ansible command. ansible-playbook /opt/cloudera-deploy/main.yml -e "definition_path=examples/sandbox" -t run,default_cluster

[Chaffelson]

ok, so all defaults. It's very interesting because in the current release logStorage isn't on that line at all, though I can see how it might cause an error looking to check if that value isn't present in the deployment being checked against. Do you already have an environment matching the name being deployed? the default name would be your name_prefix from your profile joined with -aw-env

[mearchana]

Hi, I have started from scratch . I have edited only below file for password and name_prefix. ~/.config/cloudera-deploy/profiles/default

following error I got "error": "{'base_error': ClientError('An error occurred: The following host groups are missing from the request: core_broker (Status Code: 400; Error Code: INVALID_ARGUMENT; Service: datahub; Operation: createAWSCluster; Request ID: faa2fb85-cfa5-460c-b29e-c0ead5854ddf;)'), 'ext_traceback': [' File \"/root/.ansible/tmp/ansible-tmp-1635362121.5234797-15015-46133891625934/AnsiballZ_datahub_cluster.py\", line 102, in \n _ansiballz_main()\n', ' File \"/root/.ansible/tmp/ansible-tmp-1635362121.5234797-15015-46133891625934/AnsiballZ_datahub_cluster.py\", line 94, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n', ' File \"/root/.ansible/tmp/ansible-tmp-1635362121.5234797-15015-46133891625934/AnsiballZ_datahub_cluster.py\", line 40, in invoke_module\n runpy.run_module(mod_name=\'ansible_collections.cloudera.cloud.plugins.modules.datahub_cluster\', init_globals=None, run_name=\'main\', alter_sys=True)\n', ' File \"/usr/lib64/python3.8/runpy.py\", line 207, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n', ' File \"/usr/lib64/python3.8/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n', ' File \"/usr/lib64/python3.8/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n', ' File \"/tmp/ansible_cloudera.cloud.datahub_cluster_payload_9gzvujg0/ansible_cloudera.cloud.datahub_cluster_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/datahub_cluster.py\", line 582, in \n', ' File \"/tmp/ansible_cloudera.cloud.datahub_cluster_payload_9gzvujg0/ansible_cloudera.cloud.datahub_cluster_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/datahub_cluster.py\", line 572, in main\n', ' File \"/tmp/ansible_cloudera.cloud.datahub_cluster_payload_9gzvujg0/ansible_cloudera.cloud.datahub_cluster_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/datahub_cluster.py\", line 402, in init\n', ' File \"/tmp/ansible_cloudera.cloud.datahub_cluster_payload_9gzvujg0/ansible_cloudera.cloud.datahub_cluster_payload.zip/ansible_collections/cloudera/cloud/plugins/module_utils/cdp_common.py\", line 42, in _impl\n result = f(self, *args, **kwargs)\n', ' File \"/tmp/ansible_cloudera.cloud.datahub_cluster_payload_9gzvujg0/ansible_cloudera.cloud.datahub_cluster_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/datahub_cluster.py\", line 439, in process\n', ' File \"/tmp/ansible_cloudera.cloud.datahub_cluster_payload_9gzvujg0/ansible_cloudera.cloud.datahub_cluster_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/datahub_cluster.py\", line 474, in create_cluster\n', ' File \"/usr/local/lib/python3.8/site-packages/cdpy/common.py\", line 512, in call\n parsed_err = CdpError(err)\n'], 'error_code': 'INVALID_ARGUMENT', 'violations': 'The following host groups are missing from the request: core_broker', 'message': 'Client request error', 'status_code': '400', 'rc': 1, 'service': 'datahub', 'operation': 'createAWSCluster', 'request_id': 'faa2fb85-cfa5-460c-b29e-c0ead5854ddf'}", "finished": 1,

[Chaffelson]

I'm sorry this hasn't been followed up - the issue with host groups missing was a bug in an upgrade release of the Kafka Datahub Template in CDP Public which is now resolved.

[wmudge]

Please reopen/create a new issue if you are still encountering these particular problems. Thanks!