search

awslabs / aws-service-catalog-puppet

This is a framework where you list your AWS accounts with tags and your AWS Service Catalog products with tags or target accounts. The framework works through your lists, dedupes and spots collisions and then provisions the products into your AWS accounts for you. It handles the Portfolio sharing, its acceptance and can provision products cross account and cross region.
Apache License 2.0
77 stars 41 forks source link

Puppet pipeline failed at deploy stage after upgraded to version 0.242.0 #687

Closed vasindran closed 8 months ago

vasindran commented 11 months ago

Describe the bug We have upgraded the puppet version to 0.242.0, and triggered the puppet pipeline post stack upgrade, but its throwing error at deploy stage.

To Reproduce Steps to reproduce the behavior:

  1. Go to CodePipeline after you upgraded the puppet version 0.242.0
  2. Click on Puppet pipeline's release change button
  3. Scroll down to check the deploy state for logs

  4. See error at logs Traceback (most recent call last):

    266 | File "/root/.pyenv/versions/3.9.17/bin/servicecatalog-puppet", line 8, in 267 | sys.exit(cli()) 268 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/click/core.py", line 764, in call 269 | return self.main(args, kwargs) 270 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/click/core.py", line 717, in main 271 | rv = self.invoke(ctx) 272 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/click/core.py", line 1137, in invoke 273 | return _process_result(sub_ctx.command.invoke(sub_ctx)) 274 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/click/core.py", line 956, in invoke 275 | return ctx.invoke(self.callback, ctx.params) 276 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/click/core.py", line 555, in invoke 277 | return callback(args, kwargs) 278 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/servicecatalog_puppet/cli.py", line 407, in generate_task_reference 279 | task_reference_commands.generate_task_reference(f) 280 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/servicecatalog_puppet/commands/task_reference.py", line , in generate_task_reference 281 | complete = complete_generator.generate( # hub and spokes 282 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/servicecatalog_puppet/commands/task_reference_helpers/complete_generator.py", line 197, in generate 283 | ssm_parameter_handler( 284 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/servicecatalog_puppet/commands/task_reference_helpers/generators/ssm_parameter_handler.py", line 32, in ssm_parameter_handler 285 | ssm_parameter_details.get("account_id", puppet_account_id) 286 | AttributeError: 'int' object has no attribute 'replace' 287 |   288 | [Container] 2023/10/03 15:18:28 Command did not exit successfully servicecatalog-puppet --info generate-task-reference $PWD/manifest-expanded.yaml exit status 1 289 | [Container] 2023/10/03 15:18:28 Phase complete: PRE_BUILD State: FAILED 290 | [Container] 2023/10/03 15:18:28 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: servicecatalog-puppet --info generate-task-reference $PWD/manifest-expanded.yaml. Reason: exit status 1

Expected behavior Puppet pipeline should be successfully run and deployed the stuff.

Attachments Please find the details log attached/. PuppetLog.txt

vasindran commented 11 months ago

When i have run spoke mode in latest version of puppet 0.242.0, i am getting this below error, but hub mode is works fine.

Created new drift_token: 2023-**-05 11:49:21.800021 -- 224 | Created new run_token: 2023-**-05 11:49:21.800087 225 | INFO MainProcess MainThread getting should_collect_cloudformation_events, default_region: None 226 | INFO MainProcess MainThread getting config, default_region: None 227 | INFO MainProcess MainThread getting should_use_eventbridge, default_region: None 228 | INFO MainProcess MainThread getting should_forward_failures_to_opscenter, default_region: None 229 | INFO MainProcess MainThread getting should_delete_rollback_complete_stacks, default_region: None 230 | INFO MainProcess MainThread getting should_use_product_plans, default_region: us-west-1 231 | INFO MainProcess MainThread getting global_sharing_mode_default, default_region: None 232 | INFO MainProcess MainThread getting global_share_tag_options_default, default_region: None 233 | INFO MainProcess MainThread getting global_share_principals_default, default_region: None 234 | INFO MainProcess MainThread getting spoke_deploy_environment_compute_type, default_region: None 235 | INFO MainProcess MainThread getting scheduler_threads_or_processes, default_region: None 236 | INFO MainProcess MainThread getting scheduler_algorithm, default_region: None 237 | Traceback (most recent call last): 238 | File "/root/.pyenv/versions/3.9.17/bin/servicecatalog-puppet", line 8, in 239 | sys.exit(cli()) 240 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/click/core.py", line 764, in __call__ 241 | return self.main(*args, **kwargs) 242 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/click/core.py", line 717, in main 243 | rv = self.invoke(ctx) 244 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/click/core.py", line 1137, in invoke 245 | return _process_result(sub_ctx.command.invoke(sub_ctx)) 246 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/click/core.py", line 956, in invoke 247 | return ctx.invoke(self.callback, **ctx.params) 248 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/click/core.py", line 555, in invoke 249 | return callback(*args, **kwargs) 250 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/servicecatalog_puppet/cli.py", line 407, in generate_task_reference 251 | task_reference_commands.generate_task_reference(f) 252 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/servicecatalog_puppet/commands/task_reference.py", line 36, in generate_task_reference 253 | hub_tasks = hub_generator.generate( # hub only 254 | File "/root/.pyenv/versions/3.9.17/lib/python3.9/site-packages/servicecatalog_puppet/commands/task_reference_helpers/hub_generator.py", line 74, in generate 255 | raise Exception( 256 | Exception: describe-provisioning-parameters-794537503075-us-west-1-Networking-multi-region-product-v1 depends on: portfolio-puppet-role-association-794537503075-us-west-1-Networking which is not listed in this reference 257 |   258 | [Container] 2023/**/05 11:49:22 Command did not exit successfully servicecatalog-puppet --info generate-task-reference $PWD/manifest-expanded.yaml exit status 1 259 | [Container] 2023/**/05 11:49:22 Phase complete: PRE_BUILD State: FAILED 260 | [Container] 2023/**/05 11:49:22 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: servicecatalog-puppet --info generate-task-reference $PWD/manifest-expanded.yaml. Reason: exit status 1
eamonnfaherty commented 11 months ago

Could you share your manifest file and your logs please.

eamonnfaherty commented 11 months ago

After an offline call this was diagnosed as:

When running a launch in a spoke account that is the puppet account via execution mode spoke the portfolio association for the puppet role is not added to the hub task reference file - it is programmed to run in spoke execution mode. This breaks describe provisioning parameters as the puppet role association is needed. Describe provisioning parameters should run in hub before spokes for performance reasons.