DataPipeline Failure Notice Triggers Immediately

[davidski]

With the current pipeline, when the VulnPryer node goes into the setup state the DataPipeline failure notice is triggered immediately. The control script looks to be running fine and the vulnpryer node itself seems to be setting up okay (still in process at this time). This appears to be a problem with the pipeline logic.

[abbyyacat]

I have done a failed (setup_failed) and successful stack testing and only received a single notification for each testing. Please take note of the following changes I have done to achieve a successful stack (given the latest code including the new changes)

• fixed syntax error in iam_policies/opsworks_vulnpryer_resource_role_policym used ec2:* to ensure all authorization is provided (using "ec2:DescribeVolumes" and "ec2:AttachVolume" only results to "RightAws::AwsError: UnauthorizedOperation: You are not authorized to perform this operation." in attaching volume). I also removed the condition. OpsWorks automatically tags all resources with the name of the stack and layer that they are associated with. This can be used if you want to use condition by tags.
• updated Berksfile to accommodate the new changes int the repository (from davidski/cookbooks to davidski/sch-mongodb)

$ diff vulnpryer-stg/Berksfile chef-vulnpryer/Berksfile
6,7c6
< cookbook 'sch-mongodb', github: 'davidski/sch-mongodb'
< cookbook 'sch-base', github: 'davidski/sch-base'
---
> cookbook 'sch-mongodb', github: 'davidski/cookbooks', rel: 'sch-mongodb'
9c8

A probable reason why multiple notifications are sent is when 'Start Date Time' is scheduled behind the present date/time which could result to concurrent pipeline instances (backfilling) explaining the multiple SNS notifications.

[davidski]

@abbyyacat The failure notices are being created as soon as the stack starts to come up (the main VulnPryer node has not failed). I'll check to ensure the policy in this repo matches production (I tweaked this policy a bit yesterday) where the current OpsWorks stack generated by the deploy script works when manually triggered (not use data pipeline).

[davidski]

Here's the contents of our current Berksfile, BTW:

source "https://supermarket.getchef.com"

cookbook 'apt', "~> 2.6.0"
cookbook 'build-essential', "~> 2.1.3"
cookbook 'python', "~> 1.4.6"
cookbook 'mongodb', "= 0.16.1"
cookbook 'sch-base', github: 'davidski/sch-base', branch: 'master'
cookbook 'sch-mongodb', github: 'davidski/sch-mongodb', branch: 'master'
cookbook 'vulnpryer', github: 'davidski/chef-vulnpryer'
cookbook 'git', '~> 4.0.2'
cookbook 'chef-sugar'
cookbook 'cron'
cookbook 'aws'
cookbook 'windows', '~> 1.34.0'

[abbyyacat]

I see. I have to admit I haven't experienced something like this in my staging so far. So just to make sure my understanding is correct:

• When you say "The failure notices are being created as soon as the stack starts to come up (the main VulnPryer node has not failed)" -> you mean the VulnPryer node is still in a non-online and non-terminating/shutting down state when you received the failure notice?
• is this also the same time you receive multiple notification?
• does this happen always or only in certain cases?

[davidski]

• Yes, the VulnPryer node is in the "setup" phase when alerts start coming in about pipeline failure.
• Once started, we received alerts at:
  • 01/06/2015 15:49:34 UTC
  • 01/06/2015 15:55:39 UTC
  • 01/06/2015 15:56:33 UTC
  • 01/06/2015 15:59:33 UTC
  • 01/06/2015 16:04:33 UTC
  • 01/06/2015 16:09:32 UTC
  • Pipeline then manually deleted to stop alerts.
• This was only executed once.

[abbyyacat]

The SNS notification is configured to be sent by the Data Pipeline service using the On Fail and On Success fields. It is possible that the issue is in the pipeline and we may need to consult with AWS why this is happening. Although I think you already deleted it so not sure how we can proceed.

Is it possible if we start another pipeline again and observe if this recurs? I am also staging this from my end using what is in the repository and so far I haven't experienced this. Apologies if I cannot be of help in this case.

[davidski]

Config change from #12 looks to stop this as well. Glad it was an easy (and silly) error on my part!

[davidski]

As an FYI, we had a legitimate pipeline failure this morning (rubygems was throwing a service error, causing convergence to fail). The process correctly generated a single error via SNS.