snowflakedb / SnowAlert

Security Analytics Using The Snowflake Data Warehouse
Apache License 2.0
184 stars 57 forks source link

Can snowalert send attachments from s3 in email. #541

Open syedirfanahmedansari opened 3 years ago

syedirfanahmedansari commented 3 years ago

we have requirement where we want error files/csv files to be send as an attachment.

Can it be done using SES/SNS or simply SMTP handlers.

Please respond ASAP.

Thanks! Irfan.

sfc-gh-afedorov commented 3 years ago

Could you clarify what you mean by "from S3"?

syedirfanahmedansari commented 3 years ago

Can snowalert access AWS S3 to send attachments.

sfc-gh-afedorov commented 3 years ago

Snowflake can access S3 via PIPE's and External Functions which can access S3 via boto3, and SnowAlert runs on Snowflake, so yes, SnowAlert can access S3 to send attachments.

sfc-gh-afedorov commented 3 years ago

Reviewing their code (ses.py smtp.py), I don't think any of the current handlers will reach out to S3 directly. It's probably best that way -- their role is to send emails, not read data from S3.

syedirfanahmedansari commented 3 years ago

i don't see any part in ses.py/smtp.py where they take attachments.

so this means the current handlers don't support file attachment from any where.

sfc-gh-afedorov commented 3 years ago

Yes, per their docs, seems like you'd need to add attachment to the params and —

if attachment:
  att = MIMEApplication(attachment['content'])
  att.add_header('Content-Disposition', 'attachment', filename=attachment['filename'])
  message.attach(att)

to smtp.py for that to work.

We're moving over to using External Functions for handlers, tho, and will have news on that front soon.

Can update the ticket when that work is ready for public release.

syedirfanahmedansari commented 3 years ago

at our end snowalert runs on AWS ECR. if the attachment is to be added then that file needs to present in running machine. for attachment to work the file needs to present locally. My concern is how will it attach from outside. or how do i send the file to the docker instance.

syedirfanahmedansari commented 3 years ago

Can you send me a sample for using SES handler

sfc-gh-afedorov commented 3 years ago

we don't have one for SES on hand (it's a third party contribution that my team doesn't use), but SMTP is documented here. all the handlers are the same interface, tho -- the "type" key specifies which Python module is loaded, and the rest of the keys are passed in as parameters to the "handle" method in that module.

syedirfanahmedansari commented 3 years ago

Can you please share a sample select for SMTP handler

sfc-gh-afedorov commented 3 years ago

https://docs.snowalert.com/alerts/handlers/smtp

syedirfanahmedansari commented 3 years ago

Hi,

Can you please provide Steps to configure Snowalert on AWS using Cloud formation Template.

Thanks! Irfan.

sfc-gh-afedorov commented 3 years ago

We have not deployed on cfn ourselves, but someone who has provided this code and these instructions. Does that help?

syedirfanahmedansari commented 3 years ago

Hi,

it seems we are getting duplicate alerts.

the way we are running is we have a view in snowflake to which the final Alert query is pointing and we have scheduled snow alert to run for every 5 minutes.

if any entry comes in two executions then the alerts are being published twice. how can we suppress the second alert.

sfc-gh-afedorov commented 3 years ago

Alerts are deduplicated by object and description, so if those are identical and the alerts are within the look-back period, you should be good. Could you check whether e.g. you are including the current time in the description, perhap?