Email notificaitions not sending

[joedborg]

See https://github.com/getsentry/sentry/issues/2549 for full story

>>> mail.get_send_to(project)
[1]

...shows that I'm on the list.

Clicking "send test" on the Mail admin page works fine, as does

project = Project.objects.get(slug='django')
message = MessageBuilder('test')
message.add_users((1,), project)
message.send()

But I never get emails for new events or regressions.

[mattrobenolt]

What are you using for smtp, out of curiosity? Since you need to either link an smtp container, or point to an external host.

[joedborg]

It's an external SMTP server

[mattrobenolt]

I will test this out today and verify that everything should be good, and we'll go from there. :)

[joedborg]

Thanks @mattrobenolt, let me know if you need anything. For reference, this is my docker-compose file:

redis:
  image: redis

postgres:
  image: postgres
  environment:
    - POSTGRES_PASSWORD=sentry
    - POSTGRES_USER=sentry
  volumes:
    - /var/lib/postgresql/data

sentry:
  image: sentry:latest
  environment:
    - SENTRY_EMAIL_HOST=(OMITTED)
    - SENTRY_SERVER_EMAIL=(OMITTED)
  links:
    - redis
    - postgres
  ports:
    - "9999:9000"

celery-beat:
  image: sentry:latest
  links:
    - redis
    - postgres
  command: sentry celery beat

celery-worker:
  image: sentry:latest
  links:
    - redis
    - postgres
  command: sentry celery worker

And just to confirm, I did a docker pull sentry:latest to ensure I'm at 8.0.2.

[joedborg]

Might have been fixed in 8.0.4; will test when sentry:latest is updated

[mattrobenolt]

Waiting on https://github.com/docker-library/official-images/pull/1368 :(

[joedborg]

Ah wow, I thought it was a mere docker push away!

[mattrobenolt]

Not for the official library, unfortunately. :(

Matt Robenolt @mattrobenolt

On Mon, Jan 25, 2016 at 8:06 AM -0800, "Joe Borg" notifications@github.com wrote:

Ah wow, I thought it was a mere docker push away!

— Reply to this email directly or view it on GitHub.

[joedborg]

8.0.4 hasn't fixed the issue unfortunately (have updated the HTTP server as well as Celery workers).

[mattrobenolt]

Alright. Will investigate more today. :(

Matt Robenolt @mattrobenolt

On Tue, Jan 26, 2016 at 12:34 AM -0800, "Joe Borg" notifications@github.com wrote:

8.0.4 hasn't fixed the issue unfortunately (have updated the HTTP server as well as Celery workers).

— Reply to this email directly or view it on GitHub.

[joedborg]

Not sure how relevant this is.

[Evedel]

Hello! Just want to add, that it is not only mail notifications, slack-webhooks also cannot be sent now.

[mattrobenolt]

slack-webhooks also cannot be sent now

And they did work before?

[Evedel]

It is just initial installation, so I cannot say about Sentry, but this web-hooks work fine on the other systems. It looks for me like something wrong with "rules", because it is possible to send invitations and test mails.

[mattrobenolt]

I finally got around to testing this, and I confirmed that this isn't working correctly for some reason.

Emails are sent from the web container, but not from celery containers. :( This seems odd, but at least I can reproduce.

[mattrobenolt]

@Evede1 @joedborg Does it work if you set an explicit SENTRY_SECRET_KEY environment variable on all the containers?

I'm still investigating the issue, but 90% sure if the worker and web containers have different secret keys, rules are failing to notify, but this seems odd to me.

If this is the case, I should be more explicit in documentation about setting and using SENTRY_SECRET_KEY. This is basically required, and maybe fail to start the process if there isn't one defined.

I tested out a full cluster setup with linking tianon/exim4 for SMTP relay and before setting SENTRY_SECRET_KEY, I wasn't getting email notifications, but after setting, I started to get them.

[mattrobenolt]

In 8.1, I've changed the behavior around this where a SENTRY_SECRET_KEY is required to be set. It will error otherwise starting up. This should help assert that all containers are in sync since you'd have to be explicit about passing this now.

[Evedel]

@mattrobenolt, unfortunately, I still cannot receive notifications, but after setting the SENRY_SECRET_KEY some activity is originated. The postfix logs doesn't contain any lines about these actions or errors.

In the same time

[joedborg]

@mattrobenolt Can these be added as environments in the YML?

[joedborg]

redis:
    image: redis

postgres:
  image: postgres
  environment:
    - POSTGRES_PASSWORD=sentry
    - POSTGRES_USER=sentry
  volumes:
    - /var/lib/postgresql/data

sentry:
  image: sentry
  environment:
    - SENTRY_EMAIL_HOST=(OMITTED)
    - SENTRY_SERVER_EMAIL=(OMITTED)
    - SENTRY_SECRET_KEY=(OMITTED)
  links:
    - redis
    - postgres
  ports:
    - "9999:9000"

celery-beat:
  image: sentry
  environment:
    - SENTRY_SECRET_KEY=(OMITTED)
  links:
    - redis
    - postgres
  command: sentry celery beat

celery-worker:
  image: sentry
  environment:
    - SENTRY_SECRET_KEY=(OMITTED)
  links:
    - redis
    - postgres
  command: sentry celery worker

I have used the same key for all 3.

[mattrobenolt]

@Evede1 do webhooks get delivered or any other notifications?

@joedborg which do you want in the yaml? All of these environment variables get handled by the sentry.conf.py now. It's not necessary that they exist in the yaml file.

[ocomsoft]

I had a similar issue turns out I need to add the smtp link to the celery containers ie

docker run -d --name sentry-log --restart="always" --link smtp:smtp --link sentry-redis:redis --link postgres:postgres   sentry

docker run -d --restart="always" --name sentry-celery-beat --link postgres:postgres --link smtp:smtp --link sentry-redis:redis sentry celery beat

docker run -d --restart="always" --name sentry-celery1 --link postgres:postgres --link smtp:smtp  --link sentry-redis:redis sentry celery worker

[mattrobenolt]

@ocomsoft Did you only link them to the web container first?

Seems like we could clarify our documentation around here.

[ocomsoft]

Yes - I linked it to the Web container ran the test it worked!!

But I was NOT getting notifications... Tried adding it to Celery and BINGO..

I was going to raise an issue about the documentation because Yes it could be clearer. But figured this might be a better place to mention it.

[mattrobenolt]

Ah, good to know.

I'm just going to assume these are all related to config issues.

I guess it's not obvious, but most of the work that Sentry does is handled by the workers. Much less is handled by the web process itself.

[Evedel]

@mattrobenolt, thank you! Mails notification works great, after setting were added for all containers.

[joedborg]

@mattrobenolt I just wanted to check that there was nothing obvious in the file. Any other things I can investigate?

[mattrobenolt]

I have not been able to reproduce this in any way. :( The only thing I did notice is the default From email (root@localhost) tends to cause mail to get delivered to spam, but other than that, mail is always delivered in any configuration I've tried.

[frnknglrt]

The error mentioned in this issue is caused by the fact that the celery worker in the original docker-compose file does not use the intended email server.

In order to fix this error, please make sure to set the follwing environment variables also for the sentry worker:

• SENTRY_EMAIL_HOST=(OMITTED)
• SENTRY_SERVER_EMAIL=(OMITTED)
• SENTRY_SECRET_KEY=(OMITTED)

Like so:

redis:
    image: redis

postgres:
  image: postgres
  environment:
    - POSTGRES_PASSWORD=sentry
    - POSTGRES_USER=sentry
  volumes:
    - /var/lib/postgresql/data

sentry:
  image: sentry
  environment:
    - SENTRY_EMAIL_HOST=(OMITTED)
    - SENTRY_SERVER_EMAIL=(OMITTED)
    - SENTRY_SECRET_KEY=(OMITTED)
  links:
    - redis
    - postgres
  ports:
    - "9999:9000"

celery-beat:
  image: sentry
  environment:
    - SENTRY_SECRET_KEY=(OMITTED)
  links:
    - redis
    - postgres
  command: sentry celery beat

celery-worker:
  image: sentry
  environment:
    - SENTRY_SECRET_KEY=(OMITTED)
    - SENTRY_EMAIL_HOST=(OMITTED)
    - SENTRY_SERVER_EMAIL=(OMITTED)
  links:
    - redis
    - postgres
  command: sentry celery worker