search

ducoquelicot / observ_admin

All admin and back-end for Observ
0 stars 0 forks source link

Fix mail configurations #18

Open zstumgoren opened 5 years ago

zstumgoren commented 5 years ago

In order to help debug the Flask ctx.app bug, can you update the config.py and .env files to include whatever default configurations you're using. Note that environment variables should always be in caps.

See mega tutorial code below for example:

zstumgoren commented 5 years ago

It's not clear to me if you're using Gmail or the test Python mail server for dev or testing purposes, as described in docs here.

I am however getting a stacktrace when I attempt to subscribe. The primary error appears to be related to misconfiguration of mail in my case, followed by the ctx.app error. It's not clear to me if the latter is related to and/or caused by this mail configuration issue, but will need to sort out the mail issue first. When you have a chance, please drop a note with clarification on how you're handling mail configuration on your end.

send: 'quit\r\n'
Exception in thread Thread-10:
Traceback (most recent call last):
  File "/Users/tumgoren/.local/share/virtualenvs/observ_admin-g2ZsYChj/lib/python3.6/site-packages/flask_mail.py", line 492, in send
    message.send(connection)
  File "/Users/tumgoren/.local/share/virtualenvs/observ_admin-g2ZsYChj/lib/python3.6/site-packages/flask_mail.py", line 427, in send
    connection.send(self)
  File "/Users/tumgoren/.local/share/virtualenvs/observ_admin-g2ZsYChj/lib/python3.6/site-packages/flask_mail.py", line 192, in send
    message.rcpt_options)
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/smtplib.py", line 851, in sendmail
    self.ehlo_or_helo_if_needed()
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/smtplib.py", line 599, in ehlo_or_helo_if_needed
    if not (200 <= self.ehlo()[0] <= 299):
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/smtplib.py", line 439, in ehlo
    self.putcmd(self.ehlo_msg, name or self.local_hostname)
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/smtplib.py", line 366, in putcmd
    self.send(str)
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/smtplib.py", line 358, in send
    raise SMTPServerDisconnected('please run connect() first')
smtplib.SMTPServerDisconnected: please run connect() first

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/tumgoren/code/stanford/thesis/fabienne/observ_admin/app/emails.py", line 8, in send_async_email
    mail.send(msg)
  File "/Users/tumgoren/.local/share/virtualenvs/observ_admin-g2ZsYChj/lib/python3.6/site-packages/flask_mail.py", line 492, in send
    message.send(connection)
  File "/Users/tumgoren/.local/share/virtualenvs/observ_admin-g2ZsYChj/lib/python3.6/site-packages/flask_mail.py", line 152, in __exit__
    self.host.quit()
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/smtplib.py", line 982, in quit
    res = self.docmd("quit")
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/smtplib.py", line 419, in docmd
    self.putcmd(cmd, args)
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/smtplib.py", line 366, in putcmd
    self.send(str)
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/smtplib.py", line 358, in send
    raise SMTPServerDisconnected('please run connect() first')
smtplib.SMTPServerDisconnected: please run connect() first

127.0.0.1 - - [15/May/2019 18:07:07] "POST /subscribe HTTP/1.1" 200 -
127.0.0.1 - - [15/May/2019 18:07:07] "GET /static/subscribe.js HTTP/1.1" 200 -
127.0.0.1 - - [15/May/2019 18:07:07] "GET /static/index_style.css HTTP/1.1" 200 -
127.0.0.1 - - [15/May/2019 18:07:07] "GET /static/perPage.js HTTP/1.1" 200 -
127.0.0.1 - - [15/May/2019 18:07:07] "GET /static/prepopulate.js HTTP/1.1" 200 -
127.0.0.1 - - [15/May/2019 18:07:07] "GET /static/images/backdrop_laptops.jpg HTTP/1.1" 200 -
Job "stumgoren (trigger: interval[0:00:05], next run at: 2019-05-15 18:07:17 PDT)" raised an exception
Traceback (most recent call last):
  File "/Users/tumgoren/.local/share/virtualenvs/observ_admin-g2ZsYChj/lib/python3.6/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/Users/tumgoren/code/stanford/thesis/fabienne/observ_admin/app/tasks.py", line 25, in sub_job
    send_results_email(user, sub, output)
  File "/Users/tumgoren/code/stanford/thesis/fabienne/observ_admin/app/emails.py", line 42, in send_results_email
    user=user, subscription=subscription, results=results),
  File "/Users/tumgoren/.local/share/virtualenvs/observ_admin-g2ZsYChj/lib/python3.6/site-packages/flask/templating.py", line 133, in render_template
    ctx.app.update_template_context(context)
ducoquelicot commented 5 years ago

Should be fixed now

ducoquelicot commented 5 years ago

I'm handling it by using my dev gmail that I created last quarter. I updated the env file with the configs that I'm using.

zstumgoren commented 5 years ago

Can't seem to get this working with your gmail credentials or the standard mail debugger server credentials. I should've mentioned earlier, but for dev purposes, it would be better to use the mail debug server for testing, and to avoid committing your personal mail server credentials in .env.

Anyhow, can't seem to get past this error. Don't think I'll be able to look at this again until tomorrow night or Friday. Meantime, I would try to get the stack working with the mail debug server and then verify that you're not in fact getting the same SMTP related errors that I am. It's a really long error message, and the ctx.app error occurs downstream and keeps repeating. So I'm wondering if there's a chance you're also getting this SMTP related error but possibly only seeing the ctx.app error because you didn't scroll up high enough in the logs.

To test, change the following configs in .env file:

FLASK_DEBUG=0
MAIL_SERVER=localhost
MAIL_HOST=8025

Execute flask run in your terminal (not in VSCode), so you can more easily scroll back in history to see all the server logs.

Then try triggering the error on subscribe via the app's web interface.

Once you trigger the error, immediately kill the Flask server in the terminal and then check the console for the full history of the error. This may require a fair bit of scrolling up.

Let me know if you see something like this:

Traceback (most recent call last):
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/tumgoren/code/stanford/thesis/fabienne/observ_admin/app/emails.py", line 8, in send_async_email
    mail.send(msg)
  File "/Users/tumgoren/.local/share/virtualenvs/observ_admin-g2ZsYChj/lib/python3.6/site-packages/flask_mail.py", line 491, in send
    with self.connect() as connection:
  File "/Users/tumgoren/.local/share/virtualenvs/observ_admin-g2ZsYChj/lib/python3.6/site-packages/flask_mail.py", line 144, in __enter__
    self.host = self.configure_host()
  File "/Users/tumgoren/.local/share/virtualenvs/observ_admin-g2ZsYChj/lib/python3.6/site-packages/flask_mail.py", line 158, in configure_host
    host = smtplib.SMTP(self.mail.server, self.mail.port)
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/smtplib.py", line 251, in __init__
    (code, msg) = self.connect(host, port)
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/smtplib.py", line 335, in connect
    self.sock = self._get_socket(host, port, self.timeout)
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/smtplib.py", line 306, in _get_socket
    self.source_address)
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/socket.py", line 722, in create_connection
    raise err
  File "/Users/tumgoren/.pyenv/versions/3.6.1/lib/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

127.0.0.1 - - [15/May/2019 19:39:45] "POST /subscribe HTTP/1.1" 200 -
127.0.0.1 - - [15/May/2019 19:39:45] "GET /static/subscribe.js HTTP/1.1" 200 -
127.0.0.1 - - [15/May/2019 19:39:45] "GET /static/perPage.js HTTP/1.1" 200 -
127.0.0.1 - - [15/May/2019 19:39:45] "GET /static/index_style.css HTTP/1.1" 200 -
127.0.0.1 - - [15/May/2019 19:39:45] "GET /static/prepopulate.js HTTP/1.1" 200 -
127.0.0.1 - - [15/May/2019 19:39:45] "GET /static/images/backdrop_laptops.jpg HTTP/1.1" 200 -
Job "stumgoren (trigger: interval[0:00:05], next run at: 2019-05-15 19:39:55 PDT)" raised an exception
Traceback (most recent call last):
  File "/Users/tumgoren/.local/share/virtualenvs/observ_admin-g2ZsYChj/lib/python3.6/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/Users/tumgoren/code/stanford/thesis/fabienne/observ_admin/app/tasks.py", line 22, in sub_job
    send_results_email(user, sub, output)
  File "/Users/tumgoren/code/stanford/thesis/fabienne/observ_admin/app/emails.py", line 42, in send_results_email
    user=user, subscription=subscription, results=results),
  File "/Users/tumgoren/.local/share/virtualenvs/observ_admin-g2ZsYChj/lib/python3.6/site-packages/flask/templating.py", line 133, in render_template
    ctx.app.update_template_context(context)
AttributeError: 'NoneType' object has no attribute 'app'
ducoquelicot commented 5 years ago

I tried to use the mail debugger first way back when but I couldn't get that to work either, so I had to switch to gmail in order to get it to work. So I don't think I can get any further on that than you unfortunately.

ducoquelicot commented 5 years ago

But I'll give a try and see where I get stuck.

ducoquelicot commented 5 years ago

By the way, the reason you can't get it to work is very very possibly because I just got an email that Google blocked a sign in attempt for my dev email. So, that's probably a gmail thing - there's nothing wrong with the credentials themselves.

ducoquelicot commented 5 years ago

So, just to check, I ran it with the debug settings as well as my email settings.

The issue is indeed similar at the top when I'm running in the debug mail server, I did get a ConnectionError [Errno 111] << instead of Errno 61 >> but it was indeed a 'Connection Refused' at the top.

I'll have to check if it's the same when I use my own settings.

ducoquelicot commented 5 years ago

@zstumgoren Sooooo, I tried running with my original settings again - I guess I had to restart the terminal - but here's the kicker: when I use my gmail settings (and the sub email is being sent) there is no trace of a ConnectionError at the top.

Here's the full (first) error:

Job "fabienne (trigger: interval[0:00:05], next run at: 2019-05-15 20:21:53 PDT)" raised an exception
Traceback (most recent call last):
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/home/fabienne/Desktop/Observ/app/tasks.py", line 25, in sub_job
    send_results_email(user, sub, output)
  File "/home/fabienne/Desktop/Observ/app/emails.py", line 42, in send_results_email
    user=user, subscription=subscription, results=results),
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/templating.py", line 133, in render_template
    ctx.app.update_template_context(context)
AttributeError: 'NoneType' object has no attribute 'app'

Looks like the SMTP errors are only present when one uses localhost as a debug mail server. Unfortunately.

zstumgoren commented 5 years ago

One quick addendum -- I think ctx.app bug may be related to the use of threading in the async mail function. Likely unrelated to the mail configuration issue.

zstumgoren commented 5 years ago

The more I think about this, the more I suspect this is thread-related.

The APScheduler BackgroundScheduler already runs in a thread.

And on top of that, you're sending mail inside another thread:

https://github.com/ducoquelicot/observ_admin/blob/master/app/emails.py#L14

So my best guess is that the application context is being lost or is simply unavailable to a thread (send_email) nested inside another thread (the scheduler).

I know the Mega Tutorial includes a section on sending mail asynchronously using a thread, but since we're already using an asynchronous task scheduling service, this is one area we'll need to do things differently.

I would start by removing the Thread call wrapping send_email:

 Thread(target=send_async_email, args=(observ, msg)).start()

And then see what bugs crop up.

zstumgoren commented 5 years ago

And of course, rewire the code to just send an email synchronously. For example:

   10 def send_email(subject, sender, recipients, text_body, html_body):
   11     msg = Message(subject, sender=sender, recipients=recipients)
   12     msg.body = text_body
   13     msg.html = html_body
~  14     with observ.app_context():
+  15         mail.send(msg)

This will still be run asynchronously by the scheduler, but it avoids creating an unnecessary thread inside of the scheduler.

ducoquelicot commented 5 years ago

Ok! I'll see if that works.

@zstumgoren Update: that unfortunately did not make any difference. Error message is still the same.

ducoquelicot commented 5 years ago

I was/am googling, tried this: https://stackoverflow.com/questions/31156091/attributeerror-nonetype-object-has-no-attribute-app-when-trying-to-send-mai

Didn't work either, unfortunately. Error message is still the same.

edit: sub email is still working, though, it's really only the email that is inside the job.

ducoquelicot commented 5 years ago

I think this is the problem: https://stackoverflow.com/questions/24579091/flask-periodically-rendering-html-without-context

I checked and all the other emails are being sent from within a route, so it makes sense. Unfortunately I have no idea how to use this solution on my use case. I'm sure you have some clever ideas, though!

zstumgoren commented 5 years ago

Side note, I got the debug email server to work by just using defaults of localhost and port 25. I'm sure there's a way to get it to work with a custom port, but just going with the defaults for now to avoid further yak shaving.

Yep, the missing application context is indeed the key, although the nature of threading (which is notoriously hard to debug) further complicates things. As a first step, I would definitely remove the threading in the send_async_email function (actually, you can delete it entirely) and just update the remaining send_email function to just send the mail in the standard way. We already have a threaded async job scheduler/runner in the background, so anywhere you need an email, just use the scheduler. If the message needs to be sent immediately, just omit the trigger and time arguments when you call add_job.

Once you've standardized things on that front, move on to solving the app context problem.

The Flask modules that are being used in sub_job really, REALLY want that app context that you get in a normal request-response cycle. But because this work is happening asynchronously outside the context of a normal web request, the app context is not available.

At a high-level, there are two potential solutions:

  1. Figure out a way to force the async jobs to recreate the application state, e.g. using something like the create_app function in __init__.py and then importing and executing the function inside of the module where the email functions live (or perhaps inside the functions themselves). For example, something along these lines
  2. Sidestep the Flask app context entirely and just leverage the bits of configuration and data that you need for rendering a template and sending mail. This may require using raw Jinja2 to load and render the templates if Flask's render_template requires an app context (which it seems to...) and reading in the mail configurations and setting up an SMTP mailer instance instead of trying to use the mailer instance you've created in __init__.py. I began to try this approach using Jinja2 and it worked, but then Flask Mailer complained that the application context was not available, which is a clue that you'll need to set up and configure SMTP yourself if you go with this approach.

It might be worth trying (1), though I'm not certain that will work. Option (2) fully decouples you from the Flask application itself, which might be "cleaner" and easier to code rather than trying to figure out the nuances of how to force a Flask app context into an async job.

Give one or both of those options a try and let me know if you run into trouble.

zstumgoren commented 5 years ago

On second thought, you might want to keep the send_async_email for now, until you figure out a solution to sending emails via scheduler. Otherwise, all the email functionality will be broken.

So I'd just focus on working through a solution to getting the scheduler-based emails to work. Again, I'd lean toward option (2) above.

ducoquelicot commented 5 years ago

@zstumgoren re: sending the emails via the scheduler, it' s not working. I don't know why, but when I tried to test the jobs I created, the following things happened

[I was testing the 'reset password' email here]

Traceback (most recent call last):
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/fabienne/Desktop/Observ/app/routes.py", line 135, in reset_password_request
    scheduler.add_job(func=send_password_reset_email, args=(user))
TypeError: add_job() missing 1 required positional argument: 'id'

WEIRD THING HAPPENED: the email was sent nonetheless. 

Traceback (most recent call last):
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/fabienne/Desktop/Observ/app/routes.py", line 135, in reset_password_request
    scheduler.add_job(send_password_reset_email(user))
TypeError: add_job() missing 1 required positional argument: 'func'

AGAIN the email was sent nonetheless

Traceback (most recent call last):
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/fabienne/Desktop/Observ/app/routes.py", line 135, in reset_password_request
    scheduler.add_job(id='pass_reset', func=send_password_reset_email(user))
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask_apscheduler/scheduler.py", line 160, in add_job
    return self._scheduler.add_job(**job_def)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/apscheduler/schedulers/base.py", line 434, in add_job
    job = Job(self, **job_kwargs)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/apscheduler/job.py", line 49, in __init__
    self._modify(id=id or uuid4().hex, **kwargs)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/apscheduler/job.py", line 170, in _modify
    raise TypeError('func must be a callable or a textual reference to one')
TypeError: func must be a callable or a textual reference to one

AGAIN email was being sent

No emails at this point.

I tried it without the tuple around 'user' but that gave the same error. Again, no emails. I tried it with a list around 'user' and that gave no errors, but also didn't send any emails.

When I tried a similar thing for the send_sub_email I got this error:

Traceback (most recent call last):
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/home/fabienne/Desktop/Observ/app/routes.py", line 55, in subscribe
    scheduler.add_job(id='sub_email', func=send_sub_email, args=(current_user, sub))
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/flask_apscheduler/scheduler.py", line 160, in add_job
    return self._scheduler.add_job(**job_def)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/apscheduler/schedulers/base.py", line 443, in add_job
    self._real_add_job(job, jobstore, replace_existing)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/apscheduler/schedulers/base.py", line 867, in _real_add_job
    store.add_job(job)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/apscheduler/jobstores/sqlalchemy.py", line 95, in add_job
    'job_state': pickle.dumps(job.__getstate__(), self.pickle_protocol)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f505a4771e0>: attribute lookup <lambda> on flask_login.utils failed

And so I just put everything back in its original state. Don't really know how to fix this, but will see if I can find a way to fix the email for the results and have the background thread for the async emails in there at the same time.

ducoquelicot commented 5 years ago

EDIT: I pushed to master so you can check it out for yourself

@zstumgoren so, good news and bad news. The good news is that I got the emails to work with Jinja, no comments whatsoever from Mail being angry with me so I guess we' re good? Unless you' re still stuck, in that case let me know.

New issue: when I create a subscription, everything works as expected. I get a confirmation email and then a couple of 'new results' emails (with no content, this is because I have changed the code a little bit so that it runs when there are no new results, this should of course be fixed but it's good for testing purposes)

But! When I stop the app and restart it, I get the following error:

Unable to restore job "1" -- removing it
Traceback (most recent call last):
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/apscheduler/jobstores/sqlalchemy.py", line 141, in _get_jobs
    jobs.append(self._reconstitute_job(row.job_state))
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/apscheduler/jobstores/sqlalchemy.py", line 128, in _reconstitute_job
    job.__setstate__(job_state)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/apscheduler/job.py", line 272, in __setstate__
    self.func = ref_to_obj(self.func_ref)
  File "/home/fabienne/.local/share/virtualenvs/Observ-r9e1nbjd/lib/python3.6/site-packages/apscheduler/util.py", line 283, in ref_to_obj
    obj = __import__(modulename, fromlist=[rest])
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
KeyError: 'app'

The job is removed and as such, there is no task accompanying the subscription anymore. The other job (I mean, the scraperjob) is still active though (I checked).

What I understand from the message is that something is going wrong in the obj = __import__(modulename, fromlist=[rest]) line in the APScheduler code, and that 'rest' here is replaced with 'app' and that this is a key error because this key does not exist in the dictionary.

I, however, have no idea where this ' app' is coming from. It's not in the task, and I don't really know where else to look. You any clues?