search

revpriest / listman

An email-list manager for NextCloud
GNU General Public License v3.0
19 stars 3 forks source link

Emails not being sent #11

Closed koehn closed 3 years ago

koehn commented 3 years ago

When I send a message using the web form to a list with multiple members, nothing happens. There's nothing in nextcloud.log, and my mail server is never contacted.

The admin UI shows:

image

I'm using the same host/port/username/pw that I use for nextcloud's email server, so I know it's not a configuration issue.

revpriest commented 3 years ago

Not sure what could cause this one.

Are you comfortable looking at the database?

If you look in the ob_jobs table is the one with the class "OCA\Listman\Cron\ListmanSend"

select * from oc_jobs

If not, that'd mean it'd failed to create the cron-job to do the actual sending somehow.

But the "rate" being 10 there probably means it's done a send-job a few times, with presuably the 3 mails still not leaving the queue. Not sure what could cause that really.

Does the "rate" change over the course of ten minutes? Does the number in the queue ever change?

Thanks.

koehn commented 3 years ago 
select * from oc_jobs where class like '%ListmanSend';
  id  |            class             | argument |  last_run  | last_checked | reserved_at | execution_duration 
------+------------------------------+----------+------------+--------------+-------------+--------------------
 3457 | OCA\Listman\Cron\ListmanSend | null     | 1626609630 |   1626609630 |  1626609630 |                  0
(1 row)

The rate is currently 64 per 5 mins despite me not sending anything for a few days.

revpriest commented 3 years ago

The rate should double every cron-call until the list is empty.

Sounds like it's running the cron, but failing to send to mail for some reason. It must at least realized it's failed or it'd be marking them as sent and thus eventually running out and setting the rate back to zero.

There is a "max to send per day" setting, presumably you've not got that at zero or anything? It won't send if it's over it's sending limit and if the limit is zero that'd mean never.

There's an exception-catch in there that probably should have some kinda log-the-exception in it.

Not sure off hand how to actually send something to the nextcloud log. Will have to figure that out. No time today I'm afraid.

koehn commented 3 years ago

Max send per day is set at 50. Logging would be helpful. I have some other UI/UX related asks that I'll file separately. I think you've got a useful tool under way; looking forward to seeing an even better version! Thank you for all the work!

revpriest commented 3 years ago

Added some logging of exceptions during the sending of email.

Had a thought about what it could be too: Most email providers only allow "from" addresses that are whitelisted by you clicking an emailed-link. If the "from" address the list sends from isn't in the email-provider's whitelist they might reject the mail.

Did the subscription-confirmation emails get sent? They get sent with the same system I think, just not in a cron.

Anyway, hopefully there'll be something useful in the nextcloud error log if you update.

revpriest commented 3 years ago

Hi Koehn.

How's it going? Did it turn out to be the "sender" address? Did the error get logged? Can I close this as fixed?

chrisSCM commented 3 years ago

For me, emails aren't sent either. But I don't even have a table "oc_jobs" - my Nextcloud version is 21.0.4.

For the SMTP-config, I wonder, doesn't seem to be possible to add like SSL or STARTTLS, how is it going to work then?

Thank you.

revpriest commented 3 years ago

It uses PHPMailer, which does opportunistic TLS - if the server advertises that it can do TLS, it will use it automatically.

I think I agree they should probably be options though, to disable TLS if needed or specify SSL instead perhaps.

Will look into that when I get a chance.

Not sure what could make your system not have the "jobs" table. If all your tables have a different prefix then maybe it's "X_jobs" or whatever the prefix is instead?

I think that table should be there for your next-cloud to keep track of what cron-jobs need doing, possibly there's some step in installing which involves setting up a cron? Can't rememeber if I did anything like that in my initial setup.

The emails are sent by the cron system, so if the cron system isn't working for some reason the emails won't be sent.

jlgarnier commented 3 years ago

Hi all, Sorry to add one line to the charge 😉 ut got the same issue on my Docker Nextcloud instance:

Hope this can help to solve it! Thanks Rev for the great job!

revpriest commented 3 years ago

Humm. I think that probably means that the file /var/www/html/custom_apps/listman/vendor/autoload.php isn't there, so it can't find the phpmailer.

Someone told me not to pack the Vendor directory inside the release-package, and I did as they suggested assuming that there must therefore be some sort of build-step in the install which build the vendor directory.

Sounds like that might have been wrong, and that at least part of the Vendor directory should perhaps be included in the tar package.

Won't have any time to look into it for a few days, but it'll definitely want to have the vendor directory with phpmailer in it, so if that's missing then figuring out how to get that, ideally without all the other junk in Vendor, will be what needs to be done.

jlgarnier commented 3 years ago

Thanks for the feedback! Is there any way to get this particular module and add it to the existing installation?

revpriest commented 3 years ago

Probably, maybe?

Running "make composer" from the listman directory might do it? Depending on if the right tools are installed. Not sure what tools those would be.

Suspect that kinda thing is more difficult from the Docker install but never used it so dunno?

jlgarnier commented 3 years ago

Yes, the Docker env may raise tricky issues, especially with dependencies. Do you have any "full-featured" package which I could untar and overwrite the current installation?

revpriest commented 3 years ago

Everything necessary is in this repo I think. Cloning it out into the app folder then running "make composer" should download all the stuff in the composer.json to fill the vendor directory.

It's "make composer" that I assumed must be done in a build-step when installing from the package when I was told to remove the Vendor directory from the release tarball.

I wonder if Keohn is using the Docker install too? Maybe I should try running nextcloud through Docker one day.

Might have a couple of hours to look into it Wednesday or Thursday night.

jlgarnier commented 3 years ago

Will maybe give it a try this afternoon and let you know how it went... 👍

revpriest commented 3 years ago

Something else came up tonight, unlikely to be able to get to this until the weekend at the earliest, sorry. :(

revpriest commented 3 years ago

I managed to find some time.

Have re-packaged to new version 20.0.8 - that should include the vendor/phpmailer directory now.

Seems to sill be downloading v 20.0.0.7 when I try from my dev/test system though. Maybe coz I did a test-download of that and it's cached somewhere?

If you manage to get V20.0.8 to download hopefully it should contain the mailer library and so be able to send mail now?

Or maybe I'll get an email explaining why the repackage thing didn't work?

Will try and double-check tomorrow when hopefully whatever this cache is has timed out.

I wonder how you attach release-notes to a version update?....

jlgarnier commented 3 years ago

Hi, Back from a short vacation, I'll try to download this 20.0.8 version: still the normal git clone https://github.com/revpriest/listman.git listman command? I can't see any vendor/phpmailer directory on the Code tab... Since your message, did you solve your cache issue and have the opportunity to check this new release? Thanks again for your help!

revpriest commented 3 years ago

Uh, assumed you were installing just by clicking the install button in the addons page to install the package from the link uploaded to Nextcloud's servers.

It's entirely possible I haven't been pushing to github. Lets check....

Indeed, I haven't. Oops. Have pushed here now.

But if you're cloning from Github then the fact that the phpmailer library isn't included isn't the problem, since that's not in the git anyway.

If you've cloned from Github instead of using the "install" button then you'll need to build the contrib stuff with

make composer

from the newly checked out directory. That is what fetches the phpmailer library (as well as various other dev stuff)

I don't understand why "make composer" wouldn't create the phpmailer directory though.

david-aigner commented 3 years ago

Hi @revpriest

I love the idea - thank you for the app!

Unfortunately for me, it is also not sending. The app mentions queued 2 | Rate 1 per 5 minutes

The jobs entry says 4495 | OCA\Listman\Cron\ListmanSend | null | 0 | 1636638423 | 0 | 0

Unfortunately nothing in the protocol

Anything I can check myself?

Thank you!

revpriest commented 3 years ago

I suspect it's not working for anyone other than me, but I don't know why. Guess you mostly don't see the successes so my sampling is biased?

Anyway.

That first zero means it has never run the cron-job. Not sure why that would be.

If you have shell access then you could, from the nextcloud directory, run

sudo -u www-data php -f cron.php

(or similar, your web-user might not be called "www-data" if you're not debian, I dunno)

That might force the cron-jobs to run, or show you an error message why they can't.

david-aigner commented 3 years ago

Thanks for the direction - now I at least have a error message :-D

PHP Fatal error: require(): Failed opening required '/.../cloud/apps/listman/lib/Service/../../vendor/autoload.php' (include_path='/.../cloud/3rdparty/pear/archive_tar:/.../cloud/3rdparty/pear/console_getopt:/.../cloud/3rdparty/pear/pear-core-minimal/src:/.../cloud/3rdparty/pear/pear_exception:/.../cloud/apps') in /.../cloud/apps/listman/lib/Service/ListmanService.php on line 782

fabricerouillier commented 3 years ago

The autoload.php is missing.

david-aigner commented 3 years ago

checked the vendor folder - only one directory "phpmailer" in it. On your github page, I can not find the vendor folder at all - is it created locally during setup? how can I get the file?

Thank you!

revpriest commented 3 years ago

Will try and build a new package with the autoload in it too next week.

The build steps for generating the dependencies is described in the README.

make composer make dev-setup

fabricerouillier commented 3 years ago

Will try and build a new package with the autoload in it too next week.

The build steps for generating the dependencies is described in the README.

make composer make dev-setup

I have performed the operations in my nextcloud docker container (note you have to install nodejs version 14 at most)

I was able to use the subscribe form to register.

But I still have my old messages Queued (from previous attempts) and it still Queues new messages without sending them.

How to empty the queue ?

revpriest commented 3 years ago

Okay, thanks for your help everyone.

I've pushed a new version 20.1.0 which includes a bunch of stuff in the Vendor directory: autoload.php composer/ myclabs/

And now my new test/staging install works and sends messages.

I also added a new box in the "settings", which shows the latest warning generated by the sender. That'll usually be something like "didn't send to x@y.c because already passed sending limits" or "authentication error:" or if there's an exception the message from the exception.

Hopefully that'll help figure out any other issues like wrong SMPT details etc.

Only way to delete the queue is by editing the SQL database I'm afraid.

This will kill all unsent message-sending tasks:

delete from oc_listman_sendjob where state=0;

This will remove all the ones which had some kind of error or passed sending limits:

delete from oc_listman_sendjob where state<0;

This will remove the whole lot which may mean people get duplicate messages, it will allow re-sending the same message to the same address:

delete from oc_listman_sendjob;

You may also want to wipe the records of when we last sent a message to someone, so you can for a re-send even if the flood-protection thinks they were already emailed today and is refusing to do it again:

This will wipe all last-send dates:

update oc_listman_member set lastsend=null

This will do it just for a single email:

update oc_listman_member set lastsend=null where email="x@y.z"

Hopefully that'll make it work for everyone. Please let me know so I can close the issue.

david-aigner commented 3 years ago

Thank you, @revpriest !

I had a short struggle with my cron job to run it again, but now it is sending!

For me, the issue is solved!

Thank you!

revpriest commented 3 years ago

Hurray. Hopefully it's good for everyone then.

Shout up if not, I think these things can be re-opened.