search

Mailtrain-org / mailtrain

Self hosted newsletter app
GNU General Public License v3.0
5.51k stars 693 forks source link

Mailtrain v2 Beta Available - help needed with testing #505

Closed bures closed 3 years ago

bures commented 5 years ago

Dear all,

it's my pleasure to announce general availability of Mailtrain v2 beta. You can find it in the development branch. The repo contains installation scripts for CentOS 7 and Ubuntu 18.04 LTS. There is also a Docker image with a docker-compose script provided. All info how to install it is in README.md.

This version should have all features of Mailtrain v1 with a bunch of new things added. Notably those are:

Note however that there are several features which I didn't have a chance and possibility to test. In particular:

I would like to ask you for help with testing these features above. If it turns out that there is no one to test the feature, I will assume that that feature is not need and will exclude it in stable release of Mailtrain v2.

If you file an issue, please mark it with v2.0 milestone.

Thanks. T.

hoducha commented 5 years ago

I got an error when installing Mailtrain with docker-compose. Here is the full log: https://pastebin.com/AYMkBNCu

sweetppro commented 5 years ago

Is it possible to install manually on a single domain? I am running a Centos 7 server with Plesk. And I want to install this in a subdomain(as I have already with v1)

I see that we need 3 endpoints now though. Can I host all the endpoints in subdirectories, rather than seperate subdomains?

If that’s possible, then I’m totally in for some small delivery tests with Mailgun, as I have this all workin well for mailtrain v1.

bures commented 5 years ago

@sweetppro - How strong is this requirement (of hosting on single endpoint)?

The reason there are three endpoints is the security. If you host it on a single endpoint, then there is no protection against XSS through templates or custom subscription forms. For instance one user may insert malicious JS code to template or subscription form. The code would be executed with admin permissions once admin opens the template to view it. To prevent this, the sandbox endpoint has to be a different domain (i.e. different sub-domain or different port). Having it in a subdirectory does not help, because that does not count as a separate domain.

I could generally extend Mailtrain to allow hosting from a single domain. It can't do it now, but (fingers crossed) I believe the architecture should allow it with some tweaks. It's just that doing so would completely defy the security. Mailtrain on one endpoint would generally count as if all users were admins.

So just let me know pls, if that is a reasonable use-case. I don't have experience with Plesk, so it's difficult for me to tell if there is some way of exposing three different ports on one sub-domain or three subdomains for one nodejs app. However if that would be an option, I would prefer it to hosting it on a single endpoint.

Thanks.

bures commented 5 years ago

@hoducha - thanks for giving it a try. See the last line of your log: "... no space left on device"

Just give it more space. I don't recall correctly how much space I needed, but if I'm not mistaken, it was aroud 1.5GB.

sommarnatt commented 5 years ago

At the end it says no space left on device. Did you make sure you have enough free space? // david-------- Ursprungligt meddelande --------Ämne: Re: [Mailtrain-org/mailtrain] Mailtrain v2 Beta Available - help needed with testing (#505)Från: Ho Duc Ha Till: Mailtrain-org/mailtrain Kopia: Subscribed I got an error when installing Mailtrain with docker-compose. Here is the full log: https://pastebin.com/AYMkBNCu

—You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or mute the thread.

hoducha commented 5 years ago

Oops, that's my mistake.. how could I miss that line? I've deployed Mailtrain successfully after cleaning up the Docker images. I'm exploring the Mailtrain v2 now... Thank @bures and @sommarnatt.

sweetppro commented 5 years ago

@bures - Opening up seperate ports shouldn’t be an issue on a single subdomain.

I am already running a rails app on another subdomain, and nodeBB on yet another subdomain. The parent domain hosts my static pages built with Hugo, linking to all the various subdomains.

I also already have letsencrypt setup on the parent domain as a wildcard.

I could install the seperate components on they’re own subdomains, but installing manually would perhaps be the easiest for my setup as I’m no expert on Plesk. I’m not sure how it would deal with the extra subdomains not being generated via it’s interface? I’d probably need to configure Nginx/Apache/dns manually which I could probably figure out...

bures commented 5 years ago

@sweetppro - Then let's try it as different ports on the same subdomain.

I would suggest to have: trusted on 8443, sandbox on 8444 and public on 443.

I place public on 443 because some firewalls might disallow non-standard ports and with subscribers you cannot regulate from where they will access Mailtrain.

Trusted and sandbox could be on non-standard ports because there will be only you (or limited set of people) accessing them.

Do you need any help with the setup on configuring Mailtrain to use one domain with 3 different ports? If so, please open a separate issue, we can deal with that there.

Thanks.

sweetppro commented 5 years ago

@bures - Sounds good, I’ll give it a try tonight and see how it goes

yashsaxena commented 5 years ago

@bures this is really great. We will definitely put it to test in coming weeks. thanks for releasing this version! its really really great to see this out.

bures commented 5 years ago

A small announcement - I just migrated Mailtrain v2 beta to Bootstrap 4 with CoreUI theme and the newest React. This should be hopefully the last bigger change before v2 release.

Please note that if you are upgrading an already installed Mailtrain v2 beta, you will have to reinstall modules and rebuild the client. To do so, execute the following from Mailtrain's root directory:

for idx in client shared server zone-mta; do
    (cd $idx && npm install)
done

(cd client && npm run build)
saschafoerster commented 5 years ago

First impression: it looks really good, clean and I am looking forward to the further development of mailtrain! I will write down some ideas and impression while testing everything in docker.

The overall impression is really good. First I was a bit afraid, that the new 3-domain-system would be a problem for my and upgrading my existing installation, but now I don't see it as a big problem anymore. Of course, a good explanation for upgrading the system from the latest release would be of good help. And now: I hoping this comments are helpful to leave beta behind and give Mailtrain a good fresh version.

bures commented 5 years ago

@saschafoerster Thank for the detailed suggestions. Very appreciated. Replies below

  • I would like one option to change the main-color and maybe add my own logo for the overall look.

Makes sense. I assume it is enough when this is configured in the server config and the logo is just placed in some directory, right?

  • Account / General settings: When I tried to change my login from admin:test to something else, there was no field to see the actual username, only name and email. An option to change that username would be nice additionally.

I'll check it. The option to change the username should be feasible. I don't think we need immutable usernames anywhere in the code.

  • Dashboard: I would like to see the last sent campaigns and some results: no. of mails sent, opened, clicked, and a link to see the preview of the campaign. Maybe a list of prepared but not sent campaigns as well.

Makes sense. Thanks for ideas. Could you open a separate issue on the dashboard and paste these ideas there. Let's see if anyone else has also anything to say.

  • RSS-campaigns: I tried to delete an RSS-campaign, but instead of deleting I get: "content is not defined", the campaign is still there.

I couldn't reproduce this. Could you open a separate issue and describe exactly how to reproduce the issue?

  • RSS-campaigns: I would really love the possibility to sent a weekly or daily collection of a feed in one mail, where all new articles in the feed are used to create the mailing.

Makes sense. How would you imagine it? The way I imagine it is that the RSS campaign allows you to check that it should be a "Digest RSS campaign" and then it allows you to specify the periodicity (every n-th hour, n-th day, every n-th month and starting day/time). Then there will be an merge tag the corresponds to the list of RSS items. This will require that we introduce HBS+MJML templates, so that one could create the content of the template dynamically. Anyway, having HBS+MJML templates is already in my TODO list. It's also needed in single-item RSS campaigns. If you have any more ideas, best if we move this discussion to a separate thread.

  • Namespaces/Create: I would be interested to get an explanation, what a namespace is. Maybe in general it would be good to have some place, where users get more informations about the features they see, like a questionmark in the corner?

Namespaces are about organizing entities (i.e. lists, campagins, etc.) and giving access to all entities (of a particular type) in the namespace and below. This allows you to for instance create a namespace per customer for a customer and configure the permissions in such a way that the customer cannot access entities of other customers. Or it allows you to organize entities and permissions within a larger organization where you have different groups of people/departments.

  • Global settings: Instead of tracking with Google Analytics, an option to use Matomo (Piwik) would be great, especially when you are in love with self-hosting like I am.

Nice. I didn't know about this service. I would be definitely eager to have it there. Only, it doesn't have a very high priority on my side. I could possibly implement it as the time goes, or I would be happy to accept any pull request on this.

  • I couldn't test the API, also MJML seems not to be integrated yet, but at least I get a message on the new subdomain, when I try to create a template.

Could you be more specific what you mean by "MJML seems not to be integrated yet". If there is some bug connected with MJML in templates, please create a separate issue. As far as I believe, all MJML-related stuff should be implemented in templates. The only MJML related part which is not implemented is MJML Mosaico Templates (i.e. the ability to define Mosaico Template - i.e. a template for template - in MJML). Here I have to check with @witzig , who did some work on generating Mosaico Templates with MJML.

The overall impression is really good. First I was a bit afraid, that the new 3-domain-system would be a problem for my and upgrading my existing installation, but now I don't see it as a big problem anymore. Of course, a good explanation for upgrading the system from the latest release would be of good help. And now: I hoping this comments are helpful to leave beta behind and give Mailtrain a good fresh version.

Thanks a lot. With guides and explanation how to setup things and what features it contains and how to use them, I would really appreciate some help from the community. I'm not sure I have the capacity to code it and document it at the same time. I'll be of course happy to explain certain features which might not be clear. I also like the idea with the question mark icon. I would be happy to integrate it if I receive the texts to provide there.

Thanks.

killua-eu commented 5 years ago

Had a brief look on the setup directory as I'd like to update https://github.com/vaizard/mage-mailtrain/ an ansible playbook automating mailtrain deployment to support v2 as well. I notice that the development branch

thanks for the clarification.

bures commented 5 years ago

@killua-eu - I move the ansible role discussion to a separate issue #518

julianwachholz commented 5 years ago

I would like to test this (with SparkPost) but I have some questions:

bures commented 5 years ago

@julianwachholz 1) The database migration should work automatically when you first run Mailtrain v2, but please backup your database before. The migration is still a bit shaky as I have only limited samples to test on. Generally, I would recommend to do pretty much a fresh install of Mailtrain v2 and run it on the old database.

2) I'm afraid this a new feature that needs to be implemented. Generally, it means to extend the "Subscription" options in the "List edit". This should not be difficult, but it requires about half a day of work. Feel free to do it, I'll be happy to accept a PR. If doing so, I would recommend to change public_subscribe field to subscription_mode (similar to unsubscription_mode) that would offer more options. Alternatively, I'll be happy to implement it myself later. In that case, please create a separate ticket for it.

3) The subjects for subscription/unsubscription emails are hardcoded in the code, but can be localized. How would you like to change them? Again, best to open a separate issue for it.

julianwachholz commented 5 years ago

Hi @bures, thank's for getting back to me.

I'd love to help out with No. 2, but it'll take some time for me to read into the code. :)

Re: Translations: I don't want the UI to be translated but I have two newsletter lists, one for English subscribers and one for e.g. German ones, so they'd want German E-Mail notifications and forms (which I could all successfully customize with the Custom Forms feature). So I think best solution for this would be to add an extra field to all forms like this?

bures commented 5 years ago

@julianwachholz I understand. Since the content of the email is anyway coming from custom form, it would make only sense to add something like "Title" to each custom form. If it is a web form, the title is the title of the window, if it is an email form, the title is the subject, right?

priethor commented 5 years ago

Note however that there are several features which I didn't have a chance and possibility to test. In particular:

  • LDAP

I've been testing the LDAP and can confirm that the login itself works fine with passport-ldapauth. However, I had to create the users internally in the user management page before they could login. I suspect this is not the desired use case and users should be created on first login, as Mailtrain accepts the ldap properties newUserRole and newNamespaceId, right?

bures commented 5 years ago

@priethor Yes indeed, they should be created automatically. Will you look into this bug or shall I. Honestly, I won't be able to get to it earlier than in about 1-2 weeks. Thanks.

priethor commented 5 years ago

I don't think I will be able to check it before then, either. I'm going to open an issue with this bug and what I'm guessing it's the same bug in a different context, and if I happen to look into it I'll keep you posted there.

bures commented 5 years ago

@priethor - Great. Thanks. So let's see who gets there first. :-)

pokernet commented 5 years ago

Error install local on centos7 frash . The install script just stop .

rbicelli commented 5 years ago

Hello! Is it safe to upgrade from v1 or it is better to start from a fresh install? Thanks

bures commented 5 years ago

@rbicelli - The upgrade should work, but has not been very thoroughly tested. My suggestion is to try the upgrade if you have data you want to preserve. Otherwise do a fresh install.

If you do the upgrade, MAKE SURE you backup the DB first. If you encounter any issue, please post it or fix it yourself and submit a PR.

rbicelli commented 5 years ago

@bures thanks ... better than backup db, will do a VM snapshot :-)

bures commented 5 years ago

@rbicelli Perfect. Note that when you do the upgrade from v1, you may need to create a new send configuration manually if you want to use Built-in ZoneMTA.

priethor commented 5 years ago

Hi @rbicelli,

I did myself a partial upgrade, only the database without the rest of the code, but that's because I was using a local SMTP and wanted to move to ZoneMTA, so a fresh install was better in my case, but otherwise it should be ok. The DB migration script can take a while bur works fine, just do the backup/snapshot before!

priethor commented 5 years ago

Hi @rbicelli,

I did myself a partial upgrade, only the database without the rest of the code, but that's because I was using a local SMTP and wanted to move to ZoneMTA, so a fresh install was better in my case, but otherwise it should be ok. The DB migration script can take a while bur works fine, just do the backup/snapshot before!

rbicelli commented 5 years ago

I'm trying the upgrade, but local install script doesn't fit for me. I'm using mailtrain behind a nginx reverse proxy, which is located physically on another machine. Is there a way to tell the local install script to not create database and apache config but update mailtrain only?

dabeanman808 commented 5 years ago

Messing around with V2 (amazing by the way). Was messing around with sending a email with a api call and I didn't specify the senderid which defaulted to the system one. Looking at the syslog it constantly is trying to push these couple of emails (for hours now) and im not sure how to clear them out of the queue. Ive went through all the bug reports and tried to research before I made this post. Im assuming there should be some timeout that cleans the queue (wherever that is located) after so many failed attempts. Any ideas I could try?

Here is the log: May 05 05:43:18 m-train node[1642]: ERR! Sender/default/1812[1] 16a85ec869f000179f.001 CONNFAIL domain=gmail.com local=m-train[0.0.0.0] remote=74.125.20.27:25 error=Network error when connecting to MX server gmail-smtp-in.l.google.com[74.125.20.27] for gmail.com: Connection timed out

Here was the API call I did: curl -XPOST 'https://mailtrain.myactualdomianwashere.com/api/templates/1/send?access_token=XZY’ --data ‘EMAIL=someemail@yahoo.com&SUBJECT=Test&VARIABLES[FOO]=bar&VARIABLES[TEST]=example’

bures commented 5 years ago

@dabeanman808 - If I'm not mistaken, this log message comes already from embedded Zone MTA. Check the Zone MTAs URL endpoints (https://github.com/zone-eu/zone-mta) to manage the queue. Just note that the API port of Zone MTA is 12080.

If it helped, please post a solution here. Thanks.

dabeanman808 commented 5 years ago

Great!! Thank You! That was enough information for me to figure it out how to at least stop the never-ending retries. I wish I could give you an actual solution that you could add to your code but I am essentially just a jr developer with basic experience. I will post what I found (mostly basic commands you probably already know) and keep digging, if I find something more helpful to the overall picture of this project I will edit my post and include.

jameshobden commented 5 years ago

I am currently using AWS SES in production on v1 and would be happy to test SES in V2. Please let me know if there are any specific scenarios that need testing. PS love this project!

bures commented 5 years ago

@jameshobden - thanks for the offer. We would need two things: 1) To test whether V2 can deliver through AWS. 2) I would need an example response from AWS. Mailtrain v2 prints it out on console when sending out a campaign. It's a string which says response: 250 .................. messageId: ..................

bures commented 5 years ago

@jameshobden - I did some testing and fixes for AWS. It should work (no need for sending me the example response of AWS). At this point, it's in branch development-tb. I'll move it to development, once I'm done with testing of development-tb. Thanks.

jameshobden commented 5 years ago

OK great thanks!

amitprince commented 5 years ago

Hello! I am Currently using Mailtrain version 1 in production , As i need to Create Multiple users so i need to install the Version 2 ,but i am unable to install it on ubuntu 18.04 ,is there any alternate way is there to install? Need help . Thanks

bures commented 5 years ago

@amitprince you can try also centos 7 or docker. I'm afraid, we don't have any other officially supported way of installing it at the moment.

techterbium commented 5 years ago

Hi @bures , I would like to help you with API testing. Let me know if you have any specific instructions on setting up the environment, as this is my first-time involvement in a project.

I am planning to use POSTMAN to invoke these APIs and see the response.

Thank you for amazing product!

amitprince commented 5 years ago

Untitled

and these are the logs showing 500 error

mailtrain_1 | info HTTP GET / 500 0.608 ms - - mailtrain_1 | info HTTP GET /client/mailtrain.css 500 0.719 ms - - mailtrain_1 | info HTTP GET /static-npm/jquery.min.js 500 0.443 ms - - mailtrain_1 | info HTTP GET /static-npm/popper.min.js 500 0.510 ms - - mailtrain_1 | info HTTP GET /static-npm/bootstrap.min.js 500 0.607 ms - - mailtrain_1 | info HTTP GET /static-npm/coreui.min.js 500 0.509 ms - - mailtrain_1 | info HTTP GET /static/favicon.ico 500 0.610 ms - - mailtrain_1 | info HTTP GET /static/favicon.ico 500 0.628 ms - -

Hi @bures , i tried to install the Mailtrain V2 through Docker but its showing the above image while i type my ip address.... Need Help!

bures commented 5 years ago

@amitprince - can you paste the output of the command line where you run the docker-compose?

bures commented 5 years ago

@amitprince - and please make this a separate issue.

ZephOne commented 5 years ago

I just signal that we use our own mailserver with postfix and we do not have any problem concerning delivery

bures commented 5 years ago

@ZephOne - excellent. Thanks a lot for letting me know. Have you also tried to setup handling bounces? It's the section postfixBounce in server's config. Thanks.

ZephOne commented 5 years ago

@bures For the moment we have enabled: false. The Mailtrain integration period is not finished for the moment. I just know that we didn't have any problem sending mail with Mailtrain through our postfix mailserver to our test users. I have planned to finish Mailtrain integration next week, then I will also enable postfixBounce feature. I'll let you know when it's done.

martialblog commented 4 years ago

@bures Hey, anything I can test to help with v2? Got mailtrain in docker-compose up and running.

suyash2953 commented 4 years ago

@bures in V2 beta, have you exposed APIs for sending mails via campaign and managing other tasks as well. Coz in v1, I did not find any for the same. Please assist.

suyash2953 commented 4 years ago

Hi @bures , I would like to help you with API testing. Let me know if you have any specific instructions on setting up the environment, as this is my first-time involvement in a project.

I am planning to use POSTMAN to invoke these APIs and see the response.

Thank you for amazing product!

@techterbium did you do api testing for executing campaigns. I am planning to try them out myself, but if you have already tried it out, it would save me some time.