eduardoboucas / staticman

💪 User-generated content for Git-powered websites
https://staticman.net
MIT License
2.42k stars 540 forks source link

Email notification upon replies #42

Open TWiStErRob opened 7 years ago

TWiStErRob commented 7 years ago

Usually commenting solutions allow for users to be notified via email about replies to their comments, or in case of a flat commenting stream when anyone posts a new comment. Is it possible to do this with staticman?

eduardoboucas commented 7 years ago

This is currently in development and will be released soon!

illus0r commented 7 years ago

@eduardoboucas looking forward to it! I've just discovered, I've received a first comment on my blog. This comment is useful, but I'm not able to say “Thank you” to it's author.

eduardoboucas commented 7 years ago

The feature is pretty much ready. I could use with some help with testing. Would any of you be interested in being the guinea pig? 😄

illus0r commented 7 years ago

@eduardoboucas ready!

eduardoboucas commented 7 years ago

Okay, here's how to get notifications working.

Emails are sent using Mailgun. There is an account associated with the public instance of Staticman, but it comes with a limit of 10,000 emails a month. This limit shouldn't be an issue in the immediate future, but users are encouraged to use their own Mailgun accounts instead. They are free (for the first 10k emails), but you do need a custom domain to use them with. I can possibly provide *.staticman.net subdomains.

To enable notifications, you need a notifications block in your site config and you must be using the config file format introduced in v2 (i.e. having a staticman.yml file, instead of using Jekyll's _config.yml). See the sample config file for more info.

If you're using your own Mailgun account, you'll need to add a Mailgun API key and domain to the config, which need to be encrypted. To encrypt those, you can use the following endpoint: https://api.staticman.net/v2/encrypt/{TEXT TO BE ENCRYPTED}.

Next, your entries will need to specify that a user wishes to subscribe to a specific entry. This is an example:

<form method="POST" action="https://api.staticman.net/v2/entry/joebloggs/my-site/master/comments">
  <input type="hidden" name="options[origin]" value="http://mysite.com/post1.html">
  <input type="hidden" name="options[parent]" value="867bd1e0-921c-11e6-930f-79eeedf443ea">

  <input type="text" name="fields[name]" placeholder="Title">
  <input type="text" name="fields[email]" placeholder="Email">
  <textarea name="fields[body]"></textarea>
  <input type="checkbox" name="options[subscribe]" value="email">

  <button type="submit">Go!</button>
</form>

In the from above, you can see the following:

Finally, and to make sure we don't send emails asking users to click on links that have been spoofed, you need to add an allowedOrigins field (see docs) to your site config, where you'll specify which domains are valid origins. If you add only mysite.com, then an entry where options[origin] is http://myothersite.com/post1.html will be rejected.

I'm sure there are still a few rough edges, so if any of you can try implementing this and share your feedback that'd be very useful.

Thanks!

dancwilliams commented 7 years ago

Works great! Thanks for your work on this!

zongren commented 7 years ago

Is this working on public staticman instance? I tried it a couple of times and failed. Here is my staticman.yml:

allowedFields: ["name", "email", "url", "message", "origin", "parent", "subscribe"]
requiredFields: ["name", "message"]
format: "json"
generatedFields:
  date:
    type: "date"
    options:
      format: "iso8601"
moderation: true
branch: "master"
path: _data/{options.slug}
filename: comment-{@timestamp}
transforms:
  email: md5
name: "宗仁的博客"
allowedOrigins: ["localhost", "zongren.me"]
notifications: 
  enabled: true
  apiKey: "qPB1uWby7FS6T0wgLqUfjcQkkVtLkcergXOqiEoZyTo5yqMmGU/cuzOD825KOZkvbE7m0mOYo2LKPj82v+BcQDxxcIULev8lwpQ1KZJwjv6Ei3f1HbFyIq5N2Ehmya3PyPGga3IaedFVTPFrue67DQ2W5+tu8xJX1S2PahUEgAA="
  domain: "FkFQQFNFX96xVIOYvgtB8IxeBP60lZa/sUhHv0Y3KWV3EdRjMLED0zZr6nGGC5opytzczKvQtR2Y6YJvKQ2ltOBV1aFuAvsN/HPnOZ4e5JMBI+BjGWWlaqUsKp/mNGq/q9oWDk3FT8tdfw7UBqa8lC99eYa+QMXj/k+gpPx5ki0="

and comment form:

<form class="comment-form" method="POST" action="https://api.staticman.net/v2/entry/<%- theme.staticman %>">
    <input type="hidden" name="options[origin]" value="<%- config.url %><%- config.root %><%- item.path %>">
    <input type="hidden" name="options[parent]" value="<%- item.slug %>">
    <input type="hidden" name="options[redirect]" value="<%- config.url %><%- config.root %><%- item.path %>">
    <input type="hidden" name="options[slug]" value="<%- item.slug %>">
    <span class="comment-form__input-wrapper">
      <input autocomplete="off" spellcheck="false" class="comment-form__input" name="fields[name]" type="text" placeholder="请输入姓名"/>
    </span>
    <span class="comment-form__input-wrapper">
      <input autocomplete="off" spellcheck="false" class="comment-form__input" name="fields[email]" type="email" placeholder="请输入邮箱(可选)"/>
    </span>
    <span class="comment-form__input-wrapper">
      <input autocomplete="off" spellcheck="false" class="comment-form__input" name="fields[url]" type="url" placeholder="请输入你的网站(可选)"/>
    </span>
    <span class="comment-form__input-wrapper">
    <textarea class="comment-form__input comment-form__input--textarea" name="fields[message]" placeholder="请输入评论内容(支持Markdown)"></textarea>
    </span>
    <input type="checkbox" name="options[subscribe]" value="email" id="subscribe" /><label for="subscribe">订阅此文章的评论</label>
    <button class="comment-form__input comment-form__input--button">提交评论</button>
  </form>

And there are no logs on my mailgun dashboard.

eduardoboucas commented 7 years ago

@zongren link to the repo, please?

zongren commented 7 years ago

@eduardoboucas the repository is at gitlab

zongren commented 7 years ago
qq20161130-1 2x

And I used encrypted "zongren.me" as notifications.domain.

wes-brooks commented 7 years ago

Hi Eduardo! Any chance you could check the logs of the staticman service to help me understand why my attempts to get reply notifications aren't working? I think I've configured staticman properly and have a working mailgun account, but I must be doing something wrong.

The repo is wrbrooks.github.io. Thanks in advance! -W

eduardoboucas commented 7 years ago

@wrbrooks I don't see any errors in the logs. Can you link to the page you're using to submit entries?

justinrummel commented 7 years ago

Hello @eduardoboucas I'm having issues as well w/ Staticman triggering emails with Mailgun. I'm able to validate with Mailgun's example curl statement but cannot trigger anything from my primary test page: https://www.justinrummel.com/macworld-2010-pictures/

(API Key removed)

justinrummel@Rummel-MBPr ~> curl -s --user 'api:key-12341234123412341234123412341234' \
                                https://api.mailgun.net/v3/mg.justinrummel.com/messages \
                                -F from='Excited User <mailgun@mg.justinrummel.com>' \
                                -F to=j@rummel.co \
                                -F subject='Hello' \
                                -F text='Testing some Mailgun awesomness!'
{
  "id": "<20170204203732.129129.51249.1E9B7445@mg.justinrummel.com>",
  "message": "Queued. Thank you."
justinrummel@Rummel-MBPr ~>
eduardoboucas commented 7 years ago

@justinrummel Staticman uses the Mailing Lists functionality from Mailgun, where each thread/post is created as a mailing list. If you go to your Mailgun account and navigate to Mailing Lists, do you see any lists created? If so, do you see the email addresses corresponding to the users that have subscribed to that thread?

justinrummel commented 7 years ago

I see one mailing list that I think was created from my API tests, but not from the site.

chuckmasterson commented 7 years ago

I've been planning how to integrate email notifications onto my blog. I have my comment fields defined somewhat idiosyncratically, so the author's email is authoremail instead of just email. I suppose that probably wouldn't work, since I haven't found where to set the name of the field that Staticman looks for... do you know of a way to make it work, or will I just need to sed all my authoremails into emails?

Also, I saw in your reply to #72 (which brilliantly solved a problem I was having, thanks) that the notifications feature isn't considered fully stable yet. Do you have a sense of how likely it is that an existing staticman notification configuration would continue working as the project continues toward stability?

eduardoboucas commented 7 years ago

@chuckmasterson You can specify the name of the field that contains the email address. It's the value of options[subscribe]. In your case, that would be:

<input type="hidden" name="options[subscribe]" value="authoremail">

Regarding the stability of the email notification system, I have to admit I've been struggling to find enough time to dedicate to it. In any case, I think the existing configuration format is flexible enough to accommodate any changes in the near future, so I don't foresee any breaking changes.

chuckmasterson commented 7 years ago

Completely understand about having time. If I knew the first thing about Node.js I'd try contributing, but alas.

I tried setting up with the advice you gave and got as far as getting some mailing lists created. But nothing triggered sending to these lists. I'll look at it again soon... if you find the time to read this and reply, a couple questions that might help me out are:

chuckmasterson commented 7 years ago

After writing that, I took a crack at reading the code anyhow, and I think I may have understood enough to make a suggestion that would get us partway to a solution, though not all the way. (I would write this as a pull request but my understanding of Node.js is so weak that I would almost certainly break more than I'd fix.)

Suggestion:

Use a different variable besides options.parent to control subscriptions, perhaps a new one named options.subscribeTo.

Reasoning:

Though I haven't found anywhere you gave explicit directions on how to use options.parent, @mmistakes is already using it in threaded commenting, and a few other people have copied his strategy (I'm one). This strategy relies on options.parent being written into the YAML comment file as _parent. In this strategy _parent (options.parent) is the _id of a child comment's parent.

From what I understand of the Staticman code, Staticman expects options.parent to be some identifier of a post that the user is currently subscribing to. This conflicts with the other usage. If a user is writing a top-level comment on a post, then that comment's _parent will be null, and hence even if the user clicks the "subscribe me" checkbox, no mailing list is created because this condition doesn't get satisfied:

if (subscriptions && options.parent && options.subscribe && this.fields[options.subscribe]) {
  subscriptions.set(options.parent, this.fields[options.subscribe]).catch(err => {
    console.log(err.stack || err)
  })
}

If I follow the logic aright, the only subscriptions that would work with Staticman as it stands are where a user makes a child comment and clicks "subscribe me" - in which case they would get subscribed to all replies to their comment's parent. No one can subscribe to replies to their own comment, only to replies to some top-level comment that already exists.

Although my and mmistakes' use of options.parent here is apparently nonstandard, I think there's reason to favor it over what's currently in the code:

I'm not sure what format would make sense for the hypothetical options.subscribeTo as I still don't fully understand how Staticman currently interprets options.parent.

I'm still confused on one point: you mentioned in #76 that options fields aren't written to the comment file, but options.parent sure is getting written. I'm happy with that, but it is also a bit unusual; maybe it would make sense for it to go into the documentation eventually that "options fields generally don't get written to the entry, with the exception of options.parent, which is prefixed with an underscore (_parent) to denote the difference."

I'm happy to be a guinea pig on this feature as I'd definitely like to have it, but I'm also going to be on extended travels with little to no coding starting April 1, so I hope we can all get it figured out by then.

And thanks for Staticman! I hope this helped and wasn't too full of misunderstandings of how the code works.

mmistakes commented 7 years ago

@chuckmasterson I just merged in your comment. My understanding is you should get a notification email once a new comment is merged in. I'm going to do that now and make it off the main thread (not a child of your comment to see if your hypothesis is true).

My site takes about 20 minutes to build as it crunches through a bazillion images, so give it that long to see if anything shows up.

eduardoboucas commented 7 years ago

Thanks for the suggestions, @chuckmasterson. Let me try to answer some of the questions.

How does Staticman create the "Alias Address" in Mailgun for each mailing list? Mine looks like a random series of hex digits, but it seems like it should correspond to the _id of some post, or at least something that I see in the YAML of one of the comments.

Each mailing list has a corresponding email address. This email address is a MD5 hash of the GitHub username, repository and entry id concatenated together. This happens here.

How does Staticman determine what (if any) Alias Address to send notification emails to?

Whenever we process an entry, we take its id, generate the hash described above and check if there's any mailing list with a matching address. If there is, we fire notifications to all the addresses in it. Otherwise, there's nothing to do.

From what I understand of the Staticman code, Staticman expects options.parent to be some identifier of a post that the user is currently subscribing to. This conflicts with the other usage.

My thinking was that options[parent] describes the "thread" that you want to subscribe to. If you're writing a top-level comment as you mentioned, this parent would be an identifier of the post you're commenting on (it could be a title, slug, or anything else that uniquely identifies the entry). If you're commenting on an existing comment, then the parent would be the id of the comment.

Theoretically, it could go on how many levels necessary, but as @mmistakes will be able to tell you, it's probably not a good idea to nest further than 2 levels because otherwise the site generation process becomes too complex.

So the general idea is: options[parent] can be whatever — a post title, slug or a Staticman entry id. All it does is create a mailing list with subscribers for that particular parent, which we use whenever we want to process that entry.

Does this help?

eduardoboucas commented 7 years ago

And sorry if this has caused confusion and made you have to guess what Staticman does. My intention was to document this whenever I had the chance to make it fully stable, but I understand that people want to use this as soon as it's available, so from now on I'll make sure to document things as they get deployed to the live API.

eduardoboucas commented 7 years ago

Also:

I'm still confused on one point: you mentioned in #76 that options fields aren't written to the comment file, but options.parent sure is getting written. I'm happy with that, but it is also a bit unusual; maybe it would make sense for it to go into the documentation eventually that "options fields generally don't get written to the entry, with the exception of options.parent, which is prefixed with an underscore (_parent) to denote the difference."

This is very true. This should be in the documentation.

chuckmasterson commented 7 years ago

Ahhh, it's becoming clearer. @eduardoboucas, thanks for all the answers.

@mmistakes, your comment has been processed but I didn't get a notification. I think I was right, and if I'm right about being right, we're both stuck until either Eduardo makes the change I suggested or we change our commenting mechanism so it uses options[parent] how Eduardo intended.

In your comment you said that "my understanding is options[parent] are options[origin] are two different things, the later used to track which thread you’re currently subscribed to." I don't think options[origin] is used for this at all, and in fact I haven't been paying any attention to that field. It appears to me that it's only used for (1) preventing spoofing attacks and (2) displaying a link in the processed email.

Sounds like Eduardo is saying that options[parent] is indeed what's used to keep track of the thread (Mailgun mailing list). In SubscriptionsManager.js this happens:

const compoundId = md5(`${this.parameters.username}-${this.parameters.repository}-${entryId}`)

...which would boil down to, for instance, md5("chuckmasterson-chuckmasterson.github.io-[somePostIdentifier]"). That "somePostIdentifier" looks like it's supplied by options[parent].

If not for the condition check (does options.parent exist?) that I quoted earlier, my top-level comment on MadeMistakes would've resulted in a mailing list created with a name of whatever results from md5'ing "mmistakes-made-mistakes-jekyll-", since the entryId (options[parent], blank for a top-level comment) would have been null. Since there is that check, I'm guessing if you looked in your Mailgun mailing lists you'd find that no list with my address in it has been created (it's a mail.chuckmasterson.com address).

So... if I'm right, which of us should change?

mmistakes commented 7 years ago

That makes sense @chuckmasterson. Because I'm using options[parent] for nested comments it will only fire reply emails on child comments. The main thread sends null for that variable so no email notifications.

I'll probably change the language on my site so it's clear that you're only subscribing to "replies made on your comment" and not to the entire thread. Which to be honest is probably fine. I'm less concerned about email notifications than I am with stopping the barrage of comment spam and resulting Staticman PR's I delete daily :wink:

Also I'm not using my own MailGun account. That's probably why I'm one of the few who have email notifications working. I'm just piggy backing on the public instance of Staticman since I never had much luck configuring it.

zburgermeiszter commented 7 years ago

@mmistakes you might want to have a look at the brand new reCAPTCHA feature to stop the spam.

eduardoboucas commented 7 years ago

If I understand correctly, the only thing missing in @mmistakes's implementation is to send a post identifier on options[parent] for the main thread.

@mmistakes I think I already asked you this, but have you tried a honey pot field? I found it to be quite effective in stoping spam on my own site. If that hasn't worked, feel free to open a separate issue so we can find a solution. I'm also more concerned with that than with email notifications.

mmistakes commented 7 years ago

@zburgermeiszter Gave it a go this morning but trying to POST to dev.staticman.net was giving me errors. I had the same problem when I switched to the v2 API and had to jump through some hoops to remove/add Staticman as a collaborator.

zburgermeiszter commented 7 years ago

@mmistakes Can you please report this the issue #20, with your staticman.yml please, to see what could be wrong?

chuckmasterson commented 7 years ago

@mmistakes People would be subscribing to "other replies made to the comment you're replying to", if that's alright with you.

@eduardoboucas That would work for Michael except that in his comments Liquid (which I got familiar with while I was adapting it), top-level comments are identified as such by how they're missing a _parent field. I suppose we could check whether _parent equals the post slug instead of whether it's null. That seems a little fragile though.

mmistakes commented 7 years ago

@eduardoboucas - That would probably work. I would just need to modify the logic behind the loops. Which may or may not give me a big headache and I'll just punt on. Hahha.

And Yes I have a honey pot in there. I'll drop some notes in a separate issue.

eduardoboucas commented 7 years ago

@chuckmasterson I see. Not sure what the best solution is in that case. My site is using the post slug as parent, but I'm not using threads — all comments are top-level entries.

Any suggestions?

chuckmasterson commented 7 years ago

In my migration from Blogger I ended up defining a variable at the beginning of the comments block called commentslug, which looks something like the markdown filename (for example, "2015-10-some-title" - for legacy reasons mine is without the day). That's constructed of variables that shouldn't change even if the permalink structure does:

{% capture commentslug %}{{ page.date | date: '%Y-%m-' | append: page.title | slugify }}{% endcapture %}

You might be able to do something more simple and more resistant to breaking in case of changing titles, by using {{ page.path }}, which returns the full GitHub URL of the post's original markdown file - if you can figure out a way to reliably get rid of everything up to and including the last /, since the rest will change depending on branch and whether you're on a localhost. Whoops, never mind, just checked page.path and it actually returns a path relative to the root of your repo, like _posts/2015-10-10-some-title.md. As long as you don't make a habit of changing your markdown filenames, page.path should be a good choice as far as I can tell.

chuckmasterson commented 7 years ago

As for getting notifications to work, I'm closer but not there yet. It seems like it should work: I've created a mailing list by subscribing from a top-level comment, and then I've subscribed again from another top-level comment with a different email address, and the hash was the same and the new address was added.

If the mailing list exists and Staticman is getting its name right, I don't know what would be stopping it from delivering. I sent an email to the list from my regular email and it came through fine. I'll keep pondering what may have happened and try not to ask for too much of your time, @eduardoboucas, but if there's any log you can view that tells what's been happening on your Staticman, that might offer an easier route.

eduardoboucas commented 7 years ago

@chuckmasterson There's nothing I can see in the logs. Can you share a link to the repository you're using, if it's public?

chuckmasterson commented 7 years ago

Yeah, it's under this account at chuckmasterson.github.io.

eduardoboucas commented 7 years ago

@chuckmasterson Do you mind switching your Staticman config to temporarily use the public Mailgun account? All you need to do is disable fromAddress, apiKey and domain from under notifications. This is just so I can more easily test.

chuckmasterson commented 7 years ago

Sorry, I was away from computers for a few days.

Just changed the Staticman config over as you requested. If you want to test commenting on a blog post that's up, you can use this post. I don't have a checkbox for subscribing shown right now since I haven't gotten it working, but I'll work on getting a checkbox to show on just that page if that'd be helpful. Otherwise there's also some HTML commented out in the vicinity of the submit button that you could uncomment in DevTools or Firebug.

mmistakes commented 7 years ago

@chuckmasterson Looking at this again I completely missed what you were saying about how options[parent] works in regards to a mailing list. I'm thinking it might make sense to stop using it as a child comment identifier and come up with a different field to identify a comment's parent.

Then most of the liquid could stay the same, just with a variable rename and a batch replace of _parent on all the existing comment data files. options[parent] could then be assigned with something like {{ page.url | absolute_url }} in the form which should correctly subscribe a user to the entire comment thread.

Though I do like your suggestion of options.subscribeTo since that describes that field better than options.parent. Guess that one is up to @eduardoboucas if he wants to change the name and potentially break some sites :wink:

chuckmasterson commented 7 years ago

Yeah, what you described is what I ended up doing - something like sed -i 's/^_parent/replying_to/' */*.yml.

nakoo commented 7 years ago

I also have same problem. Mailgun only makes mailing list, not sending the email.

ghost commented 7 years ago

Trying again to get the notifications working correctly with a mailgun account. For the mailgun domain entry in the staticman.yml, mailgun provides this: API Base URL https://api.mailgun.net/v3/www.somedomain.com Does this need to be encrypted before adding to staticman.yml? Encrypting with this command: https://api.staticman.net/v2/encrypt/https://api.mailgun.net/v3/www.somedomain.com fails with this message: Cannot GET /v2/encrypt/https://api.mailgun.net/v3/www.somedomain.com Probably something simple ... but I'm not seeing it. Alec

chuckmasterson commented 7 years ago

@alecsatin Don't use "API Base URL", use the bit that's just labeled "Domain" in the Mailgun interface. In your case I think it'd be just www.alecsatin.com. I tried encrypting that and it worked.

ghost commented 7 years ago

Thank you. Using https://api.staticman.net/v2/encrypt/www.somedomain.com (with the actual domain) to encrypt works. I've updated the staticman.yml.

I'm still having an issue on the mailgun side. One of the txt entries will not validate no matter what I do at the registrar side. The other txt entry (mx._domainkey.www.somedomain.com) is fine.

nakoo commented 7 years ago

Is there any settings to use public instance of mailgun, not just adding enabled option? I have failed so many times for using this.

mattseemon commented 7 years ago

Hi... trying to get notifications working. Right now, my mailgun account seems to be setup correctly. I followed @mmistakes guide and was able to get everything working fine. At this moment, the only thing not working is the actual notification. While I am testing, I used different email ids to register for the notifications. I can see the mailing list getting updated with the new email id's... however, i am still not receiving the notifications. Is there something I missing. I followed @mmistakes guide to the letter.

Hund commented 7 years ago

Will there be any alternative to Mailgun? Something more anonymous, that doesn't require you to own the domain and doesn't require a phone number?

I have my blog at GitHub pages.

willymcallister commented 7 years ago

@eduardoboucas: Staticman (public) and my own MailGun account are working great at spinningnumbers.org. Thanks so much for creating a great service.

A feature request: When Staticman sets up a new mailing list at MailGun, is it possible for Staticman to initialize the "Name" field to the post slug? Some sort of clear text indication of what the mailing list is for.

eduardoboucas commented 7 years ago

@willymcallister That's great news, thanks! Any chance you could write a little guide describing the process you followed to get it working? I'm sure it would be useful to a lot of people.

As for your feature request, it makes perfect sense. I'll try to look into it soon. If you can, please open a separate issue so it doesn't get lost in this thread.

binarymist commented 6 years ago

Hi @eduardoboucas.

I'm currently unable to receive notification emails.

This is my staticman.yml with the relevant notifications config enabled and the encrypted apiKey and domain.

This is where I post comments from in code.

I'm not receiving any error from the Staticman API and no activity in my mailgun account.

Is my config and POST correct? Do I need to do something in my mailgun account that I've missed from the docs?

Thanks.