16) Does Github have a daily digest function?

[alintheopen]

Github is generating too many emails for some people and is likely to be off-putting to those unused to working with this platform. Gmail collates these fairly painlessly, but other programs like Outlook do not, meaning there's a lot of apparent "spam". Is there a digest function for GitHub to limit the number of emails sent per day?

[alintheopen]

A number of people have complained about the receipt of too many notifications. This page from GitHub provides great advice for streamlining notifications:

https://help.github.com/articles/notifications

[mattodd]

Still can't see a way to alter frequency of the incoming emails... https://help.github.com/articles/configuring-notification-emails Anyone?

[mattodd]

Pinging @PatrickThomson here since this was just discussed at OSM's website/design meeting, and the suggestion was made to contact some people at Github to see if they could help with this. http://malaria.ourexperiment.org/osddmalaria_meeting_/8562

[PatrickThomson]

Yes, I have had a significant discussion about this with @mikemcquaid just now. Essentially, this is an issue because "Issue Trackers" aren't really used the way we're using it and this isn't something that any major Issue Tracker will do. The solutions are:

1) Educate users about how to filter email.

2) Ask users to unfollow the repository and browse it manually for updates.

3) Split into multiple repositories such as OSM_To_Do_Chemistry, so that people only get updates on areas that interest them

4) create a github user called "OSMMailingBot" that forwards all update mails to a mailing list (most likely Google Groups) that anyone can subscribe to but nobody else can post to. The mailing list is set to generate daily digests by default. GitHub's emailing behaviour would remain unchanged, but we can advertise a set of simple instructions for disabling email from GitHub directly and instead following "osm_daily_digest@google".

I think that 1-3 all present significant barriers to uptake by potential contributors, and favour 4 (which is not my idea, credit where credit's due). I would be happy to (attempt to) arrange this, but might need @alintheopen to help me on bolstering the user-friendliness of any guide I end up writing.

[PatrickThomson]

TO expand a little on "isn't used the way we're using it", Issue trackers tend to be an all-or-nothing. The "I fix bugs" person will want to be notified immediately, but other users will want to view specific links, subscribe to specific issues, or browse the entire repository indiscriminately. There's not really a use case for bug and issue trackers in the world of software where people want to view every post, but in a delayed digest.

We want those features of a mailing list or forum, but with an Issue Tracker's tags, ownership, progress, milestones, and so on and no single product exists to do those. The question of whether Open Science needs new tools to do this, is a much larger discussion that perhaps we should be having, but separately as a significant development akin in scale to an Electronic Lab Book itself.

[edit] The problem is also somewhat exacerbated by the fact that we're exposing relatively (for Open Source) tech-unsavvy end users directly to github's email feed, whereas the core members are either intimately involved with every single issue, or familiar with email filtering tools.

[mattodd]

Yes, I like 4), above - seems neat, provided the instructions are super-easy. It strikes me, though, that if we're going to start creating a mailing bot, we could use that to collate all kinds of activity into one daily email, could we not? i.e. new activity on all platforms, grouped? If we did that, we'd need to allow people to subscribe to e.g. weekly digests, because there would be a lot. This could, in other words, double as the Newsletter Alice was mentioning, only a lot more frequent.

[alintheopen]

Yes, I favour option 4 too. How about we get this up and running by the end of the week @PatrickThomson and then ask @mattodd, @murrayfold and @PaulWillisMMV to subscribe to the feed and feedback? Sound reasonable?

[PaulWillisMMV]

Option 4 sounds by far the best and worth a try, I’m happy to be part of the trial.

I would also be a strong supporter of finding a way of grouping or structuring the GITHUB topics they do not appear on OSM as a seemingly random collection of unrelated tasks

Paul

From: alintheopen [mailto:notifications@github.com] Sent: Tuesday, 18 February 2014 5:24 AM To: OpenSourceMalaria/OSM_To_Do_List Cc: Paul Willis Subject: Re: [OSM_To_Do_List] 16) Does Github have a daily digest function? (#134)

Yes, I favour option 4 too. How about we get this up and running by the end of the week @PatrickThomsonhttps://github.com/PatrickThomson and then ask @mattoddhttps://github.com/mattodd, @murrayfoldhttps://github.com/murrayfold and @PaulWillisMMVhttps://github.com/PaulWillisMMV to subscribe to the feed and feedback? Sound reasonable?

— Reply to this email directly or view it on GitHubhttps://github.com/OpenSourceMalaria/OSM_To_Do_List/issues/134#issuecomment-35351459.

---

This email and any attachments may contain confidential information. If you are not the intended recipient, please delete the email and any attachments and notify the sender immediately.

---

[murrayfold]

A weekly digest with links to issues is the best option, if it can be automated and still readable then even better.

As someone who knows how the project works but is no longer working on it all the time, I find it hard to keep up to date with all the threads in github. I think this could put a lot of new people off.

I have also turned off the email notifications as I was getting too many. I now only get emails for threads I'm mentioned in.

[PatrickThomson]

Google mail and Google groups have several safeguards in place that explicitly prevent this from happening with a single-platform solution.

I have created the github user @OSM-MailerBot and the Google Group osmdigests - but getting the two to talk together is specifically blocked. Google groups will not allow me to give GitHub posting rights - and I cannot add GitHub's outgoing email address as a mailing list user. Several people suggested having an intermediate email address that does forwarding so I created the new user "osmmailerbot@gmail.com" - but GMail prevents automated forwarding to a Google group. Solutions are:

1) Don't use Google Groups to manage the mail collation and digestion

2) Use Google Groups, but don't use GMail as the proxy forwarding account

3) I've missed something

Any suggestions?

[PatrickThomson]

3), I missed something - the gmail "forwarding verification" was being dropped from the mailing list. This comment I am currently writing serves as a correction and a little test message as well and should be one of the last guilt-laden extraneous emails I generate on the project.

[PatrickThomson]

Google groups is not a very full featured mailing list software and the following problems are outstanding:

a) Digest options are limited to "email once every 25 posts, or once a day, whichever is more frequent" and emails are sent at any time in a 12 hour window each day.

b) A user subscribing to the daily digest is only possible when the user has a google account. Non-google addresses can subscribe to daily digests but this has to be done manually by an administrator*

Does anyone have any suggestions for a better mailing list software or workarounds?

_{^{*Specifically, non-google addresses can join but default to receiving "All Mail". Also, administrators can not be automatically notified when people join.}}

[alintheopen]

I don't think we need a google group for this @PatrickThomson. I would suggest that we just use GitHub to collate a weekly digest of linked issues say on a Thursday and then all who are watching this feed (every member of the OSM team plus interested extras) would receive one GitHub email a week rather than the 10s or sometimes 100s that watchaholics receive each week.

[PatrickThomson]

The issue is that GitHub (and issue trackers in general) don't support any kind of digest or collated email whatsoever; what I've put together is not a fully fledged Google Group in the sense of a sharing space or "another account", rather I'm just using it as a quick convenient way of bunching up emails into one big notification about all activity that's happened. Trying to get this feature added to GitHub is ... a bigger request than I think we realise.

[alintheopen]

Cheers Patrick, yeah I get you I was just thinking of a way around this issue and watching one feed that collates links posts may be one way to circumvent this issue for the moment - certainly worth a trial and something that we would have to all share responsibility for as its too much work for one person if the job is manual to begin with.

[PatrickThomson]

Not sure what you mean about "watching one feed that collates links posts"? The mailing list solution only requires manual intervention to add people to the list of recipients (they can remove themselves)

EDIT: Each daily email would be something like:

From: Patrick Subject: I made stuff Chemistry! --click to view on github

From: Alice Subject: I made stuff Chemistry! --click to view on github

--click to unsubscribe from mailing list "osmdigests"

[alintheopen]

This keeps the summary on GitHub, people then subscribe or don't subscribe to the post and for those non-GitHubbers, the text from the feed can be copied and pasted and sent as an email or a link to GH. This keeps the 'bulletin' interactive.

[PatrickThomson]

You're talking about manually updating a post of all relevant or watched issues? I think that sounds like a different feature to the one that's been requested of "slightly less individual emails, but still keeping me up to date". We might need to get a more clearly defined request then, because I put something together for a specification based only on what I heard in the meeting. Again to clarify, the only thing an email digest would do is run on the text of individual emails from GitHub, keeping all the links in place, and still containing all the text of the individual posts in the order they were made.

[alintheopen]

Manual is definitely not what we're aiming for so what you're putting together is certainly the right way to go but from what you've said we probably can't use google groups for the reasons that you have mentioned. I was suggesting an interim solution which is not the way we should be doing it as we need an automated feature. The landing page works by pulling from the GH feed so don't we need a similar script to do the same but on a weekly basis?

[PatrickThomson]

I corresponded briefly with @chuckfitzpatricksf about this - my understanding is that a tool that that took the website feed and sent it as a collated email would actually involve lots of coding and having a new system that sends emails and lets people manage subscriptions, and it'd probably may even have to be built from scratch (maybe as much work again as the website itself). It might be something we would do further down the line but it'd be a very big ask.

Whereas the mailing list has drawbacks, sure, in that every new user (for now) would need a brief checkbox checked, but I don't view that as a massive problem - the number of individual users is likely to be small (but each of those users is significant). It's a more immediate interim solution because I could be adding people to it tomorrow to test it. And, people with a google account (which is most of us) can add and manage our subscription ourselves, it's only people without google accounts who would need to be manually added and that's only a flaw with the choice of mailing list software - google is quick, simple, and free but not the only option. Both of our universities offer institutional mailing list software which is far more powerful and flexible than GG.

[PatrickThomson]

I do like the idea of having a single issue that's kept regularly updated with links to e.g. all compounds under synthesis, but it sounds like the manual intervention requirement would be quite a load not unlike keeping the wiki up to date - if it's necessary to commit that level of effort to keep contributing scientists engaged then I'll be the first to step up to update it.

[mattodd]

Very useful, and excellent attempts there Patrick. Shame the different components are being tricky in communicating.

The discussion about Github issues makes me think, though. These issues are the primary means of alerting people to things that need doing, when properly managed, tagged and curated. But there is other activity in the project, for example blog post updates, tweets and other suggestions. I'm wondering if the high value outcome here is a weekly newsletter that is manually assembled on a listserv: http://www.wikihow.com/Create-a-Listserv

Such a system: 1) Allows anyone to opt in or out of receiving notifications 2) Allows grouping of bursts of activity into a daily digest, should that happen 3) Allows the occasional mail-out of news, which would be done weekly, and allows anyone to post a message to the group 4) Is useful in a number of online communities I subscribe to, i.e. tried and tested by communities who are content with advanced online platforms as well as those more used to email.

To the best of my knowledge the contents of such emails cannot include pictures, is that right? That's a pain for "molecules being synthesised"

So to do this we'd have to assemble, each week, a summary of activity. Manual is bad, but manual creates high value content that is not spammy. And if anyone could post to the list, we could share the burden and invite guest newsletters.

I also have this idea that the contents of listserv emails can be maintained publicly? I have in the past found such discussions through Google searches.

Perhaps I could assemble an example of how I see this, and we can give our opinions on whether this is sustainable?

[alintheopen]

A quick reply for now, but I think I agree. We could probably do with both but people can tell when they are receiving a 'mailbot' automated response or summary. This list needs to be engaging, informative and probably a single A4 page so that it doesn't overwhelm a reader. I think we could take it in turns to do this newsletter/email so that it doesn't become too much of a burden to one person.

On Thu, Feb 20, 2014 at 4:35 PM, Mat Todd notifications@github.com wrote:

Very useful, and excellent attempts there Patrick. Shame the different components are being tricky in communicating.

The discussion about Github issues makes me think, though. These issues are the primary means of alerting people to things that need doing, when properly managed, tagged and curated. But there is other activity in the project, for example blog post updates, tweets and other suggestions. I'm wondering if the high value outcome here is a weekly newsletter that is manually assembled on a listserv: http://www.wikihow.com/Create-a-Listserv

Such a system: 1) Allows anyone to opt in or out of receiving notifications 2) Allows grouping of bursts of activity into a daily digest, should that happen 3) Allows the occasional mail-out of news, which would be done weekly, and allows anyone to post a message to the group 4) Is useful in a number of online communities I subscribe to, i.e. tried and tested by communities who are content with advanced online platforms as well as those more used to email.

To the best of my knowledge the contents of such emails cannot include pictures, is that right? That's a pain for "molecules being synthesised"

So to do this we'd have to assemble, each week, a summary of activity. Manual is bad, but manual creates high value content that is not spammy. And if anyone could post to the list, we could share the burden and invite guest newsletters.

I also have this idea that the contents of listserv emails can be maintained publicly? I have in the past found such discussions through Google searches.

Perhaps I could assemble an example of how I see this, and we can give our opinions on whether this is sustainable?

Reply to this email directly or view it on GitHubhttps://github.com/OpenSourceMalaria/OSM_To_Do_List/issues/134#issuecomment-35590747 .

[miike]

To be honest, the easiest way to do this would be to subscribe to an existing easy to use mailing service (such as Mailchimp) and auto generate a daily or weekly email to users who are members of that list. We can autogenerate the email using whatever content we want and if that's just material from the issues on Github that's very easy. We can even label certain issues (as say, 'mailing-list') and select only new posts from those issues to go on the mailing list.

[PatrickThomson]

Agreed @miike - a weekly newsletter is great, would be a good primer/introduction and a good archive to rely on to quickly assemble an overview of the project. I suggest we start a new issue for for a "manually curated and engaging weekly newsletter".

The problem this issue is titled is "Does GitHub have a daily digest for issue emails?" - it's far easier to solve and the solution looks something like this:

This is the "full text digest" email that was generated a few hours ago. Each topic is threaded and the pale blue links at the top are clickable and skip to the sections of interest. You can see in @IngaTopolnicki 's activity in issue #150 - the emails are run together but with links to the individual posts (again this is all generated automatically as a workaround for GitHub not doing a digest email, a feature a few people have asked for). There's also an "abridged" mode, that only sends a summary of emails each day.

I suggest that we (ethically) experiment on @murrayfold as Murray is a good model for "people who turned off github because of the volume of individual emails".

Additionally I have thought of a workaround so that the daily digest list can be automatically followed and unfollowed by anyone just by emailing "listname"+subscribe@googlegroups.com or "listname"+unsubscribe@googlegroups.com - no manual intervention necessary. Of course, MailChimp would be much more powerful (thanks @miike for the suggestion) so I'll look into it.

[murrayfold]

Feel free to use me as a guinea pig. I'd prefer weekly updates rather than daily. I personally would be more likely to pay attention to weekly emails than daily ones.

[mattodd]

@miike - the kind of service you're talking about:

1) is it possible for different subscribers to have different frequencies, i.e. some people to receive daily, some to receive weekly. Is that achieved by having different lists?

2) What other kind of content can be pulled in?

3) Is it possible to mix auto-generated content with manually added content? i.e. for an email to be "drafted" with auto content, be edited and then an approver to click "send"?

Patrick - it's possible these two issues (Github frequency and newsletter) are separate, but there's a chance they can both be solved with the same solution. If not, we should fork the issues, yes. I am viewing this from the user end. What would I like to see? Maybe I'd like an email that arrives with a frequency I can select, and I'd like to have one email that contains everything - Github issues in one section, other stuff in other sections. Of course it may well be that we keep these things separate if it is seen that a hardcore OSM element will want regular Github updates and that other people would be happy to receive these once a week with other stuff. Perhaps one subscription event can be to two email sources. One can select either/or Github and/or Full News, and in each case receive emails a) as things come in, b) daily or c) weekly. I'd like to explore this topic a bit more before we fork the issues.

On 20 February 2014 20:47, Murray Robertson notifications@github.comwrote:

Feel free to use me as a guinea pig. I'd prefer weekly updates rather than daily. I personally would be more likely to pay attention to weekly emails than daily ones.

Reply to this email directly or view it on GitHubhttps://github.com/OpenSourceMalaria/OSM_To_Do_List/issues/134#issuecomment-35603413 .

MATTHEW TODD | Associate Professor School of Chemistry | Faculty of Science

THE UNIVERSITY OF SYDNEY Rm 519, F11 | The University of Sydney | NSW | 2006 T +61 2 9351 2180 | F +61 2 9351 3329 | M +61 415 274104 E matthew.todd@sydney.edu.au | W http://sydney.edu.au/science/chemistry/research/todd.html

CRICOS 00026A This email plus any attachments to it are confidential. Any unauthorised use is strictly prohibited. If you receive this email in error, please delete it and any attachments.

[miike]

@mattodd

1) I believe so yes, it wouldn't be particularly difficult to have two (or for that matter more) mailing lists one as a daily digest and one as a weekly digest.

2) Really any content we want is possible. I know that Github content is likely to be the easiest because they have a solid API which we can pull data from.

3) I'd have to investigate but tenuously yes. For each email we intend to send out it could be autogenerated, saved as a draft, and then be subject to manual sending (or alternately have a time at which the email is generated, an interval period for manual editing and then an automatic send out later).

[PatrickThomson]

Definitely agree that these are separate issues - what most people seem to want here is some much larger, more complex, "ideal solution" which will involve new design and upkeep; I think we need to be very clear about what people are asking for, and the following questions need answers before anything else should be done:

1) Is the frequency of emails a barrier to adoption and contribution? 2) How many people have unsubscribed from the repository because of the frequency of emails? 3) With people who have unsubscribed, how many of them actually want or need access to every single post? 4) Can these needs only be met by a human-created summary of all new chemistry? 5) Would a "daily" and/or "weekly" option in GitHub's email settings actually be sufficient?

Specifically, I have jumped the gun and made a solution assuming that question 5 is answered "yes" but all of these need to be answered before we actually do anything.

As a larger philosophical point - as someone who's spent time in the software and chemical worlds, I can see some problems here with a lack of definition in what we're asking for. Specifically, answering the question "can we make a daily digest of GitHub emails" with "Can't we just have a newsletter with structures and manual updates so on" is roughly equivalent to answering the question "can't we make the hydrochloride salt?" with "Can't we just put a new methyl group in this position?". The only reason it seems like these are the same type of question is a lack of experience, and the word "Just" is somewhat misguided.

[mattodd]

OK, so I'm going to close this Issue now, because the answer to the Issue "Does Github have a daily digest function" is "No".

What I'd suggest is that Patrick continues his excellent work in setting up a solution to the Github grouping of issues, and that @murrayfold provides feedback on whether that helps. If Murray is happy, we trial it on Paul Willis before activating for others. I foresee that the target of such a digest would be a member of OSM who is technically able and willing to help with one part of OSM (e.g. synthesis, or data handling etc) but who is less interested in Issues that might crop up outside of that area. Clearly for Paul it works since oversight can be provided without lots of emails. I envisage this as being attractive for medchem champions we need. If the grouping works, then we might all feel happier listing more Issues up here for smaller things.

So Patrick - would you like to create a new Issue on this with a resolvable title, so for example "Can we generate a workable periodic email summary of recent Github issues?" Would you mind, in the planning stages, trying to allow for the user to select how often they get emails? I'm thinking a) as they come in, b) daily, c) weekly.

In parallel to that I will start looking at the newsletter issue with Alice. I will start an issue about developing a mock-up of what we want any such thing to contain, and to what extent it might be generated by machine. This separates out the two threads.

Clearly it would be desirable if these two things talked to each other, so that, for example, were there to be a good solution to the Github grouping, that the output of that could be fed into a newsletter. I think we proceed as things stand, but keep an eye on interoperability. That was, I think, the subject of much of the freeform discussion above.

Closing. Please only reopen for major disagreements about the way forward.

[mwpowellhtx]

a digest feature would be great. most mailing lists support something like this. would help with the inbox clutter, for sure. one or three routers would probably thank ye as well. :+1:

[gusaiani]

Sorry to get everyone notified via email about exactly this :), but a "digest" option when following a repo would be so welcome.

Imagine being able to: · follow · unfollow · ignore · get a daily digest · get a weekly digest

If done well in the UI, this would be great.

[drc007]

Sounds like a volunteer?

On 14 Oct 2015, at 18:59, Gustavo Saiani notifications@github.com wrote:

Sorry to get everyone notified via email about exactly this :), but a "digest" option when following a repo would be so welcome.

Imagine being able to: · follow · unfollow · ignore · get a daily digest · get a weekly digest

If done well in the UI, this would be great.

— Reply to this email directly or view it on GitHub https://github.com/OpenSourceMalaria/OSM_To_Do_List/issues/134#issuecomment-148136630.

[mattodd]

sure - in fact the one that @PatrickThomson made, and which I signed up to, is still working very nicely (thanks Patrick). So nicely that I'd forgotten it wasn't a standard feature... Does anyone else use it? And to revisit the question from scratch - does Github itself yet have a digest feature?

[pmucci]

This is a must-have feature for github, and I truly can't image how something so simple is not implemented. The lack of such a feature is evidence of design-by-engineers not design-for-users. The fact that there's a debate about this solidifies the fact that my groups wouldn't consider paying github for anything unless this is implemented.

Come on folks. This can be implemented with a few dozen lines of Python ...

[jcrben]

You can vote on this at the unofficial public feature request / bug tracker at https://github.com/isaacs/github/issues/340 but you should be sure to send an email to Github if you care about it...

[cryptoralf]

+1 Please implement daily digest emails.

[olafdimigen]

+1 Please do.

[imtrobin]

I hope microsoft add it in.

[alexholcombe]

Gitlab doesn't offer digests, but maybe gives more control of notifications? https://docs.gitlab.com/ee/workflow/notifications.html

[imtrobin]

It's different. I want to know the project progress, but not when there's 10 emails coming in every hour

[JohannesMP]

+1 Please add a setting to limit notification emails to a single daily digest on subscribed repos. Either as a global setting or on a per-repo basis, both would be fine.

[pmucci]

It's 2020 folks. I'm now unfollowing repos due to the lack of a digest function.

[0xdevalias]

While not a native solution, I came across these while looking today:

• https://www.cockroachlabs.com/blog/creating-a-digestible-github-digest/
• https://github.com/spencerkimball/repo-digest