Create issues@ and builds@ lists and update notifications [LUCENE-8951]

[mikemccand]

Issue to plan and execute decision from dev mailing list https://lists.apache.org/thread.html/762d72a9045642dc488dc7a2fd0a525707e5fa5671ac0648a3604c9b@%3Cdev.lucene.apache.org%3E

1. Create mailing lists as an announce only list ✔
2. Subscribe all emails that will be allowed to post ✔
3. Update websites with info about the new lists ✔
4. Announce to dev@ list that the change will happen ✔
5. Modify Jira and Github bots to post to issues@ list instead of dev@ ✔
6. Modify Jenkins (including Policeman and other) to post to builds@ ✔
7. Announce to dev@ list that the change is effective ✔

---

Legacy Jira details

LUCENE-8951 by Jan Høydahl (@janhoy) on Aug 14 2019, resolved Sep 18 2019 Linked issues:

• INFRA-18933
  • INFRA-19012
  • INFRA-19013

[mikemccand]

I'm starting with step 1. Who wants to be moderators for the lists? There will be no real message moderation but you will have karma for whitelisting posters like bots etc.

[Legacy Jira: Jan Høydahl (@janhoy) on Aug 14 2019]

[mikemccand]

List of addresses that should be whitelisted for posting (please fill in) :

issues@:
git@apache.org
jira@apache.org

builds@:
jenkins@builds.apache.org
jenkins@thetaphi.de

[Legacy Jira: Jan Høydahl (@janhoy) on Aug 14 2019]

[mikemccand]

Requested list creation. I randomly added elyograg, msokolov and dsmiley as moderators in addition to myself. But there will be no moderation as this is an announce only list. However I believe you will have karma to manage the list.

[Legacy Jira: Jan Høydahl (@janhoy) on Aug 23 2019]

[mikemccand]

Filed INFRA-18933 for changing reply-to address for builds list, whitelist thetapi jenkins and add @thetaphi  as moderator for both lists.

Anyone that have time to update the websites and draft an announce mail? My next few days are quite full. The announce mail should point to the website for sign-up instructions and say that bot traffic will move to the new lists within the next few days...

[Legacy Jira: Jan Høydahl (@janhoy) on Aug 26 2019]

[mikemccand]

I [added jenkins@thetaphi.de|mailto:added%C2%A0jenkins@thetaphi.de] to allow-subscribe and INFRA completed the other tasks.

[Legacy Jira: Jan Høydahl (@janhoy) on Aug 26 2019]

[mikemccand]

@thetaphi  perhaps you can try to send policeman jenkins results to the new list already, and confirm that it works. I'll get to the website updates and ANNOUNCE email later if no-one beats me to it.

[Legacy Jira: Jan Høydahl (@janhoy) on Aug 28 2019]

[mikemccand]

I didn't see anything come in.  Will they sign up my @apache.org email address?

[Legacy Jira: Shawn Heisey (@elyograg) on Aug 28 2019]

[mikemccand]

I didn't see anything come in.

No-one are auto subscribed to the new lists, will send out an email to dev@ to ask people to subscribe if they wish.

Also, I expect no emails have been sent to those lists either, so even if you are a moderator there has been nothing to moderate yet :)

[Legacy Jira: Jan Høydahl (@janhoy) on Aug 28 2019]

[mikemccand]

OK, I changed Policeman Jenkins. I will send a test e-mail, but before that I will subscribe to both new lists.

[Legacy Jira: Uwe Schindler (@uschindler) on Aug 28 2019]

[mikemccand]

Hi, I sent a test email, but it went to moderation. I then used replay-all, to put the mail on the allow list of the ML, but it did not help (it sent it to 2 mails, accept-xxx@ and allow-xxx@). I did a 2nd test mail was also stuck in moderation.

Maybe somebody has to manually put the mail on the allow list.

[Legacy Jira: Uwe Schindler (@uschindler) on Aug 28 2019]

[mikemccand]

I sent another one, I keep it up to your to enable (maybe I have not enough rights on the laist). The CC address of the moderation mail is the the one to allow the sender forever (this is why reply-all works).

[Legacy Jira: Uwe Schindler (@uschindler) on Aug 28 2019]

[mikemccand]

FYI, please DO NOT SUBSCRIBE jenkins@thetaphi.de!!!! This mail cannot receive mails, all mail coming in is rejected with some special error code and mailer-daemon message. It's only to send mails (a typing do-dot-reply one). If yozu subscribe it, the ML will remove him asap because of error code. So you must put it on the ALLOW list of the mailing list, not the subscription list.

[Legacy Jira: Uwe Schindler (@uschindler) on Aug 28 2019]

[mikemccand]

What I did 2 days ago was sent an email to builds-allow-subscribe-jenkins=thetaphi.de@lucene.apache.org (using https://whimsy.apache.org/committers/moderationhelper to help build the command), and then got a response

Hi! This is the ezmlm program. I'm managing the
builds-allow@lucene.apache.org mailing list.

I'm working for my owner, who can be reached
at builds-allow-owner@lucene.apache.org.

I respectfully request your permission to add

  jenkins@thetaphi.de

to the subscribers of the builds-allow mailing list. This request
either came from you, or it has already been verified by
the potential subscriber.

To confirm, please send a short reply to this address:

  builds-allow-rc.1566814551.iejafhgalokpajfflood-jenkins=thetaphi.de@lucene.apache.org
... 

I tried replying to that mail but obviously it is still not added. I think maybe the reply has to come from that email itself? I'm not very steady in ezmlm commands.

[Legacy Jira: Jan Høydahl (@janhoy) on Aug 28 2019]

[mikemccand]

I ran the allow-list command and got this reply

Fra: builds-allow-help@lucene.apache.org
Emne: Subscriber list for builds@lucene.apache.org
Dato: 28. august 2019 kl. 13:34:58 CEST
Til: janhoy@apache.org

Hi! This is the ezmlm program. I'm managing the
builds-allow@lucene.apache.org mailing list.

I'm working for my owner, who can be reached
at builds-allow-owner@lucene.apache.org.

Subscribers to this list are:
jenkins@thetaphi.de

 ======> 1

[Legacy Jira: Jan Høydahl (@janhoy) on Aug 28 2019]

[mikemccand]

Ugh; why do mailing lists have to be such a usability nightmare.  Ancient software.

[Legacy Jira: David Smiley (@dsmiley) on Aug 28 2019]

[mikemccand]

No idea, ask Infra. Did you get the moderation of my 3rd test mail?

[Legacy Jira: Uwe Schindler (@uschindler) on Aug 28 2019]

[mikemccand]

I did not get a moderation email for 3rd policemen mail.

[Legacy Jira: Jan Høydahl (@janhoy) on Aug 29 2019]

[mikemccand]

The Jenkins server already sent several mails, but all got lost (not even appearing in moderation). It looks like they are swallowed completely. Could it be that there is some size limit configured on this mailing list (in contrast to dev@)? Maybe the server drops them because jenkins mails are quite big because of log files.

This is a failed build and I neither got a mail notification through mailing list nor any moderation mail: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/24630/console

Aug 29 09:58:05 serv1 postfix/smtpd[8811]: connect from ip6-localhost[::1]
Aug 29 09:58:05 serv1 postfix/smtpd[8811]: 502FE10800C7: client=ip6-localhost[::1]
Aug 29 09:58:05 serv1 postfix/cleanup[8814]: 502FE10800C7: message-id=<1214521720.14.1567072685329.JavaMail.jenkins@serv1.sd-datasolutions.de>
Aug 29 09:58:05 serv1 postfix/qmgr[3889]: 502FE10800C7: from=<jenkins@thetaphi.de>, size=105988, nrcpt=1 (queue active)
Aug 29 09:58:05 serv1 postfix/smtpd[8811]: disconnect from ip6-localhost[::1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
Aug 29 09:58:06 serv1 postfix/smtp[8815]: 502FE10800C7: to=<builds@lucene.apache.org>, relay=mx1-he-de.apache.org[2a01:4f8:c2c:2bf7::1]:25, delay=1.6, delays=0.1/0/0.22/1.3, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as E24BE7D722)
Aug 29 09:58:06 serv1 postfix/qmgr[3889]: 502FE10800C7: removed

Not sure what happend with that mail! I think we may ask infra on Slack, all required information to identify that mail should be above.

[Legacy Jira: Uwe Schindler (@uschindler) on Aug 29 2019]

[mikemccand]

I also found no bounces on my company's incoming server.

[Legacy Jira: Uwe Schindler (@uschindler) on Aug 29 2019]

[mikemccand]

I only see those two first test mails in PonyMail as well. Great if you follow up with Infra on this Uwe.

[Legacy Jira: Jan Høydahl (@janhoy) on Aug 29 2019]

[mikemccand]

Will do this later as most of them are sleeping.

[Legacy Jira: Uwe Schindler (@uschindler) on Aug 29 2019]

[mikemccand]

Hi, INFRA also does not know exactly why the mails are lost. It looks like they are filtered because of non-text or other attachments. The builds@ settings are different from dev@ where also HTML and attachments are allowed.

They enabled HTML now, but that was not the issue. Analyzing a mail from jenkins looks like it uses multipart/mixed format: https://lists.apache.org/api/source.lua/d415bf5c26db9aaaa1ab17a2b92f664728de9927cd788fc07afd8fd3@%3Cdev.lucene.apache.org%3E

I asked them to enable it. @Humbedooh, any news?

[Legacy Jira: Uwe Schindler (@uschindler) on Aug 31 2019]

[mikemccand]

Updated website to list the two new lists. Also added our Slack channel and fixed some old wiki links:

https://lucene.apache.org/solr/community.html 

[Legacy Jira: Jan Høydahl (@janhoy) on Sep 06 2019]

[mikemccand]

Also updated these pages:

• https://lucene.apache.org/index.html#6-september-2019-new-mailing-lists
• https://lucene.apache.org/core/discussion.html 

Will send this ANNOUNCE email to general@ and dev@:

[ANNOUNCE] New builds@ and issues@ mailing lists

The Lucene project has added two new announce mailing lists, issues@lucene\.aparche\.org and builds@lucene\.apache\.org. High-volume automated emails from our bug tracker, JIRA and GitHub will be moved from the dev@ list to issues@ and automated emails from our Jenkins CI build servers will be moved from the dev@ list to builds@.

This is an effort to reduce the sometimes overwhelming email volume on our main development mailing list and thus make it easier for the community to follow important discussions by humans on the dev@lucene\.apache\.org list.

Everyone who wants to continue receiving these automated emails should sign up for one or both of the two new lists. Sign-up instructions can be found on the Lucene-java[1] and Solr[2] web sites.

[1] https://lucene\.apache\.org/core/discussion\.html

[2] https://lucene\.apache\.org/solr/community\.html

[Legacy Jira: Jan Høydahl (@janhoy) on Sep 06 2019]

[mikemccand]

Who has the karma to redirect Jira, GitHub and Apache Jenkins traffic?

[Legacy Jira: Jan Høydahl (@janhoy) on Sep 06 2019]

[mikemccand]

Jan:

Many thanks for putting the effort into this....

[Legacy Jira: Erick Erickson (@ErickErickson) on Sep 07 2019]

[mikemccand]

Sent the email to notify people on what will happen.

Who knows how to reconfigure Jenkins, Jira & GitBox?

[Legacy Jira: Jan Høydahl (@janhoy) on Sep 09 2019]

[mikemccand]

The mails by Jenkins are still getting lost. I had no time to keep track. It looks like we need to open an issue and tell them to reconfigure the lists to disable any content filters. Jenkins sends huge mails with various MIME types in multipart/mixed.

[Legacy Jira: Uwe Schindler (@uschindler) on Sep 09 2019]

[mikemccand]

Uwe, can you followup on the Jenkins (builds@) part, and I'll ask INFRA to reconfigure JIRA and GitBox to send to issues@.

[Legacy Jira: Jan Høydahl (@janhoy) on Sep 11 2019]

[mikemccand]

INFRA-19012 is the issue for Jira and GitBox

[Legacy Jira: Jan Høydahl (@janhoy) on Sep 11 2019]

[mikemccand]

INFRA-19013 is the jenkins problem

[Legacy Jira: Uwe Schindler (@uschindler) on Sep 11 2019]

[mikemccand]

Hi, INFRA-19013 looks like solved, but there were 2 problems:

• Jenkins sends all mails with Precendence: bulk and by default all mails using that flag are discarded. This was fixed: According to the docn [1], X means "Do not reject bulk senders, limit size, nor remove parts"
• The list had the flag "moderate all subscribers"

[Legacy Jira: Uwe Schindler (@uschindler) on Sep 13 2019]

[mikemccand]

Thanks Uwe. Will you try to redirect Apache Jenkins as well?

INFRA-19012 (issues@ redirects) is still in the queue, will ping them next week if still unresolved (I'm assuming none of us have karma to change GitBox or Jira notifications)

[Legacy Jira: Jan Høydahl (@janhoy) on Sep 13 2019]

[mikemccand]

Jan, I will take care of ASF Jenkins after lunch. I am not sure about status of ~steve_rowe and Elasticsearch Jenkins. I hope ASF Jenkins allows bulk changes... Uwe

[Legacy Jira: Uwe Schindler (@uschindler) on Sep 13 2019]

[mikemccand]

Not sure how we handle the "announce list" only. We had to remove the "moderate subscribers" flag, as this also affects the "allow" list.

I think the problem is solved by the Reply-To setting, but it's now possible that subscribers can send mail to the list (if they use mail address explicitely).

[Legacy Jira: Uwe Schindler (@uschindler) on Sep 13 2019]

[mikemccand]

I am on INFRA Slack if we can use the configuartion slicing plugin for the change on ASF jenkins, its way to many jobs to do it one by one.

[Legacy Jira: Uwe Schindler (@uschindler) on Sep 13 2019]

[mikemccand]

OK, all jobs changed by them with "Configuration Slicer" -> "Editable e-mail-notification".

[Legacy Jira: Uwe Schindler (@uschindler) on Sep 13 2019]

[mikemccand]

I will take care of ASF Jenkins after lunch. I am not sure about status of Steve Rowe and Elasticsearch Jenkins.

My Jenkins doesn't email any lists.

[Legacy Jira: Steven Rowe on Sep 13 2019]

[mikemccand]

Testing notification :)

[Legacy Jira: Jan Høydahl (@janhoy) on Sep 13 2019]

[mikemccand]

There's a typo on the news page "aparche".  I tried to use the CMS bookmarklet to update it but I'm forgetting all the steps I need to do for this since my change seems to not be published yet.

[Legacy Jira: David Smiley (@dsmiley) on Sep 13 2019]

[mikemccand]

Finally it published without timing out this time.  Done.

Following all the comments here is a lot so can someone re-iterate if there is going to be any true moderation tasks for me to do?  I've received some moderation emails and took no action.

[Legacy Jira: David Smiley (@dsmiley) on Sep 13 2019]

[mikemccand]

Thanks for fixing the typo.

Yea, I don't want to be subscribed on my apache address either. Is it possible to unsubscribe and still be a moderator?

[Legacy Jira: Jan Høydahl (@janhoy) on Sep 13 2019]

[mikemccand]

Yea, I don't want to be subscribed on my apache address either. Is it possible to unsubscribe and still be a moderator?

+1 here.  I'd rather use the personal email address I've dedicated for use with apache stuff, where my @a.o address forwards to. Skip the forward, make my filters easier to write.

[Legacy Jira: Shawn Heisey (@elyograg) on Sep 13 2019]

[mikemccand]

Looks like I'm getting two emails for everything going to issues@a.o. One sent to my @a.o address and forwarded, and one sent directly to me.

[Legacy Jira: Shawn Heisey (@elyograg) on Sep 13 2019]

[mikemccand]

@elyograg: Maybe because you are "watcher" of this issue? I also get 2 mails: One to the mailing list and one to me privately. But thats usual.

[Legacy Jira: Uwe Schindler (@uschindler) on Sep 14 2019]

[mikemccand]

Maybe because you are "watcher" of this issue?

If you count that, I actually get 3. But the one generated by watching isn't sent to issues@ and gets filtered to a different mail folder, because it doesn't have the List-Id header. I think I'm getting the other two because I signed up my usual apache@elyograg.org email, and I'm also a moderator with elyograg@apache.org.

[Legacy Jira: Shawn Heisey (@elyograg) on Sep 14 2019]

[mikemccand]

I think I'm getting the other two because I signed up my usual apache@elyograg.org email, and I'm also a moderator with elyograg@apache.org.

Yep me too. I just sent an unsubscribe request from my @apache.org address so I only get the mails to one account. I think perhaps I'm still a moderator with my apache mail even if I don't get all posts there...?

[Legacy Jira: Jan Høydahl (@janhoy) on Sep 14 2019]

[mikemccand]

One more thing that we discussed earlier.

It is possible to get all [jira] [Created] emails to dev@ list, i.e. a one-time mail for every new JIRA. I think that could be smart so you can stay on top of new issues even without following issues@ list, and add yourself as watcher to the ones you want to monitor. WDYT? (Perhaps I just felt the dev list became a bit too silent :) now)

[Legacy Jira: Jan Høydahl (@janhoy) on Sep 14 2019]

[mikemccand]

Should be possible to change in the notification scheme. Maybe lets send "created" mails *additionally" to dev@. This simplifies things. So we just need to add one more "user" to the list of notified people.

[Legacy Jira: Uwe Schindler (@uschindler) on Sep 15 2019]

[mikemccand]

Resolving this. I'll open a discussion thread on dev@ about whether to include [Created] notifications there.

[Legacy Jira: Jan Høydahl (@janhoy) on Sep 18 2019]