Consider formalizing a comment period before accepting new standard names

[ethanrd]

The Rules for Changing Standard Names page gives general guidelines for reaching consensus but is not explicit about announcing agreement and having a comment period to allow for any objection before accepting new terms.

@japamment explains the current practice in this comment to issue cf-convention/vocabularies#74 (which lines up with the current rules).

I suggested that a more formal announcement and comment period would be more transparent.

[ethanrd]

Hi Alison (@japamment),

I agree that it is important to allow for quick acceptance of new standard names when they match existing patterns and definitions and there is not a lot of discussion or dissent. However, as you have said, it is also very important to get it right by ensuring a thorough discussion. So I do think it is important, in all cases, to formally announce agreement and provide at least some period of time for further questions or objections. There could be two (or more) time periods depending on the level of controversy, length of discussion, number of people involved in the discussion, and how closely a new standard name follows existing patterns.

Also, though I am not finding this spelled out at the moment, I believe once a standard name is accepted users can start using it even before it is formally published. I think this alleviates at least some of the urgency to accept new standard names before a given update to the standard names table.

[roy-lowry]

One issue with long delays between acceptance and publication of Standard Names is data integration systems with automatic gatekeepers that will not allow data past until the Standard Names have been verified against the published list. I'm becoming aware of increasing importance being given to automated standard conformance verification in operational oceanographic systems.

Publication is now a regular process so one suggestion might be to have an announcement of what is to be published to allow final objections one or even two weeks before the publication. My concern is that there is a danger people who haven't been involved in the discussion will see the announcement, react without reading the thread and raise an objection that takes the discussion back to its beginning.

[davidhassell]

I, too, understood that an accepted standard name can be used straight away. I struggled to find this written down, bar a couple of comments on the old e-mail list from 2008 and 2012:

I can formally accept the names, which means that they will definitely go into the table on January 13th and in the meantime you will be free to start using them to write your data.

The usual procedure is to see if there are comments and objections on this email list. If there are, a discussion will no doubt take place. If not, after 2-3 weeks you can probably start using them on the assumption that they're going to be included in the standard_name list when it is next updated.

As @ethanrd suggests, I think it could work if we added the usual three week cooling off period before acceptance that we have for other parts of CF, and made it clearer in the standard name rules that the name can be used from that moment on. I'm guessing that the added complexity of choosing a time period in each case would not be needed.

Given that discussions often go on for much more three weeks, and can span one or more publications of new standard name versions (the cf-convention/vocabularies#74 discussion that got all this going lasted ~4 months) - is a three week cooling off period a genuine concern? It will either make no difference, or at worst it might mean that the name goes into the next-but-one release, that could be as little as only 4 weeks and 1 day away. Please say if my assumptions don't apply to real life.

I've just seen @roy-lowry's post come in - I share your concern about bringing accepted names back to everyone's attention at a later date. It's good to advertise the new names, but not to invite objections so soon after they have already been properly discussed.

[ethanrd]

While I understand the concerns about reopening the discussion, I think it is important (as an open and transparent community process) to advertise agreement, invite objection, and allow some time for any objection once the final form of a new standard name and description has been agreed.

This would ensure (and this is the reason I raised this issue) that everyone involved in the discussion is aware of the agreement and the timeline for acceptance.

Yes, it will also open the door for others to comment (a good thing) which may result in people reacting without careful review of what has already been discussed (not so good). However, rather than avoid that situation, I think we need to find ways to minimize and handle that situation.

[graybeal]

Funny, I always thought the 3-week rule applied to Standard Names. No wonder I was so confused at times.

In my experience, the best way to manage the possibility of 'late comments' is to state the 'final proposed change' as quickly and definitively as possible in the request handling process. If a request is likely uncontroversial (which I think at least one of Jonathan or Alison needs to support), then any of a few designated parties could describe the final change. Or someone could be paid to do so quickly, if that's a priority.

The posting of the formal notice has a way of focusing attention. I remember making the final call for ACDD comments after about 18 months of work, when a major player stepped in with a major objection—I can only conclude that the individual didn't take the prospects seriously until we got to that stage. So that's a risk. But CF (umm, parts of it) has always required the final proposal to survive the cooling off period, and I think that's appropriate given work cycles (deployments, vacations, grants and publications, …). I don't think a second announcement is a requirement. Though automatically (daily) updating on the web a standing list of pending items and finalization dates would be nice, if that's feasible someday.

[erget]

I'm with @ethanrd on this one - more visibility is a benefit in my view, and I could imagine that many, like @graybeal, have assumed that the 3-week rule applies to everything across CF. This is an intuitive thing to believe and it seems also in other areas that we're moving towards that as a kind of universal CF cycle length. In my view this would be a valuable addition to the Rules for Changing Standard Names page.

[ngalbraith]

If we're going to address this issue, which I support, could we also mention the situation where there is NO discussion on a proposed standard name? Are those discarded, or approved? How do we generate comments?

I have a couple of issues proposing new standard names I need for surface fluxes; direction_of_surface_downward_stress and wind_relative_to_surface_current. These have not garnered any comments in about 2 weeks, and I don't know if that's because there's some reason why they shouldn't be added, or because they're obviously OK. This is a little confounding!

[davidhassell]

Hi @ngalbraith,

I agree - new standard names are not "defects", so it follows that there should be some requirement for other contributions.

The rules for changing the conventions say "at least three contributors have expressed support for it, including at least two members of the conventions committee". Perhaps that's a bit much for standard names? but something well defined that insists that at least someone else agrees would be good.

We always have the standard names management eyes of Alison and Fran, of course.

[JonathanGregory]

If no-one else comments on a standard name proposal, I think that we should depend on the manager of the standard names (Alison) or a deputy (Fran) at some point to study the proposal and either state any concerns or state that it should be accepted as proposed. In the latter case it should then be accepted if no-one objects within some cooling-off period (such as three weeks). I wonder whether Alison and Fran can suggest on what timescale they could expect to comment on a standard name proposal - not just of this kind, but of any kind, since all proposals in the end depend upon their intervention to be concluded.

[roy-lowry]

Another possibility would be to give each member of the Standard Names committee responsibility for specific domains to ensure that some comment - at least a 'This looks OK' - is made. I try and do this for the oceanographic measurements domain but I considered Nan's latest requests outside of my comfort zone so kept quiet. I don't know how others feel but I wouldn't object to a gentle reminder e-mails from Alison or Fran to the Committee should it look like a proposal is being ignored for too long.

[ngalbraith]

Another possibility would be to give each member of the Standard Names committee responsibility for specific domains

I also try to comment on any SN proposal that is closely related to my area, but I'm not sure that we need to (or can) formalize this - in part because it's hard to identify domains in advance - would we create a list of domains, and have new SN proposals tagged with those?

Also, I'm not sure if this is the appropriate place to raise this, but it's closely related: we might want to update the README.md for the 'discuss' page, where new standard names are proposed, to tell people what they should do to 'close' discussion on a proposed name. Once again, I have a couple of names, wind relative to surface currents and direction_of_surface_downward_stress, that should probably move along to the next step in the process, but I'm not sure how to do that. Flag the issues somehow, for Alison or Fran?

I'm ready to use these names, but I'm not sure I can do that yet, without generating errors in the CF checkers.

[JonathanGregory]

If Alison or Fran could indicate a timescale on which they would routinely look at all proposals which are currently on discussion, in order to summarise the position, or to comment if no-one else has done so, that would be helpful.

[feggleton]

Hi all,

I have been considering this discussion for a while now and my opinions on it. I'm sure Alison will also give her own opinion on It shortly (one with far more experience in managing the standard names!). In my opinion, I would prefer a one-week cooling-off period. However, if the community is agreed that a three-week cooling-off period would be more beneficial then that is fine, we should go with that. What I don't want to happen is for things to become over-complicated, with certain timings having to be implemented in certain circumstances and Alison and I having to implement multiple statuses for each phase and setting reminders in my calendar for 1/2/3/4 weeks for every single ticket (if that makes sense e.g. cooling off period 3 weeks, acceptance period 1 week, list for update 2 weeks before etc). I also don't think that a list of what terms are being published 1/2 weeks before the standard name table is updated would work in practice. But these are just my opinions and I have only been managing standard names for just over a year so I am still learning. I do not have a specific date/time during the week when I work on/review standard names. My role in standard names is one part of my wider role working at CEDA and so I dedicate some portion of my time to standard names (reviewing issues, updating the cfeditor, reading up etc) but currently, I do not have a fixed routine/schedule to review proposals. As you can probably tell by my comments. Some weeks I have more time, some weeks I have less. I follow the GitHub repo, which sends all comments to my email inbox. I keep up with conversations by reading them as they come in (otherwise I would have lots to read!).

In relation to proposals with no comments, I try my best to get things moving and am aware that some tickets sit with no discussion for a while. I don't like to jump straight in as, in my opinion, people in the community should have time to get round to looking at the requests (they may be away, busy etc), as it is a community lead standard! I do really appreciate it when members of the committee and people with specific expertise comment, this is very helpful! In my view, the following comment already happens...(bar the 3 weeks cooling off period) we typically say if there are no comments after a week it can be accepted (once we are happy with it).

If no-one else comments on a standard name proposal, I think that we should depend on the manager of the standard names (Alison) or a deputy (Fran) at some point to study the proposal and either state any concerns or state that it should be accepted as proposed. In the latter case, it should then be accepted if no-one objects within some cooling-off period (such as three weeks). I wonder whether Alison and Fran can suggest on what timescale they could expect to comment on a standard name proposal - not just of this kind, but of any kind since all proposals in the end depend upon their intervention to be concluded.

Fran

[ngalbraith]

Thanks, Fran.

... Alison and I having to implement multiple statuses for each phase and setting reminders in my calendar for 1/2/3/4 weeks for every single ticket

I think the responsibility of watching the timing of a proposed addition to the SN table could rest with the proposer, not be added to Alison's and Fran's duties. The person or group making the request for a new name, in theory, is motivated to make sure the discussion moves along and to get the new name approved.

It would be helpful, though to know what a reasonable amount of time is for each step in the process.

[sadielbartholomew]

I was thinking about this a while back and maybe now (given there seems to be consensus emerging) is a good time to jump in with an idea, relating to the monitoring of whatever time period is chosen for the "cooling off". Regarding Fran's concern about the extra work it would necessitate for herself and others, and Nan's comment:

I think the responsibility of watching the timing of a proposed addition to the SN table could rest with the proposer, not be added to Alison's and Fran's duties

I don't think it needs to be done by either party (though if it did have to be one or the other I'd absolutely agree with @ngalbraith ) because to me management of the countdown process would be the perfect job for a dedicated bot which we could code up quite easily and quickly (happy to volunteer). @davidhassell and I have recently (in submitting a paper) encountered a great example recently of the 'whedon' editorial tool for the JOSS review and publication process, whereby a reviewer or editor can make a short comment to initiate automated tasks, for example see this listing here in the JOSS documentation. I think we could use something similar to help with the case at hand and indeed also other tasks relating to the review process. (Actually this ties in well with some standard names tooling work I begun under the guidance of Jonathan, Alison and David H though I haven't had time to do much on it for many months, but can get back on it in 2021, so this could come under that umbrella.)

Basically, for the issue at hand, I'm envisioning that when Alison or Fran are happy for acceptance, they make some short designated comment e.g. '@<botname' trigger cooling period' or similar which would trigger the bot to start up a process whereby it would:

• Immediately make some noticeable comment saying there is acceptance and the cooling-off time period of X weeks had begun.
• If there are no further comments, continue to post weekly to say there are X - 1 and then X - 2 etc. weeks left and eventually, a few days before the cooling off period ends post to give a final warning.
• Or, if there are in comments raised during the cooling off period, pause the countdown and tag the proposer so they are aware, then based on the outcome of further discussions it could be agreed and via some command activated to continue with the cooling-off countdown at the previous time point, else cancel it so the whole thing would need to be restarted at a later point when consensus was newly-attained.
• Once the cooling down period completes without any further discussion, tag Alison and/or Fran so they know to start the usual acceptance process.

With a bot doing the work above, neither Alison or Fran without having had to be involved with the extra complication of the management of the cooling off period, to alleviate Fran's very understandable concern about this proposal of a longer cooling-off period.

And the bot would have the same standing as any commenter, notably the only people who would be notified it had commented would be those already watching the post. So it wouldn't "advertise" the thread at risk of re-initiating discussion when discussion had already converged, as per David's concern:

It's good to advertise the new names, but not to invite objections so soon after they have already been properly discussed.

What do people think about this idea? (I'm not tied it BTW so feel free to turn it down, but I do think it could enable this proposal to go through without creating extra work for anyone long-term).

[feggleton]

@sadielbartholomew That is a great idea! I was thinking previously that there must be something we could do in GitHub to help with these things. Thank you so much for commenting and your suggestion. Really like that idea.

[erget]

@sadielbartholomew I agree, this is a really cool idea. I'd be really happy to see such an example if you have time to code it up!

[sadielbartholomew]

Glad to hear you like the idea Fran and Daniel.

On that note I've noticed there is already some sort of bot related to the CF Conventions, @cf-conventions-bot. Does anyone know what that is used for and who (or what code) controls it? Because if everyone is happy to go with the bot idea to manage the longer cooling-off period we could perhaps re-purpose that username for the bot. Creating another would be no problem, but I think it would be nicest to have a single bot to provide any and all automation utilities relating to the standard names or perhaps more generally the CF Conventions as a whole.

[erget]

@sadielbartholomew the bot belongs to the CF Information Management and Support Team. If you're interested, you're welcome to join the team - it's kind of like the Avengers, in that it's an elite group of superheroes who join forces at an ad-hoc and voluntary basis, with the time commitment being self-determined, in order to save the world. Depending on your definitions of "elite", "superheroes", and "save the world", you may disagree with portions of that statement, but I think you get the gist of it.

I agree that it makes sense to have the bot do these types of actions. I have access to the bot, but I've never used it before - I believe all it's doing currently is autogenerating the Conventions Documents when changes are merged to master.

I support the use of the bot to give reminders, which would also be useful for the main Conventions document, so that people don't have to remind themselves e.g. when the 3-week rule is up. If we do it though we'll need to offer some docs or training on how it works. Would you mind setting up a separate issue to pursue this?

[davidhassell]

I very much support the bot idea.

[sadielbartholomew]

@erget I would be delighted to join the highly mysterious but clearly quite important CFIMST, assuming the induction process is not too dangerous! Thanks for the invitation and for the further info on the existing bot.

Sure, I'm happy to set up an issue. I'll do that now. It is indeed a good idea to push this to a separate issue so as not to distract from the main item on the agenda here which is the comment cooling-off period.

So for this issue at hand, the main point to note stemming from my original comment https://github.com/cf-convention/vocabularies/issues/93, and those in response to it, and hence my final word here (at least on this note), is that any new or amended comment cooling-off period does not have to result in any significant increase in workload on those managing the agreement and acceptance process e.g. Alison and Fran, because we can automate the management of this particular aspect.

[ngalbraith]

Great idea - thanks for suggesting this. One concern I have is that, if the bot generates a lot of emails (I'm thinking 4 extra emails per issue, minimum?) members will be filtering out all except the 'final notice' ones, via their email apps.

the only people who would be notified it had commented would be those already watching the post

I'm not sure how this will work, in practice - not a github expert, obviously. To opt out of the notices for threads that are outside my area of interest, would I need to unwatch each of those threads via github, just not comment on them, or not view them? Currently, I only 'visit' issues that seem relevant to my work, and don't bother to unwatch the others.

There are 63 people 'watching' this particular issue - I guess I don't know what that number represents.

[sadielbartholomew]

One concern I have is that, if the bot generates a lot of emails (I'm thinking 4 extra emails per issue, minimum?) members will be filtering out all except the 'final notice' ones, via their email apps.

That's a very good point @ngalbraith. Even if automated a cooling-off period will still create noise that might be disruptive to anyone following an issue.

There are 63 people 'watching' this particular issue - I guess I don't know what that number represents.

For context to the point above, I think this is anyone who, assuming they haven't manually unsubscribed ('subscribe' == 'watch' interchangeably in GitHub terms it seems, and see here in the documentation if interested for further detail):

• has commented or been tagged here;
• is watching the whole repo either because they selected to or because they have push access to it and didn't specify not to watch repos they have such access for;
• has manually subscribed to the issue e.g. via some option in the 'Notifications' menu above.

I'm not sure how this will work, in practice - not a github expert, obviously.

I'm no expert there either, but thinking about how to mitigate the disruption for people who are already watching, there may be a way to disable any notifications when there are certain comments e.g. those by the bot (I can't find any immediate indication online as to whether that is possible so would need to look into it in more detail).

Though anyone can unsubscribe from notifications from an issue at any time, so one simpler potential means to avoid all watchers getting extra emails could be, when Alison or Fran indicate acceptance and start the cool-down timer, it is suggested that everyone unwatches the issue if they are already satisfied with it. Because I imagine it is unlikely there will be new objections, but if there are and this leads to new discussion that has to be ironed out, maybe the bot can re-tag by name anyone who had previously commented to notify them (tags give notifications to the person even if they are unsubscribed, assuming they haven't 'ignored' the issue or repo which blocks anything).

So I think there are some possibilities to avoid this pitfall that we could try if people would like that.

To opt out of the notices for threads that are outside my area of interest, would I need to unwatch each of those threads via github, just not comment on them, or not view them? Currently, I only 'visit' issues that seem relevant to my work, and don't bother to unwatch the others.

Looking at the GitHub documentation on subscription management, I think that your options for opting out on the GitHub notifications inbox side of things are (1) as you suggest to unwatch all of the issues you aren't interested in, or (2) do the inverse and unsubscribe from the whole repository but check-in regularly and watch the particular issues (only) that interest you, or (3) just ignore them in the inbox (this is what I do). Viewing issues shouldn't influence your watching or unwatching status.

If you are specifically referring to emails you get, it's a bit different as you can filter out emails via your email system, so there may be a solution for fine-grained filtering of notifications there.

If anyone has a nice solution for filtering out of CF Conventions issues that aren't of interest to them, it would be useful to hear! I think it must be a common problem (generally for work on GitHub) to get noise from all issues when really many or even most of us only want to keep up with a subset.

[taylor13]

I am glad that a way will be sought to minimize unimportant messages. I'm pretty sure I get every communication from the CF github repository, which I generally like because I can see what is going on. But I wouldn't like to get housekeeping items.

[ngalbraith]

I'd also like to suggest that we might have a tag - a milestone or a keyword - that could be used to indicate to Alison and Fran that the proposer (or another participant) thinks the discussion has ended and agreement has been reached. I just tagged @japamment on a couple of my standard name requests - not sure if that's the best way, but I wanted to indicate that I think the names are 'done' and ready to be added to the table.

[ethanrd]

Hi all - I’m unclear about the need for multiple messages about the cooling-off period for each new standard name issue. With proposals/issues on the conventions document, the moderator sends a message stating that consensus appears to have been reached. That message includes the date, 3-weeks out, when the proposal/issue/PR will be accepted if there is no further objection. Do the new standard name issues/proposals need something more than that? Or are the other notifications mentioned more reminders for behind-the-scenes actions by Alison and Fran (and/or the committees)?

For that type of reminder or “housekeeping” notifications, I think they should be sent to specific people or GH teams rather than directly to an issue. Too many “housekeeping” notifications and especially encouraging people to filter or “unwatch” certain notifications will, I fear, end up being another barrier to getting and keeping people engaged in CF.

I think there are some GitHub Actions that might support some of these “housekeeping” type notifications. I’ll add a comment to the “bot to automate ...” issue shortly.