Finish PR 5500 initiative signature gauge

[virgile-dev]

We've started the PR with Codegram but didn't manage to finish it properly. Here is a comment on the PR with the remaining tasks.

The idea is to make a PR on this branch with the remaining changes.

[ace]

Hi @virgile-dev. I think I need a bit more detail about the remaining tasks.

• Check that events issued from initiatives are working properly. Are there any specifics about what to check or which events may be affected?

• Check that the edge cases defined here work well. As far as I understand the linked doc, the main issue is having scopes below the level of detail in the authorization. In that case the signature is applied to all child scopes since we can't identify a specific one. Is that right?

• Display a warning in the admin if an initiative does not have a global scope and only global scope config flag is activated on the initiative type. Not sure I understand this use case. I mean, is it possible to get to that situation? If that flag is activated, is it possible to create an initiative without a global scope? Maybe we should enforce that? Do you have a sample of how to display the warning?

• Fix the initiative admin form when the scope is global. What's the issue there? Any kind of process error or just a visual fix?

• Apply design in mockup for signature gauges. Is this related to the bottom margin issue you mentioned in that PR or is there anything else to be done? Not sure how the 'see all' link (mentioned in https://github.com/OpenSourcePolitics/dev/issues/9) should work. Does it 'unfold' the rest of the data or is it a link to a separate page (which one?).

Thanks!

[virgile-dev]

Hello @ace Here are my answers :

Check that events issued from initiatives are working properly. Are there any specifics about what to check or which events may be affected?

• Milestone events are the one that we need to adapt (see this PR if you don't know what I'm talking about)
• We’d like to send notifications for child scope milestones too (same percentages : 25, 50, 75 and 100%). Notifications would go to authors, promotting committee, followers for all the milestones (25, 50, 75 and 100%). Admins should receive a notification only upon reaching 100%.

Check that the edge cases defined here work well. As far as I understand the linked doc, the main issue is having scopes below the level of detail in the authorization. In that case the signature is applied to all child scopes since we can't identify a specific one. Is that right?

• yeah and I think it’s taken care of already


Display a warning in the admin if an initiative does not have a global scope and only global scope config flag is activated on the initiative type. Not sure I understand this use case. I mean, is it possible to get to that situation? If that flag is activated, is it possible to create an initiative without a global scope? Maybe we should enforce that? Do you have a sample of how to display the warning?

• I’m guessing this is more of compatibility issue. If there were already initiatives and you activate the global scope only flag what happens. IMO if you created an initiative with the child scope only the child scope gauge should be display. When it has a global scope then you display the child scope jauge if « child scope signature » flag is enabled. 


Fix the initiative admin form when the scope is global. What's the issue there? Any kind of process error or just a visual fix?


• This is more a bad UX thing. The default value when creating a new scope on the initiative type is the global scope. When you click on the scope selector it is thus already selected and you can’t click on « select » you have to close the modal. Here is a screen capture of this behaviour

Apply design in mockup for signature gauges. Is this related to the bottom margin issue you mentioned in that PR or is there anything else to be done? Not sure how the 'see all' link (mentioned in #9) should work. Does it 'unfold' the rest of the data or is it a link to a separate page (which one?).

• we won’t take care of this in this first draft

[ace]

Thanks @virgile-dev Just one thing, do you have any example of how that warning in the admin should look like?

[virgile-dev]

@ace which warning are we talking about ? This one ? I'm not even sure it needs a warning as it should be compatible. Don't you.

I’m guessing this is more of compatibility issue. If there were already initiatives and you activate the global scope only flag what happens. IMO if you created an initiative with the child scope only the child scope gauge should be display. When it has a global scope then you display the child scope jauge if « child scope signature » flag is enabled. 


Warning display the same on the front end and in the back end.

[ace]

Yes, that one. Not sure if it's needed, to be honest. If it is OK to be in such a state, does the warning provide any help or anything to the admin? I tend to think it does not, so maybe it is not needed at all. So, if you are OK with that, I'll leave it for now.

[virgile-dev]

I'm ok with no warning at all @ace

[ace]

I'll be keeping Issues I find and whether I've fixed them in this comment, so they can be checked quickly.

[x] Migrations fail (code updating the existing votes is wrong) [x] Admin editing initiative with offline votes crashes, I18n translation issue (using scope as interpolation key) [x] Offline votes saved from admin form as string arrays instead of integers and with no total. Example: online_votes => {"1"=>1, "total"=>1}; offline_votes => {"1"=>["15"], "2"=>["6"], "4"=>["4"]} [x] Required votes just adds up all the scopes required votes. For a parent (1000 votes required) and two children (500 votes required each), that means 2000 votes are needed, with no further check on where they apply. [x] Total votes just adds up all votes for all scopes. For a parent (1000 votes) and two children (600 and 400 votes), the total is 2000, again, with no check on where they apply. (Note that the two examples will result in an accepted initiative, 2000 = 2000, even though one child is not fulfilled). [x] Breaks sorting by signatures count in the admin [x] Sorting by signatures count fail when nils (almost always?) [x] Migrations fail because one of them removes a column that is used in a later migration

[ace]

@virgile-dev while I write the doubts we talked about (seeing some code that clarifies some of them), one more thing I'd need to clarify.

* We’d like to send notifications for child scope milestones too (same percentages : 25, 50, 75 and 100%). Notifications would go to authors, promotting committee, followers for all the milestones (25, 50, 75 and 100%). Admins should receive a notification only upon reaching 100%.

This is the content we are sending right now for those milestones for the author and others:

        milestone_completed:
          affected_user:
            email_intro: Your initiative %{resource_title} has achieved the %{percentage}% of signatures!
            email_outro: You have received this notification because you are the author of the initiative %{resource_title}.
            email_subject: New milestone completed!
            notification_title: Your <a href="%{resource_path}">%{resource_title}</a> initiative has achieved the %{percentage}% of signatures.
          follower:
            email_intro: The initiative %{resource_title} has achieved the %{percentage}% of signatures!
            email_outro: You have received this notification because you are following %{resource_title}. You can stop receiving notifications following the previous link.
            email_subject: New milestone completed!
            notification_title: The <a href="%{resource_path}">%{resource_title}</a> initiative has achieved the %{percentage}% of signatures.

How should we adapt those texts for the child scopes? Keeping them unique may lead to confusions. Same question for the admin messages for 100% in the other PR (https://github.com/decidim/decidim/pull/6098/files#diff-76dc41a8aba2e3fa43b6dfee29b71177).

One other thing to give a thought, not needed for now. With the process as it is now, we send a 100% milestone notification (and admin notification) for the parent scope when that scope gets enough votes, but that doesn't guarantee that the initiative is completed, because it may lack votes in child scopes. Maybe that will be confusing for the users. In fact all percentages for the parent scope may be missleading.

[virgile-dev]

@ace thank you so much for your patience on this. Our back and forth are helping a lot. I agree that the milestones notifications are going to be a source of mis-understanding.

For now, I think that we should not modify how it works to keep it simple. We will adapt the locales so that it's less confusing for the user.

Moreover we are working on a spec for the AN that allows the admin to configure the milestone on the initiative type.

If we don't touch this, is there still much to do ?

[ace]

No problem. The only thing pending from the initial list is that UX issue on the admin when selecting the global scope. Then there's what you told me on our chat about the condition for the admin to be able to accept or reject an initiative depending on the votes. However, there are some things that I think are not working as expected. Let me try to comment them here.

How it is working now, given a parent and children scopes:

• Assigning votes.

  • If the auth matches one children, it creates a vote for the child and one for the parent. That seems OK.
  • If the auth matches the parent, only a vote for the parent is created. That's not what was being discussed on notion (third case from the bottom, the one with 4 comments). There it talks about creating a vote for all children when the auth is for the parent. I tend to think the actual code is more reasonable, if we can't determine the lower level, don't vote for it. The potential problem in this case is never reaching the required votes for the children.
  • If the auth is child level, but matches none of the children (the case we discussed online, gaps in the children vs. the parent), the vote goes only for the parent. That seems reasonable to me, but would probably need some discussion.
  • Unvote removes all user votes, so that's OK in any case.
  • Offline votes in the admin have no validation, any number combination can be entered. It also does no calculation, contrary to the online votes, that area assigned to child and parent, here you have to specify numbers for each scope, if you leave the parent as 0, it will store 0, not the sum of the children votes. There's no help text explaining that.

• Counting votes.

  • Online votes keep a value for each scope and a 'total' value adding all of them. That total then is (roughly) double the number of votes (each child vote generates a parent vote).
  • Offline votes don't create that 'total' value, leading to always counting 0 offline votes. Also, the scopes votes are saved as arrays instead of numbers (this prevents adding online and offline votes for a scope as just two numbers).
  • Total votes just adds up online and offline totals. Ignoring the previous offline issue, it's again (roughly) double the real votes count.
  • Required votes just adds up all the scopes required votes. I think this is wrong, isn't it? First, that's not a single number, there should be several, one for each scope. Second, if there's one number to be considered, it would be the required votes for the parent scope, with then the children ones as extra conditions.
  • Thresholds just use total and required numbers above, since they are not right, I believe the thresholds are mostly wrong.
  • Goal reached. It just compares total and required votes, it's wrong since those numbers are wrong. It does no check on the individual scopes, so having enough votes from one child to complete the parent gets the goal reached.
  • The rake task that moves initiatives into accepted or rejected status is using that goal reached count, so it can accept initiatives not matching all the requirements.

My issues comment above has this problems listed. A review of my findings would be great, just in case I misunderstood anything.

I already fixed a couple of them that resulted in errors and prevented me from even working on this. Should I go on fixing those issues or are there other priorities?

Mostly unrelated thing I just found while investigating how all this works, that feels strange to me. How votes are displayed on the initiative page. Only online votes are shown and counted for the progress bars. But for the 'need more votes / most popular initiative' text under the counter, the offline votes are also taken into account. So you can potentially see an initiative with not enough (online) votes and a not full progress bar, but 'most popular initiative'. Just to give some thought.

[ace]

One more thing, I just realized the changes in this PR break the sorting by signatures count we did in one of my first issues.

[virgile-dev]

@ace We gave it some thought with Alain :

First of all, we need to think separately of the signatures and the totals.

Example of setting for an initiative type

• Belgium —> global scope : 9000 signatures required
  • Brusells —> Scope (child of global) : 3000 signatures required
  • Wallonia —> Scope (child of global) : 3000 signatures required
  • Flanders —> Scope (child of global) : 3000 signatures required

In the application the following scope structure is created :

• Region (ex Flanders)
  • Province (Ex Anvers)
    • Arrondissement ( Ex : Malines)
      • Commune (Ex : Berlaar) —> This is the one associated to the user in its authorization

The signatures are saved with the scope that is associated to the users. This is one thing.

Ex with structure above : signature saves "Berlaar".

About totals (counting the votes)

The totals (the parent and the child ones) are incremented every time a signature happens. Ex with structure above, user signs with "Berlaar", the parent initiative scopes have their totals incremented : Flanders + 1 / Global scope +1. This way the jauge have accurate totals. This is important performance wise. We save totals so that they are easily retrieved and don't have to be computed every time the page is displayed.

About edge cases

In this first version of the feature we are going to assume that the configuration has been done properly at the scope level /admin/scopes and at the initiative type scopes level. The config of the initiative should ensure that the totals are consistent : the sum of child scope totals should amount to the parent total.

To help the admins we can add help texts bellow the config flags associated with this feature and we will provide a read me with details and examples.

• [ ] Config flag "Allow only global scope initiatives" : "If you enable this make sure you configured your scopes and initiative type scopes properly, see this [link](doc in decidim github repo) to understand how. This is usually activated with "Enable child scope signatures". By enabling this scope selection will be skipped in initiative creation wizard"

• [ ] Add an help text bellow the config flags "Enable child scope signatures" : "This config flag doesn't support offline votes, it enables sub-jauges and works with an authorization handler that associate associates a scope to the user, make sure you select that authorization bellow. For more info on how this configuration works see [this link](doc in decidim github repo)".

• If the auth matches the parent, only a vote for the parent is created. The configuration of the initiative type, the scopes created on the organisation and the way we associate a scope to the user in the authorization should ensure it doesn't happen. If a user doesn't have the right scope (meaning it doesn't match any of the parent but the global scope) to sign we should use the permission system to ensure he can't vote and displaying a disabled sign button. In the event the user is still able to vote then it's the normal behaviour that the vote is counted on the parent level.

• If the auth is child level, but matches none of the children (the case we discussed online, gaps in the children vs. the parent), the vote goes only for the parent If the scope associated to the user doesn't related to any of the child scope. Before clicking on the sign button the permission system should detect that the user can't vote, disable the sign button and display a message saying the user isn't authorized to vote because based on the info of his profile we cannot identify for which scope he is supposed to vote for.

About offline votes

For now, this feature doesn't support offline votes.

About validation rules (totals) For now notifications about milestones should only be sent for parent level.

For status changes rejected and accepted :

The button to apply them in the admin should be visible if the goal (number of signatures required) are reached for the child scopes and the parent scope defined in the initiative type scopes

[ace]

Fixed the relevant issues and the calculations. https://github.com/decidim/decidim/pull/6143