Use the internal census system

[andreslucena]

Ref: VOC-SEC-02

Currently, for setting up the module, and administrator need to upload a CSV file with an email and a token as a census.

On Decidim there's already an API for allowing the participants to make some actions inside this particular component, the Authorizations API. As the documentation says:

There are several use cases for this, such as

• Sending an SMS code to users to verify that their have a valid cellphone
• Allowing users to upload a photo or scanned image of their identity document
• Sending users a code through postal code
• Allowing users to go to a physical office and check their documentation
• Checking some information through other systems (as a Municipal Census in the case of Municipalities, Cities or Towns)
• Having a list of valid users emails

As an example of a real world kind of voting where this was used, we can find processes like the Voting for the Sindicatura de Greuges de Barcelona 2021-2026 in Barcelona. It was used with the Municipal Census verification API, an external SOAP service that checks if someone resides in the city of Barcelona.

Adapt the voting module to make use of the Decidim’s internal census system to create Vocdoni elections

There should be two options for the "Census type":

1. CSV file, with the current behavior: uploading a CSV file, sending it to Vocdoni API when setting up the election, asking for this token when accessing the voting booth with or without a participant account.
2. Decidim permissions / verification systems: where an admin can select a verification system supported by Decidim. It'll work with the Decidim APIs for that (such as asking for this kind of verification if the user isn't verified, or not asking if there's any of the selected verifications that the user fulfills.) In the case that there isn't any "Verifications system" selected, then it should ask for at least a registered participant account.

Mockups

CSV file

Verifications systems

Acceptance criteria

• [x] Given that I'm an administrator, When I create a new Election, Then I have two options in the "Permissions to vote" section:
  • "CSV file"
  • "Decidim permissions / verification systems"
• [x] Given that I'm an administrator, When I create a new Election, And select the option "CSV file" in the "Permissions to vote" section Then it shows the "Upload a new census with a CSV file" card And it works as now
• [x] Given that I'm an administrator, When I create a new Election, And select the option "Decidim permissions / verification systems" in the "Permissions to vote" section Then it shows the permissions that are defined in that Decidim instance
• [x] Given that I'm an visitor, When I go to an Election that's configured with the "Decidim permissions / verification systems", Then the platform asks for a registered participant account (login modal)
• [x] Given that I'm a participant, When I go to an Election that's configured with the "Decidim permissions / verification systems" and there isn't any selected Permission, Then I can vote correctly
• [x] Given that I'm a participant, When I go to an Election that's configured with the "Decidim permissions / verification systems" and there is a selected Permission, Then I need to have this verification fulfilled to vote correctly

[andreslucena]

This is the issue with a high grade of difficult after our meeting. Some pointers that the Vocdoni team shared with us:

1. The technical implementation should be based on the CSP feature of the SDK
2. API definition documentation
3. This is the library that they're using for doing it on TypeScript: blindsecp256k1-js. We should study Vocdoni's repositories to find how they use it after we understand the process

[paarals]

Andres I assign this issue to you as you said that is in your roof. Let us know if you need anything! @microstudi

[andreslucena]

Adding some notes from our last meeting regarding the implementation

Technical implementation

We discarded the blind CSP option as we think it maybe overkill at the moment. Another reason to prefer the dynamicCensus approach is that we can provide more transparency to the whole process, as anyone can check the changes made in the Census in real time.

In the SDK this will use the

dynamicCensus: true, // if the census can be changed

From the electionType configuration. This means that in the Decidim side we need a callback so that each time a new user verifies agains the Decidim application it gets sent to the Vocdoni API.

This callback would be using ActiveSupport::Notifications or whatever Rails API that we see better. Probably for the batch process we need to use ActiveJob and a rake task with some documentation on how to configure (via Cron as explained in the Decidim official docs).

For performance reasons and to not make too many requests to the Vocdoni API in cases where potentially dozens of participants are verifying at the same time, we will queue all the requests and send them on a batch every minute. That means that a participant needs to wait for a maximum of 70 seconds (1 minute + 10 seconds for the Vochain to confirm the operation). We need to show a spinner telling the participant that we're adding them to the census in this period of time.

Some snippets of code regarding the implementation at the SDK level:

1. Create a census with the SDK census service:

const numVotes = 10;
const service = new CensusService({ url: URL, chunk_size: CENSUS_CHUNK_SIZE });
const participants: Wallet[] = [...new Array(numVotes)].map(() => Wallet.createRandom());

const census = await service.create(CensusType.WEIGHTED);

await service.add(
  census.id,
  participants.map((p) => ({ key: p.address, weight: BigInt(1) }))
);

await service.publish(census.id);

2. With this in the census object you would have something similar to { id: string; auth: string }. The census.id is the census identifier (as usual) and the census.auth is a token that proves that you're the census owner (so you can modify it). We need to save that auth string in our database.

3. Now, later, when we want to modify the census, we need to do:

const service = new CensusService({ url: URL, chunk_size: CENSUS_CHUNK_SIZE, auth: { identifier: census.auth } });
await service.add(census.id, [{ key: Wallet.createRandom().address, weight: BigInt(1) }]);
const info = await service.publish(census.id);

4. Now that we've used the id and auth from the census generated before and we've added a new person, when publishing it, we have the following information in the info object: { censusID: string; uri: string }

5. Finally, you need to change the election in the census:

await client.changeElectionCensus(electionId, info.censusID, info.uri);

[paarals]

We detected one bug @antopalidi.
If I am added later with the census update, I receive a message that I cannot vote, as I have already voted. This is not correct, as this is the first time I have accessed this screen. To reproduce: 1- Create a vocdoni election 2- Census by verifier (in this case, organisation census) 3- The user is not verified before making the election. 4- Voting starts, the user is verified. 5- Update the census dynamically (manually) 6- You get this error

Also @carolromero will make some mock up for texts

[carolromero]

@antopalidi I just went through the whole voting flow with the internal census system, and you've done a great job!! :tada:

Here are some suggestions to improve some of the flow steps:

Setup the census of the election

• [x] When configuring the census, I'd change the underlined strings for a better understanding of the task to be performed. Also, if selecting the internal census, move the "Save census" button outside the authorization form, as it could give the impression that an authorization must be selected in order to use the Decidim internal census:

External census

Internal census

• [x] Given the different possible combinations in the census, I'd show the selected option in the confirmation message:

• [x] Also on the dashboard, to minimize errors

Voting using the internal census

Scenario 1: Election without additional authorization required:

• [x] Show this message (displayed for a few seconds)

Scenario 2: Election with additional authorization required. The participant has not validated the authorization: - [ ] Once the participant validates the authorization, can we redirect her to vote? Or is this a general bug of the authorizations I should report?

Scenario 3: Election with additional authorization required. The participant has validated the authorization:

• [x] Show this message (displayed for a few seconds)

Scenario 4: Election with additional authorization required. The participant has validated the authorization but the census is not updated:

• [ ] Show this message. I am not convinced by the solution and I don't know if it is technically feasible either, but it is the only thing I can think of to tell the participant that she does not need to perform any more tasks. Overall, I would NOT recommend this dynamic census configuration option

Updating the internal census while the election is running

• [x] Again, here we remind the admin what type of census and authorizations they have selected. And for the time being, until we better narrow down the cost of this transaction I would leave the sentence about credits. In short, it would look like this:

• [x] Clicking on "Update census now" is a bit confusing because the success message appears immediately, but the operation is not yet complete. Could we display the success message once the census has actually been updated?

Clicked on "update census now"

After a while

I think with this we should have this feature ready! Please let me know if you have any questions 😊.

[carolromero]

@antopalidi I almost forgot! Regarding the scenario of using an authorization without records at the time of setting up the election (when technical voter record should be created), I have not been able to reproduce it. Selecting for example the organization's census and making sure it is empty, it tells me that there are 6 records when setting up the election:

And, even ignoring this, when I get to setup the election I get an error :thinking::