Feat/add mosparo spam protection

bellisk commented 9 months ago

To run the mosparo server locally: set it up with Docker Compose, as documented here: https://documentation.mosparo.io/docs/installation/install/docker#use-docker-compose

Go to http://localhost:8080, create an account there, and then create a project for OGDCH.

Get the values MOSPARO_PROJECT_ID, PUBLIC_KEY and PRIVATE_KEY from the project and put them into the code in this PR in the appropriate places.

Run CKAN locally. You should see the mosparo box in the dataset subscribe form on the dataset page (you can also set the box to be invisible, in the project settings on mosparo).

Current status:

the mosparo box on the form works OK and can already prevent spam emails (test this by creating a rule in the mosparo project that a certain email address has a spam rating of 5, and then try to subscribe with that email address)
we still need to verify submissions that get through to the CKAN backend: see https://documentation.mosparo.io/docs/integration/custom/#performing-verification for details on this. There is code for this in the method OgdchSubscribeController.validate_and_signup, but it's not currently working - there are a lot of steps where you hash things and combine hashes, so I probably got something wrong there. This is the response I get currently: https://github.com/mosparo/documentation/blob/master/docs/8_api/4_troubleshooting.md#api-error-message-request-invalid
we can't use the mosparo Python library https://github.com/mosparo/python-api-client because it's Python 3 only, but it is useful for checking how they do the verification there

zepich commented 9 months ago

Hi @bellisk

I'm the developer of mosparo, and I saw your pull request and your description that only some things are working right now.

I know the verification process can be complex, and our documentation is not the best right now. A better one with more details will be published on the 11th of January, together with version v1.1 of mosparo. In this release, we've also added an API debug mode, which will give you a better explanation of the issue.

I've had a look at your code and found these possible issues:

1. A small typo

I guess it's just a typo in the committed code, but you wrote PUBLIC_KEY in the variable private_key and vice-versa (https://github.com/opendata-swiss/ckanext-switzerland-ng/pull/310/files#diff-6dbd191ea43833452f724ebcfb7d34956b52db2f1cd23a4355261d5ad8e20e6cR25). If you accidentally switch these two values, it can explain the error message you received.

2. JSON-encoded strings

Python encodes the JSON in a more "beautiful" form with spaces after the colon and the comma, while PHP does not do that. Because of this difference, the signature Python generates differs from the signature PHP creates to verify the request. This results in the Request invalid response.

PHP payload for the request signature

/api/v1/verification/verify{"submitToken":"test123","validationSignature":"4f403fed9c3fcd2fd813905962bc74cf25d333abe7be1d95dd6316f2c9c45260","formSignature":"b50f89b8d2fff64dcb26c4e4850ea0e3d06492d851590a79e6ded864738d7c70","formData":{"email":"test@example.com"}}

Python payload for the request signature

/api/v1/verification/verify{"submitToken": "test123", "validationSignature": "4f403fed9c3fcd2fd813905962bc74cf25d333abe7be1d95dd6316f2c9c45260", "formSignature": "b50f89b8d2fff64dcb26c4e4850ea0e3d06492d851590a79e6ded864738d7c70", "formData": {"email": "test@example.com"}}

To fix this, you can add the separators like this:

request_signature = hmac.new(
    private_key,
    api_endpoint + json.dumps(request_data, separators=(',', ':')),
    sha256
).hexdigest()

Of course, this is a problem for all JSON-encoded strings you created for the verification process, so you have to add the separators' definition in all the json.dumps.

3. The form data dict

As soon as the API authenticated your request, there is a third issue in your code: You put all data in the form_data dict, but since your form contains only the email field, only the value of this field should be added to the form data dict. Additionally, you have to generate the SHA256 hash of the value. We only receive the hash of the value in the verification process to minimize the data that is transferred from one host to the other.

hash_obj = sha256()
hash_obj.update(email)
email_hash = hash_obj.hexdigest()
form_data = {
    "email": email_hash,
}

Please let me know if this explanation helps you to solve the issue. Please let me know if the problem is not resolved or if you need more information.

Kind regards,

zepich

bellisk commented 9 months ago

Hey @zepich, thank you so much for the helpful comment! I will try again using your tips and let you know if I'm still having problems. :smile:

opendata-swiss / ckanext-switzerland-ng