TheLastProject / PyEssent

A wrapper around Essent's API
Apache License 2.0
4 stars 4 forks source link

Does this api also work for Belgian customers of essent? #1

Open davidclaeysquinones opened 5 years ago

davidclaeysquinones commented 5 years ago

I'm wondering if I could use this project to gather data from my energy supplier since I've also Essent at home. The only difference is I live in belgium :p

TheLastProject commented 5 years ago

I honestly have no clue, I didn't even know Essent was in Belgium too!

Could you try changing API_BASE from https://api.essent.nl/ to https://api.essent.be/ in https://github.com/TheLastProject/PyEssent/blob/master/pyessent/pyessent.py#L9.

Then follow the instructions in the README and report back with the output (censoring possibly private info)?

davidclaeysquinones commented 5 years ago

I'm afraid that didn't work.

The following json is send to https://sso.essent.be/am/json/authenticate when you log in : {"authId":"eyAidHlwIjogIkpXVCIsICJhbGciOiAiSFMyNTYiIH0.eyAib3RrIjogIjVkZHE5MzdxYm5rbGpqOGthMmdibmlrYm01IiwgInJlYWxtIjogIm89ZXNzZW50YmUsb3U9c2VydmljZXMsZGM9ZXNzZW50LGRjPWl3ZWxjb21lLGRjPWNvbSIsICJzZXNzaW9uSWQiOiAiQVFJQzV3TTJMWTRTZmN3eGhuVDFYXzh2dXJ4UjNMdFFqRTlJMHZ2bDBqR1Y2eU0uKkFBSlRTUUFDTURJQUFsTkxBQk15TkRVNU1UVTBPVE0yTkRFMU9ERTNOVEF5QUFKVE1RQUNNREUuKiIgfQ.MUdNQf-WEXQjAfAPvCde92lbXUSolHKcLKztPgNWFQg","template":"","stage":"LDAP1","header":"Sign in to OpenAM","callbacks":[{"type":"NameCallback","output":[{"name":"prompt","value":"User Name:"}],"input":[{"name":"IDToken1","value":"email"}]},{"type":"PasswordCallback","output":[{"name":"prompt","value":"Password:"}],"input":[{"name":"IDToken2","value":"password"}]}]}

I wrote the following test script :

from pyessent import PyEssent

print('Hello World!')
essent = PyEssent(username='email', password='password')
for ean in essent.get_EANs():
    essent.read_meter(ean)

And when executing I get the following output :

File "test.py", line 4, in <module>
    essent = PyEssent(username='***', password='***')
  File "C:\Users\DavidClaeys\source\repos\PyEssent\PyEssent\pyessent\pyessent.py", line 114, in __init__
    PyEssent.User.authenticate_user(username, password)
  File "C:\Users\DavidClaeys\source\repos\PyEssent\PyEssent\pyessent\pyessent.py", line 105, in authenticate_user
    data=request_xml.format(username, password, str(get_contracts).lower()))
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python27\lib\site-packages\requests\adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='sso.essent.be', port=443): Max retries exceeded with url: /am/json/authenticate/selfservice/user/authenticateUser (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)'),))
TheLastProject commented 5 years ago

Hmm, interesting. I wonder if the API is the same after login, but login already being JSON instead of XML doesn't seem hopeful.

If you manage to get login to work for the Belgium API it may be worth seeing if the rest of the API is similar to the Dutch one. Or just browse around the portal with your browser and see if that does XML requests in the network tab of developer tools.

davidclaeysquinones commented 5 years ago

The Belgian version seemse to use json instead of xml. It's probably another API. Is there a chance the library can be expanded in order to work with both API's ? I haven't got to much experience in python but I'm willing to help out.

https://prod.nova.essent.be/nova-crm/Api/V8_Custom/List/ConditionalMessages :

{
    "status": 200,
    "data": {
        "settings": {
            "title": "We hebben nieuws voor jou",
            "displayFooter": false,
            "actionData": {
                "parentId": "E7B1BEA7-E8A0-4FAD-9991-84853CF74A4F"
            },
            "responsive": false
        },
        "topBar": false,
        "headers": [
            {
                "label": "",
                "colSize": null,
                "cellType": "list_action_cell",
                "cellLines": {
                    "line1": false,
                    "line1CsvHeader": "",
                    "line2": false,
                    "line2CsvHeader": "",
                    "line3": false,
                    "line3CsvHeader": ""
                }
            },
            {
                "label": "",
                "colSize": null,
                "cellType": "list_simple_two_liner_cell",
                "cellLines": {
                    "line1": true,
                    "line1CsvHeader": "",
                    "line2": false,
                    "line2CsvHeader": "",
                    "line3": false,
                    "line3CsvHeader": ""
                }
            }
        ],
        "rows": [
            {
                "id": "e88c0e28-62a4-eb9d-4b9a-59b6978215b4",
                "cells": [
                    {
                        "type": "list_action_cell",
                        "class": "cell__action",
                        "options": {
                            "action": {
                                "id": "navigate_to_myessent_invoices",
                                "recordId": "E7B1BEA7-E8A0-4FAD-9991-84853CF74A4F",
                                "recordType": "Accounts",
                                "listKey": "ConditionalMessages"
                            },
                            "label": "",
                            "icon": "icon-credit",
                            "clickable": true,
                            "params": [],
                            "linkTo": ""
                        },
                        "cellLines": {}
                    },
                    {
                        "type": "list_simple_two_liner_cell",
                        "class": "cell__text",
                        "options": {
                            "line1": "Zou het kunnen dat je nog (een) openstaande factu(u)r(en) hebt? Kijk even je betalingen na.",
                            "line2": "",
                            "link": null,
                            "icon": null,
                            "linkTo": "",
                            "params": []
                        },
                        "cellLines": {
                            "line1": true,
                            "line2": false,
                            "line3": false
                        }
                    }
                ],
                "createDate": "0.69583200 1558505564",
                "rowData": [],
                "sortByOrder": "desc",
                "sortBy": {
                    "date": "2017-09-11 16:03:18.000000",
                    "timezone_type": 3,
                    "timezone": "Europe/Brussels"
                }
            },
            {
                "id": "9475edaa-2266-05bb-aabf-5aa645e81171",
                "cells": [
                    {
                        "type": "list_action_cell",
                        "class": "cell__action",
                        "options": {
                            "action": {
                                "id": "navigate_to_my_essent_update_communication_preferences_b2c",
                                "recordId": "E7B1BEA7-E8A0-4FAD-9991-84853CF74A4F",
                                "recordType": "Accounts",
                                "listKey": "ConditionalMessages"
                            },
                            "label": "",
                            "icon": "icon-marketing",
                            "clickable": true,
                            "params": [],
                            "linkTo": ""
                        },
                        "cellLines": {}
                    },
                    {
                        "type": "list_simple_two_liner_cell",
                        "class": "cell__text",
                        "options": {
                            "line1": "Mis geen enkel klantenvoordeel! Schrijf je in voor de acties van essent.be.",
                            "line2": "",
                            "link": null,
                            "icon": null,
                            "linkTo": "",
                            "params": []
                        },
                        "cellLines": {
                            "line1": true,
                            "line2": false,
                            "line3": false
                        }
                    }
                ],
                "createDate": "0.69600300 1558505564",
                "rowData": [],
                "sortByOrder": "desc",
                "sortBy": {
                    "date": "2018-03-12 10:18:41.000000",
                    "timezone_type": 3,
                    "timezone": "Europe/Brussels"
                }
            }
        ],
        "fileName": null,
        "pagination": {
            "page": 1,
            "size": 5,
            "sortBy": null,
            "total": 2,
            "pages": 1
        },
        "async": false
    },
    "message": "SUCCESS"
}

https://prod.nova.essent.be/nova-crm/Api/V8_Custom/List/My%20Essent%20-%20UnpaidInvoicesOnAccount_NEW

{
    "status": 200,
    "data": {
        "settings": {
            "title": "Mijn openstaande facturen",
            "displayFooter": true,
            "actionData": {
                "parentId": "E7B1BEA7-E8A0-4FAD-9991-84853CF74A4F"
            },
            "responsive": false
        },
        "topBar": false,
        "headers": [
            {
                "label": "Datum",
                "colSize": null,
                "cellType": "list_simple_two_liner_cell",
                "cellLines": {
                    "line1": true,
                    "line1CsvHeader": "Datum",
                    "line2": false,
                    "line2CsvHeader": "Datum",
                    "line3": false,
                    "line3CsvHeader": "Datum"
                }
            },
            {
                "label": "Bedrag",
                "colSize": null,
                "cellType": "list_link_bold_top_two_liner_cell",
                "cellLines": {
                    "line1": true,
                    "line1CsvHeader": "Bedrag",
                    "line2": true,
                    "line2CsvHeader": "Bedrag",
                    "line3": false,
                    "line3CsvHeader": "Bedrag"
                }
            },
            {
                "label": "Status",
                "colSize": null,
                "cellType": "list_simple_two_liner_cell",
                "cellLines": {
                    "line1": true,
                    "line1CsvHeader": "Status",
                    "line2": false,
                    "line2CsvHeader": "Status",
                    "line3": false,
                    "line3CsvHeader": "Status"
                }
            },
            {
                "label": "Type",
                "colSize": null,
                "cellType": "list_simple_two_liner_cell",
                "cellLines": {
                    "line1": true,
                    "line1CsvHeader": "Type",
                    "line2": false,
                    "line2CsvHeader": "Type",
                    "line3": false,
                    "line3CsvHeader": "Type"
                }
            },
            {
                "label": "Vervaldatum",
                "colSize": null,
                "cellType": "list_link_bold_top_two_liner_cell",
                "cellLines": {
                    "line1": true,
                    "line1CsvHeader": "Vervaldatum",
                    "line2": true,
                    "line2CsvHeader": "Vervaldatum",
                    "line3": false,
                    "line3CsvHeader": "Vervaldatum"
                }
            },
            {
                "label": "",
                "colSize": null,
                "cellType": "list_link_pink_down_two_liner_cell",
                "cellLines": {
                    "line1": false,
                    "line1CsvHeader": "",
                    "line2": true,
                    "line2CsvHeader": "",
                    "line3": false,
                    "line3CsvHeader": ""
                }
            }
        ],
        "rows": [
            {
                "id": "VKA1901639513",
                "cells": [
                    {
                        "type": "list_simple_two_liner_cell",
                        "class": "cell__text",
                        "options": {
                            "line1": "15-05-2019",
                            "line2": "",
                            "link": null,
                            "icon": null,
                            "linkTo": "",
                            "params": []
                        },
                        "cellLines": {
                            "line1": true,
                            "line2": false,
                            "line3": false
                        }
                    },
                    {
                        "type": "list_link_bold_top_two_liner_cell",
                        "class": "cell__text",
                        "options": {
                            "line1": "€ 103,32",
                            "line2": "",
                            "link": null,
                            "icon": null,
                            "linkTo": "",
                            "params": []
                        },
                        "cellLines": {
                            "line1": true,
                            "line2": true,
                            "line3": false
                        }
                    },
                    {
                        "type": "list_simple_two_liner_cell",
                        "class": "cell__text",
                        "options": {
                            "line1": "€ 103,32",
                            "line2": "",
                            "link": null,
                            "icon": null,
                            "linkTo": "",
                            "params": []
                        },
                        "cellLines": {
                            "line1": true,
                            "line2": false,
                            "line3": false
                        }
                    },
                    {
                        "type": "list_simple_two_liner_cell",
                        "class": "cell__text",
                        "options": {
                            "line1": "Voorschot",
                            "line2": "",
                            "link": null,
                            "icon": null,
                            "linkTo": "",
                            "params": []
                        },
                        "cellLines": {
                            "line1": true,
                            "line2": false,
                            "line3": false
                        }
                    },
                    {
                        "type": "list_link_bold_top_two_liner_cell",
                        "class": "cell__text",
                        "options": {
                            "line1": "03-06-2019",
                            "line2": "",
                            "link": null,
                            "icon": null,
                            "linkTo": "",
                            "params": []
                        },
                        "cellLines": {
                            "line1": true,
                            "line2": true,
                            "line3": false
                        }
                    },
                    {
                        "type": "list_link_pink_down_two_liner_cell",
                        "class": "cell__text",
                        "options": {
                            "line1": "",
                            "line2": "Download",
                            "link": "********",
                            "icon": null,
                            "linkTo": "",
                            "params": []
                        },
                        "cellLines": {
                            "line1": false,
                            "line2": true,
                            "line3": false
                        }
                    }
                ],
                "createDate": "0.02386500 1558505565",
                "rowData": [],
                "sortByOrder": "desc",
                "sortBy": null
            }
        ],
        "fileName": null,
        "pagination": {
            "page": 1,
            "size": 10,
            "sortBy": null,
            "total": 1,
            "pages": 1
        },
        "async": false
    },
    "message": "SUCCESS"
}

https://prod.nova.essent.be/nova-crm/Api/V8_Custom/List/MyEssent_ActivePackagesOnAccount

{
    "status": 200,
    "data": {
        "settings": {
            "title": "Mijn pakketten",
            "displayFooter": false,
            "actionData": {
                "parentId": "E7B1BEA7-E8A0-4FAD-9991-84853CF74A4F"
            },
            "responsive": false
        },
        "topBar": false,
        "headers": [
            {
                "label": "",
                "colSize": null,
                "cellType": "list_icon_text_cell",
                "cellLines": {
                    "line1": false,
                    "line1CsvHeader": "",
                    "line2": false,
                    "line2CsvHeader": "",
                    "line3": false,
                    "line3CsvHeader": ""
                }
            },
            {
                "label": "",
                "colSize": null,
                "cellType": "list_simple_two_liner_cell",
                "cellLines": {
                    "line1": true,
                    "line1CsvHeader": "",
                    "line2": false,
                    "line2CsvHeader": "",
                    "line3": false,
                    "line3CsvHeader": ""
                }
            },
            {
                "label": "",
                "colSize": null,
                "cellType": "list_simple_two_liner_cell",
                "cellLines": {
                    "line1": true,
                    "line1CsvHeader": "",
                    "line2": false,
                    "line2CsvHeader": "",
                    "line3": false,
                    "line3CsvHeader": ""
                }
            },
            {
                "label": "",
                "colSize": null,
                "cellType": "list_link_bold_top_two_liner_cell",
                "cellLines": {
                    "line1": true,
                    "line1CsvHeader": "",
                    "line2": false,
                    "line2CsvHeader": "",
                    "line3": false,
                    "line3CsvHeader": ""
                }
            }
        ],
        "rows": [
            {
                "id": "c29b3a54-5215-4f73-9911-778a884bc4cc",
                "cells": [
                    {
                        "type": "list_icon_text_cell",
                        "class": "cell__text",
                        "options": {
                            "cssClasses": "icon-opportunity",
                            "text": "",
                            "params": [],
                            "linkTo": ""
                        },
                        "cellLines": {}
                    },
                    {
                        "type": "list_simple_two_liner_cell",
                        "class": "cell__text",
                        "options": {
                            "line1": "MIG_ONLINE_B2C",
                            "line2": "",
                            "link": null,
                            "icon": null,
                            "linkTo": "",
                            "params": []
                        },
                        "cellLines": {
                            "line1": true,
                            "line2": false,
                            "line3": false
                        }
                    },
                    {
                        "type": "list_simple_two_liner_cell",
                        "class": "cell__text",
                        "options": {
                            "line1": "Address number  , ZIP City",
                            "line2": "",
                            "link": null,
                            "icon": null,
                            "linkTo": "",
                            "params": []
                        },
                        "cellLines": {
                            "line1": true,
                            "line2": false,
                            "line3": false
                        }
                    },
                    {
                        "type": "list_link_bold_top_two_liner_cell",
                        "class": "cell__text",
                        "options": {
                            "line1": "Producten bekijken",
                            "line2": "",
                            "link": null,
                            "icon": null,
                            "linkTo": "my-essent",
                            "params": {
                                "myEssentId": "my_essent_products",
                                "recordType": "Accounts",
                                "recordId": "E7B1BEA7-E8A0-4FAD-9991-84853CF74A4F"
                            }
                        },
                        "cellLines": {
                            "line1": true,
                            "line2": false,
                            "line3": false
                        }
                    }
                ],
                "createDate": "0.59841300 1558505564",
                "rowData": {
                    "id": "id",
                    "contract_number_c": "contract_number",
                    "accounts|id": "account_id",
                    "aos_products_quotes|id": "**",
                    "aos_products_quotes|addresses_aos_products_quotes|id": "**",
                    "aos_products_quotes|addresses_aos_products_quotes|address_street": "street",
                    "aos_products_quotes|addresses_aos_products_quotes|address_number": "number",
                    "aos_products_quotes|addresses_aos_products_quotes|address_bus": "",
                    "aos_products_quotes|addresses_aos_products_quotes|address_addition": "",
                    "aos_products_quotes|addresses_aos_products_quotes|address_postalcode": "zipcode",
                    "aos_products_quotes|addresses_aos_products_quotes|address_city": "city",
                    "aos_products_quotes|packages|id": "c991324c-c182-0255-1bd4-598d92e74cf5",
                    "aos_products_quotes|packages|name": "MIG_ONLINE_B2C"
                },
                "sortByOrder": "desc",
                "sortBy": "3202583"
            }
        ],
        "fileName": null,
        "pagination": {
            "page": 1,
            "size": 10,
            "sortBy": "contract_number_c desc",
            "total": 1,
            "pages": 1
        },
        "async": false
    },
    "message": "SUCCESS"
}
TheLastProject commented 5 years ago

That JSON looks like the JSON Dutch Essent has to render the webpage. Not like API output

Is there a chance the library can be expanded in order to work with both API's ?

Sure, however, I can't do that myself. The first step would be to try to login successfully, so we can see how much the rest of the API differs. The login screen identifies itself as OpenAM so this seems to be how: https://backstage.forgerock.com/docs/am/6.5/dev-guide/#sec-rest-authentication.

You probably want to start trying to just do it with curl. If that works, it can be switched around to Python.

davidclaeysquinones commented 5 years ago

I've played a bit around and have figured out the login part.

Authentication

A post message is sent to https://sso.essent.be/am/json/authenticate

It returns the following json :

{
    "authId": "eyAidHlwIjogIkpXVCIsICJhbGciOiAiSFMyNTYiIH0.eyAib3RrIjogIjU5M2o0OWdqcHFqaXM1dTkwaGwwbHFyZDUxIiwgInJlYWxtIjogIm89ZXNzZW50YmUsb3U9c2VydmljZXMsZGM9ZXNzZW50LGRjPWl3ZWxjb21lLGRjPWNvbSIsICJzZXNzaW9uSWQiOiAiQVFJQzV3TTJMWTRTZmN3RmFKby1Ub0hYclpnQ2s0X1pjMkVlZnhLazUzUkZMUEUuKkFBSlRTUUFDTURJQUFsTkxBQlF0TXpNeE5qY3lPREF4TWpnNE16WXdOVFE0TlFBQ1V6RUFBakF4KiIgfQ.aFfIUVlIkPYKnn9OKEnHSfulMei5zda7LEKsja0Qq-g",
    "template": "",
    "stage": "LDAP1",
    "header": "Sign in to OpenAM",
    "callbacks": [
        {
            "type": "NameCallback",
            "output": [
                {
                    "name": "prompt",
                    "value": "User Name:"
                }
            ],
            "input": [
                {
                    "name": "IDToken1",
                    "value": ""
                }
            ]
        },
        {
            "type": "PasswordCallback",
            "output": [
                {
                    "name": "prompt",
                    "value": "Password:"
                }
            ],
            "input": [
                {
                    "name": "IDToken2",
                    "value": ""
                }
            ]
        }
    ]
}

authId is a JWT token that has the following tokens :

now a request needs to be sent again to https://sso.essent.be/am/json/authenticate but this time with the following body :

{
    "authId": "eyAidHlwIjogIkpXVCIsICJhbGciOiAiSFMyNTYiIH0.eyAib3RrIjogIm04cTg1cjMwaXZhZmV0bTVtYzE3cXNlZnFyIiwgInJlYWxtIjogIm89ZXNzZW50YmUsb3U9c2VydmljZXMsZGM9ZXNzZW50LGRjPWl3ZWxjb21lLGRjPWNvbSIsICJzZXNzaW9uSWQiOiAiQVFJQzV3TTJMWTRTZmN3czhyX05seDBMZUtCUE5hUTVpY3FTdXlBYVVrWm1ITFEuKkFBSlRTUUFDTURJQUFsTkxBQk15TXprek9EWXlNekExTVRNeU5qRTNPRFV6QUFKVE1RQUNNREUuKiIgfQ.K4y8nYHPXBP3gtMY2OdXyp2PsLK6dhZuqPfW8vCsPZE",
    "callbacks": [
        {
            "type": "NameCallback",
            "output": [
                {
                    "name": "prompt",
                    "value": "User Name:"
                }
            ],
            "input": [
                {
                    "name": "IDToken1",
                    "value": "username@example.com"
                }
            ]
        },
        {
            "type": "PasswordCallback",
            "output": [
                {
                    "name": "prompt",
                    "value": "Password:"
                }
            ],
            "input": [
                {
                    "name": "IDToken2",
                    "value": "passwordUser"
                }
            ]
        }
    ]
}

then you get the following response : { "tokenId": "tokenId", "successUrl": "https://prod.nova.essent.be/dwp/#/start/dashboard/home/​" }

Getting current user id

In order to get the user id you need to send a post request to https://sso.essent.be/am/json/users?_action=idFromSession

You need to add a cookie to this request with the following specifications:

This returns the following json :

{
    "id": "userId",
    "realm": "/",
    "dn": "id=f6d60016-f95e-47b4-ad8c-cbd79e3bbd82,ou=user,o=essentbe,ou=services,dc=essent,dc=iwelcome,dc=com",
    "successURL": "https://prod.nova.essent.be/dwp/#/start/dashboard/home/​",
    "fullLoginURL": "/am/UI/Login?realm=%2Fessentbe"
}

Getting current user information

In order to get the current user information you submit a get request to https://sso.essent.be/am/json/users/userId

You need to add a cookie to this request with the following specifications:

The json output will look like this :

{
    "username": "userId",
    "realm": "/essentbe",
    "telephoneNumber": [
        "telephone"
    ],
    "mail": [
        "email@example.com"
    ],
    "givenName": [
        "FirstName"
    ],
    "objectClass": [
        "iwelcomeUser",
        "top",
        "organizationalPerson",
        "person",
        "inetOrgPerson",
        "sunFMSAML2NameIdentifier",
        "iplanet-am-user-service"
    ],
    "dn": [
        "uid=f6d60016-f95e-47b4-ad8c-cbd79e3bbd82,ou=people,dc=essent,dc=iwelcome,dc=com"
    ],
    "cn": [
        "Full Name"
    ],
    "modifyTimestamp": [
        "sometimestamp"
    ],
    "createTimestamp": [
        "sometimestamp"
    ],
    "privateEmailAddress": [
        "email@example.com"
    ],
    "iWelcomeAuthAttr10": [
        "iwMyApps",
        "customer_mijnessent_full",
        "iwPasswordReset",
        "ROLE_default"
    ],
    "iWelcomeAccountName": [
        "email@example.com"
    ],
    "uid": [
        "f6d60016-f95e-47b4-ad8c-cbd79e3bbd82"
    ],
    "iWelcomeAuthAttr1": [
        "essentbe"
    ],
    "universalid": [
        "id=f6d60016-f95e-47b4-ad8c-cbd79e3bbd82,ou=user,o=essentbe,ou=services,dc=essent,dc=iwelcome,dc=com"
    ],
    "preferredlanguage": [
        "NL"
    ],
    "sn": [
        "Last Name"
    ],
    "iWelcomeCustomerName": [
        "essentbe"
    ],
    "iWelcomePrivateEmail": [
        "email@example.com"
    ],
    "sun-fm-saml2-nameid-info": [
        "https://sso.essent.be:443/am|prod-crm-essent|email@example.com|https://sso.essent.be:443/am|urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress|null|prod-crm-essent|DualRole|false"
    ],
    "sun-fm-saml2-nameid-infokey": [
        "https://sso.essent.be:443/am|prod-crm-essent|email@example.com"
    ],
    "roles": [
        "ui-self-service-user"
    ]
}

Api

The rest of the api seems to be located at https://prod.nova.essent.be/ although https://sso.essent.be probably has some methods that I haven't discovered.

I don't think it's worth it to further examine this API since it's very UI driven. I'm wondering how I could explore the rest of the methods on https://sso.essent.be

TheLastProject commented 5 years ago

I don't think it's worth it to further examine this API since it's very UI driven.

Well, the Dutch Essent API has no documentation either, I just traced what my browser did.

I wonder though, what happens if you open https://api.essent.be/ in your browser and login? Maybe we're overcomplicating things.

davidclaeysquinones commented 5 years ago

I just have tried https://api.essent.be/selfservice/user/authenticateUser supplying the following xml :

<AuthenticateUser>
            <request>
                <username>user@example.com</username>
                <password>password</password>
                <ControlParameters>
                    <GetContracts>True</GetContracts>
                </ControlParameters>
            </request>
</AuthenticateUser>

And get the following response :

<html>
    <head>
        <title>401 Unauthorized</title>
    </head>
    <body>
        <h1>Unauthorized</h1>
        <p>This server could not verify that you
are authorized to access the document
requested.  Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
    </body>
</html>

Did I mess up the xml ? In case https://sso.essent.be needs to be figured out I see the following problem : https://prod.nova.essent.be/ is the api that is used for the data retrieval of the application and probably gets its data from https://sso.essent.be. How can I figure out which methods of https://sso.essent.be are being called without having direct acces to the server running https://prod.nova.essent.be/?

TheLastProject commented 5 years ago

No, just https://api.essent.be/ in your browser. It pops up a login screen.

davidclaeysquinones commented 5 years ago

It popups a login screen in the browser but the credentials don't seem to work :(

TheLastProject commented 5 years ago

Could you try the be_login branch and tell you how far you get when following the README?

davidclaeysquinones commented 5 years ago

The output of my test looks like this :

File "test.py", line 7, in <module>
    essent = PyEssent(username='***', password='*******', country=country)
  File "C:\Users\DavidClaeys\source\repos\PyEssent\PyEssent\pyessent\pyessent.py", line 129, in __init__
    data['callbacks'][0]['input']['value'] = username
TypeError: list indices must be integers, not str

This looks way more promising than my previous attempt. I see you've taken into account my research so probably it's some programming error.

TheLastProject commented 5 years ago

Thanks for testing, I created a new commit in the branch. Hopefully that'll let you get further?

davidclaeysquinones commented 5 years ago

I got further. Now I'm getting this message :

  File "test.py", line 7, in <module>
    essent = PyEssent(username='**', password='**', country=country)
TypeError: __init__() should return None
TheLastProject commented 5 years ago

Whoops. Should be fixed now.

davidclaeysquinones commented 5 years ago

I'm affraid we have a problem

   File "test.py", line 8, in <module>
    for ean in essent.get_EANs():
  File "C:\Users\DavidClaeys\source\repos\PyEssent\PyEssent\pyessent\pyessent.py", line 148, in get_EANs
    customer_details_request = PyEssent.Customer.get_customer_details()
  File "C:\Users\DavidClaeys\source\repos\PyEssent\PyEssent\pyessent\pyessent.py", line 47, in get_customer_details
    r.raise_for_status()
  File "C:\Python27\lib\site-packages\requests\models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://api.essent.be/selfservice/customer/getCustomerDetails?GetContracts=false

I had the same issue when trying to reach the url with postman.

TheLastProject commented 5 years ago

Hmm, interesting. I added a new commit, just an experiment. Probably won't work but it's worth a try.

davidclaeysquinones commented 5 years ago

Nope

File "C:\Users\DavidClaeys\source\repos\PyEssent\PyEssent\pyessent\pyessent.py", line 149, in __init__
    r.raise_for_status()
  File "C:\Python27\lib\site-packages\requests\models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://sso.essent.be/am/json/users?_action=idFromSession
davidclaeysquinones commented 5 years ago

Did you put a cookie on the request as I specified ?

TheLastProject commented 5 years ago

Nope, I was hoping it would be set with the successful login by itself. Try again now?

davidclaeysquinones commented 5 years ago

There is probably an error when setting the cookie :

  File "C:\Users\DavidClaeys\source\repos\PyEssent\PyEssent\pyessent\pyessent.py", line 145, in __init__
    SESSION.cookies['iwessent'] = r.json['tokenId']
TypeError: 'instancemethod' object has no attribute '__getitem__'
TheLastProject commented 5 years ago

Oops, typo, fixed

davidclaeysquinones commented 5 years ago

There is still an error but this time with api.essent.be

  File "C:\Users\DavidClaeys\source\repos\PyEssent\PyEssent\pyessent\pyessent.py", line 158, in __init__
    PyEssent.User.authenticate_user(username, password)
  File "C:\Users\DavidClaeys\source\repos\PyEssent\PyEssent\pyessent\pyessent.py", line 111, in authenticate_user
    r.raise_for_status()
  File "C:\Python27\lib\site-packages\requests\models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://api.essent.be/selfservice/user/authenticateUser
TheLastProject commented 5 years ago

I've just pushed another commit, but honestly, there is no way I'm going to be able to figure this out, I simply don't have enough info.

I'm afraid at this point you will need to try and suggest (or code yourself) more possible ways to get authenticated yourself or maybe share your login info or plan a TeamViewer session or so so I could look around myself. If you do want to share your login, you could email it to me (sylvia@hackerchick.me), but if you do have more ideas yourself I'd rather try that first because asking people for login info is just... kinda nasty.

davidclaeysquinones commented 5 years ago

I'm not that keen into providing my own credentials. I will fiddle more around and in case I don't figure it out I'll set some dummy password in order to provide you the necessary credentials. Are they any ideas I could explore ?

TheLastProject commented 5 years ago

One idea is to see if they have some smartphone app and if there's anything special you need to do to be able to sign in there? The API identifies itself as "Mobile API" so... maybe that's relevant?