Create a service which pulls current charging station data from OpenChargeMap.
The following fields will be imported:
operatorInfo
statusType
addressInfo
Connections
ConstrainsmaxResults is defined by default to 100. In average it takes less than 1000ms to fetch the first 100 items:
But it has no pagination:
To retrieve all the data, we can either:
a. to increase maxResults, which increases the response time proportionally.
b. to make multiple requests.
Option (a) doesn't scale well, it can lead to timeouts.
Option (b) can be performed using queuing mechanisms, executing multiple requests in parallel, but this may result in rate limits.
Expanding option (b), we can use some query parameters like: countryid, dataproviderid or operatorid to segment the data. The more parameters used in the requests, the smaller the chunks.
Good news is that those ids can be fetched beforehand, with one request to https://api.openchargemap.io/v3/referencedata.
Approach: Option (b) creating jobs (rabbitMQ queue) per each operatorid, this thanks to the relationship: one operator belongs to a country. Requests can be filtered by allowed countries, sending then only the operatorid that belongs to them.
To research
OpenChargeMap Rate limits.
Pulling frequency. To avoid oversaturate Mongo with write operations.
An example
Acceptance Criteria
Pull the data in small chunks per countries + data providers.
Store the data in the DB: POI list.
Local setup with docker-compose: MongoDB + RabbitMQ.
Create a service which pulls current charging station data from OpenChargeMap.
The following fields will be imported:
operatorInfostatusTypeaddressInfoConnectionsConstrains
maxResultsis defined by default to 100. In average it takes less than1000msto fetch the first 100 items:But it has no pagination:
To retrieve all the data, we can either: a. to increase
maxResults, which increases the response time proportionally. b. to make multiple requests.Option (a) doesn't scale well, it can lead to timeouts. Option (b) can be performed using queuing mechanisms, executing multiple requests in parallel, but this may result in rate limits.
Expanding option (b), we can use some query parameters like:
countryid,dataprovideridoroperatoridto segment the data. The more parameters used in the requests, the smaller the chunks.Good news is that those ids can be fetched beforehand, with one request to
https://api.openchargemap.io/v3/referencedata.Approach: Option (b) creating jobs (rabbitMQ queue) per each
operatorid, this thanks to the relationship: one operator belongs to a country. Requests can be filtered by allowed countries, sending then only theoperatoridthat belongs to them.To research
An example
Acceptance Criteria