Self-hosted read-only API Mirrors

[webprofusion-chrisc]

For users looking to control the availability of the API (so that downtime doesn't affect them) there is a proposed solution whereby they can run their own clone of the OCM API.

This would be a .net core 3.0 based service connecting to MongoDB, current database size is about 400MB, app size is about 250MB. The service could run on linux or windows and would need to be able to continuously sync with the main database.

API users who are interested in this feature should +1 here.

[JohannaSte]

Thank you, I am interested in this feature!

[webprofusion-chrisc]

This feature is now available for those who want to try it out. Currently you have to clone and build the API, with a MongoDB service running locally. See build info: https://github.com/openchargemap/ocm-system/blob/master/README.md

The plan is to ultimately provide a docker image for this.

[webprofusion-chrisc]

So we are now serving some of our API reads from a beta version of the mirror, in this case linux with dotnet core 3.1 and MongoDB. https://api-router.openchargemap.io

You can get involved in development and testing by checking out: https://github.com/openchargemap/ocm-system/tree/master/API/OCM.Net/OCM.API.Worker

[webprofusion-chrisc]

Side note: Webprofusion Pty Ltd (my company) is now offering hosted & managed instances of the Open Charge Map API commercially - these will have an SLA for avg response time and availability and are available as monthly or annual subscription (from $199 USD monthly, depending on requirements). Most people will still use the free API but some users may require a commercial level of service.

[remuslazar]

@webprofusion-chrisc I am trying to setup the OCM mirror (the OCM.API.Worker) in docker and I'm getting this kind of errors:

app_1  |       Request Failed: Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 25 - Connection string is not valid)

I thought that for a mirror node, using the "IsCacheOnlyMode": "true", configuration option should not require the SQL server at all, right?

Could u please take a look at my current setup here?

https://github.com/ev-freaks/ocm-mirror

Basically the Dockerfile there just follows your instructions to setup the .NET application and inject a custom configuration file to set the mongoDB hostname correctly.

I have to admit, I am not a Windows guy and I do also not have any knowledge of .net.

Thanks!

[webprofusion-chrisc]

Thanks for trying, yes your configuration looks good but double check that the appsettings are being loaded - I suspect they're not? You may need to name it appsettings.prod.json or similar from memory, I can check it on more detail tomorrow.

[remuslazar]

Thank you for your quick reply! Yes, you are right. I'm just about to fix it..

[remuslazar]

@webprofusion-chrisc just a quick note: I was able to fix it. I did also update the repository, now it works out of the box using the supplied docker-compose.yml file:

app_1  | info: OCM.API.Worker.Worker[0]
app_1  |       Client: fetch completed: 3.3569409s
app_1  | info: OCM.API.Worker.Worker[0]
app_1  |       Latest POI update :: 1000 POIs 2000 in total 01/27/2021 14:04:00

Also web HTTP requests work!

Currently I'm testing it in docker on a linux VM behind a load balancer.

If everything works fine, I will create a more production grade ECS setup (https://aws.amazon.com/ecs/).

[webprofusion-chrisc]

This is great! I think for most users who are interested in using this they will just be spinning up a generic docker environment, I haven't tried ECS myself but it's great to have the option.

[remuslazar]

@webprofusion-chrisc I want to give you a quick update on that: I was able to finalize my ECS Setup, now running the OCM Mirror "production grade" on AWS / ECS:

• using a small ECS Cluster formed by 2 x t3a.micro (1GB RAM) Spot Instances
• using this ECS Setup: https://github.com/ev-freaks/ocm-mirror/tree/master/ecs

Using AWS Spot Instances the total costs (for both EC2 instances) are < $5 p.m. incl. full redundancy and fault tolerance (currently I am running 2 instances in two availability zones) behind an ELB. Not including traffic and the load balancer, we're using a shared one here.

The Task Definition configuration includes both the application itself and also the MongoDB container, so everything is self contained and will run out of the box. The memory reservation settings are quite low (256MB for the application, 512MB for mongo), I will see if this has to be tweaked in future.. For now, it runs quite smooth and fast. Both containers do share the 960MB available right now.

Using the provided Makefile one can build the images, push them and also create/update the ECS Services in AWS.

The ECS Setup is quite tailored to our AWS ECS setup, so this has to be adapted but I think this can be at least a good starting point for everybody.

[webprofusion-chrisc]

Excellent, thanks for the update. It's very interesting to see :)

[webprofusion-chrisc]

We (OCM) current use multiple lightsail instances for our linux based mirrors but it would be fascinating to use something like this so we can dynamically spin up new instances a little easier

[remuslazar]

@webprofusion-chrisc ECS is just great for that!

Basically just increase the number of tasks and ECS will take care of the rest, also distributing the tasks across different AZs and registering them behind the Load Balancer and so on. We're doing everything in an automated fashion using the AWS CLI and Makefiles + CloudFormation but one can also use the AWS Web Console for near everything as well, here a screenshot:

[webprofusion-chrisc]

Great, I need to play around with that :)