search

SocialHarvest / harvester

The Social Harvest server that exposes an API and harvests data from the web to be analyzed.
Other
112 stars 43 forks source link

Microservice architecture #68

Open tmaiaroto opened 9 years ago

tmaiaroto commented 9 years ago

I'm thinking about switching Social Harvest to a microservice architecture. Though it could all be ran from a single binary...There would be harvester services which would each gather data and then pass to a shipper service (which would store in the database, log file, etc.)

This approach would make for an easier to maintain (and extend) data pipeline. It'd be more modular.

My choice is Koding's Kite (which also has a front-end JavaScript client library which the dashboard might be able to use), but there's also Google's new gRPC which has clients in multiple languages and also handles authentication. gRPC also uses protocol buffers and HTTP/2. It's FAST. Though I don't know if protobuf will be all that useful given JSON is going to be more convenient throughout the pipeline.

Kite also has Kontrol, a service registry which gRPC does not (yet) have. So I'm leaning toward Kite but could feasibly see a desire to switch to gRPC later on down the road once it matures. It's quite attractive to know that people could be interacting with Social Harvest from so many different languages.

tmaiaroto commented 9 years ago

The other nice thing about the microservice architecture is that the memory concerns with the geocoder can be resolved in a nice way... Different geocoding services with different data sets and memory requirements... So a user could choose one (less accurate) data set but need less memory (cheaper hosting) in exchange. Microservices make these kind of trade-offs and high level configurations easy.