[P2] HTTP-based Ingestor Service

airavata-courses / vulcan

Spring 2022 Project

Apache License 2.0

1 stars 3 forks source link

Closed amberramesh closed 2 years ago

amberramesh commented 2 years ago

Switching Kafka implementations to HTTP in Ingestor Service.

amberramesh commented 2 years ago

Closing notes:

Kafka producers/consumers were replaced with web API frameworks (FastAPI + uvicorn) supporting async requests. Added/updated libraries for fast JSON parsing and data modelling.
Added logging support for viewing status in containers/eventual support with FileBeats input plugin for Logstash + ES.
Added configurations for Docker.