Project Submission

[pai]

Project Name

zakar-agumbe-ai

Project Type

• [X] early-warning system (Anomaly Detection)
• [X] Visualization
• [X] Both

Project Description

GitHub Org: https://github.com/agumbe-ai Confluence: https://agumbe.atlassian.net/wiki/home JIRA: https://agumbe.atlassian.net/jira/software/projects/DEV/boards/1

Zakar Wildfire Alert System is a wildfire early detection system designed to forecast and measure wildfire events. It is built on geospatial data dimensions, including temperature and text data, and actual wildfire alerts.

Our predictive model is trained on seven years of historical data - to measure Dimensions(Temperature, Tweets, actual Wildfire Alerts) and forecast the occurrence of a (Wildfire) Metric. It contains the following microservices:

auth Repository: https://github.com/agumbe-ai/auth.git Status: Ready

• an authentication service (authN).
• an authorization service (authZ).
• Includes basic authentication i.e username and password.
• Has it's own MongoDB collection.
• Exposes REST APIs.

data-collector Repository: https://github.com/agumbe-ai/data-collector.git Status: Ready

• Consumes data from memphis stations. Pre-processes the data to include collectorTimeStamp (required for Elasticsearch time series analysis).
• Collector time stamp starts on Aug 1 2016, uses unique seconds to mock timestamps.
• Stations being consumed: zakar-tweets, zakar-temperature-readings, zakar-fire-alerts. Has MongoDB collections for each station.
• Exposes REST APIs.

data-elastic-search Repository: https://github.com/agumbe-ai/data-elastic-search.git Status: Needs Work

• Starts analysis jobs on Elasticsearch using data from stations.
• Fetches analysys and search results, makes available for dashboard.
• Has it's own MongoDB collection.
• Exposes REST APIs.

data-analysis Repository: https://github.com/agumbe-ai/data-analysis.git Status: Needs Work

• ML analysis on incoming messages.
• Trains ML on historical messages. We fetch fire-alert event days and notification days, fetch tweet and temperature data for the days in between. Do sentiment analysis on text. Temperature readings analysis. Trains model on what could be a "wildfire" sentiment. Updates model with trainings.
• Trains ML on all other days' data to clarify what is not a "wildfire".
• Produces alerts to zakar-fire-alerts station
• Has it's own MongoDB collection.
• Saves models to MinIO storage.
• Exposes REST APIs.

streamlit-dashboard Repository: https://github.com/agumbe-ai/streamlit-dashboard.git Status: Ready

• Visualises data as-is using pandas.
• Displays Elasticsearch data.
• Displays heatmaps.
• Displays Latest Alert.

npm-commons Repository: https://github.com/agumbe-ai/npm-commons.git Status: Ready

• shared library for middlewares, custom errors, etc.

metrics Repository: https://github.com/agumbe-ai/metrics.git Status: Todo

• Author, manage Metrics (ex: Wildfire) in the system.
• Make available metrics for ML model training.
• Manages Metric lifecycle.
• Exposes REST APIs.

dimensions Repository: https://github.com/agumbe-ai/dimensions.git Status: Todo

• Author, manage Dimensions (ex: Temperature, Tweets) in the system.
• Make available dimensions for ML model training.
• Manages Dimension cardinality and lifecycle.
• Allows linking Dimensions to Metrics, etc.
• Exposes REST APIs.

tenants Repository: https://github.com/agumbe-ai/tenants.git Status: Ready

• Allows multi-tenancy.
• Manages Tenant lifecycle.
• Exposes REST APIs.

manifests-index Repository: Repository: https://github.com/agumbe-ai/manifests-index.git Status: Ready

• Contains YAML files for Kubernetes deployment.
• Uses Kustomize with base definitions and overlays for each namespace.
• Critical for scaffolding developer environment.
• Contains generate-manifests.sh file to automate manifest builds.
• Contains skaffold yaml.
• Contains argo base and overlays for argo application creation automation.

Postman APIs Collection:

Stack currently in use: Memphis.dev(Cloud), Streamlit, GitHub, Confluence(Cloud), Elasticsearch(Cloud), Kibana(Cloud), Python, TypeScript, Jest, Kubernetes (GKE), Argo, Kustomize, Skaffold, Postman, Mongo DB (Atlas)

Additions in the coming weeks: GX(Cloud), Spark, Presto/Trino, KubeFlow Pipelines, Grafana, Seldon Core, MinIO

Streamlit link

http://zakar.agumbe.ai/

Source code link

https://github.com/agumbe-ai

Instructions for the reviewers

Thanks for taking the time to review. Much appreciated! Feedback on all aspects of the project (Design, Product, DevEx, Approach, Over Engineered?) are encouraged.

Notes

• Please do send across GitHub id's to add to Org for reviews.
• We follow Database Per Service approach.
• All the latest sources are on development branches.
• We make heavy use of GitOps and GitHub workflows. Approach is published here: https://medium.com/itnext/12-factor-microservice-applications-on-kubernetes-db913008b018
• We love K8s and Argo! and Kustomize!
• Readmes exist for some repos only. But we believe the code is self-explanatory and straight-forward.
• We encourage you to take a look at Confluence! There are diagrams explaining the flows for each part of the Project. https://agumbe.atlassian.net/wiki/spaces/SD/pages/5341201/Project+Main+Template
• Confluence has information on Infrastructure setup, development environment setup, and other related subjects.
• We believe that the initial development environment setup could be challenging for some - but it's smooth sailing when one comes to grips.
• We have kept our repositories private due to Git Workflows and all our sensitive repository secrets. We read on Discord that this is allowed.
• Development index is tracked in JIRA.
• The environment hosted on zakar.agumbe.ai is a preview environment (uses skaffold run). It might need to be taken down sometimes, but will be back up in ~15-30mins.
• The dashboard response is slow when the complete dataset is available. Needs optimization.
• There is no certificate yet on the K8s cluster. Please type thisisunsafe on the browser to reach the dashboard.

Challenges

• Had to mock Time Stamps on incoming data from stations to make it Time Series friendly. Fairly challenging to get accurate random timestamps! We like the final results.
• Could not connect to Elasticseach at all via Python! Tried everything! Something basic missing, perhaps Free Trial does not allow access to ES ML models?
• Exported CSV manually to Elasticsearch and created Anomaly jobs using the Time Series friendly data, and got some nice results. Alas, due to no connectivity with ES, had to abandon the results.
• Could not post app on Streamlit Community! Something basic for Mongo missing! The app is published on zakar.agumbe.ai instead.
• Data Analysis is currently broken because the model is overwritten (update instead) for each batch of incoming messages. Needs MinIO integration for fix.

Todos

• Implement a model registry for experiment tracking.
• Implement MinIO for storage.
• Implement Elasticsearch.
• Implement Great Expectations.
• Implement Kubeflow Pipelines.

What made you decide to build this particular app?

The Hackathon arrived at a fortunate moment when we were actively seeking producer datasets to construct our Proof of Concept (POC). The timing couldn't have been better! Participating in the Hackathon has aided us in advancing towards our Minimum Viable Product (MVP).

Additionally, Headphones are always a big draw!

Other information

Thanks for the introduction to Streamlit! This was a nice time-bound challenge! Contributions to the project are welcome!

Reachout

• santosh@agumbe.ai
• santosh@sevenport.io

Good luck!

Confirm submission of private form

• [X] Completed

[pai]

notifications Repository: https://github.com/agumbe-ai/notifications.git Status: Ready

• Tweets are not analysed as ML model is not ready
• Temperature if greater than 100 generates an alert
• Produces alerts to zakar-fire-alerts station

[pai]

https://app-zakar-agumbe.streamlit.app/ works now!

[rnowling-memphis]

@pai Can you make the Streamlit dashboard public? Thanks!