search

opensearch-project / project-website

The source to opensearch.org
https://opensearch.org
BSD 3-Clause "New" or "Revised" License
40 stars 479 forks source link

[BLOG] Monitoring Apache Spark using opensearch #3130

Open avichaym opened 3 months ago

avichaym commented 3 months ago

Describe the blog post your would like to write

Opensearch can be an ideal choice for continuous monitoring of apache spark jobs in production. This year , we released the Guidance for Analytics Observability on AWS reference solution , which enhances Any spark ditro (OSS, Glue , EMR) with a plugin JAR that outputs various spark metrics to Opensearch via Data Prepper. This solution also provides multiple operational dashboards for apache spark for root causing common issues like Data Skew , and also provides a Vega visualization for historical Spark-SQL plans as a Graph.

The blog will also demonstrate how opensearch anomaly detection features can further enhance the spark monitoring capabilities.

What is the title of the blog post? Monitoring Apache Spark using opensearch

Who are the authors? Avichay Marciano, Vincent Gromakowsky

What is the proposed posting date? September 1st

pajuric commented 3 months ago

Thank for the blog idea. When you are ready, please feel free to load a draft of the blog here or send me a copy to pattijur@amazon.com.

pajuric commented 1 month ago

@avichaym - Following up to see if you still intend to write and contribute this blog, or if we are able to close this issue? Please advise.

pajuric commented 3 weeks ago

@avichaym - Moving this into backlog and awaiting input from the the author about whether they plan to m0ve this forward.