-
Estimated Time: 8 weeks
Tasks and Detailed Requirements:
1. Implement Low-Latency Data Processing Pipeline:
○ Time: 4 weeks
○ Tools Required: Apache Kafka, Spark Streaming (within Azure)
○…
zepor updated
1 month ago
-
Proposal:
We create a **wayang-kafka** module inside of **wayang-platforms**.
This leads to a cleaner approach to add these operators.
Right now we edit both **wayang-java** and **wayang-spark…
-
## Description
Hi!
After upgrading my Spark Operator Helm chart from version `1.1.26` (hosted previously in the https://googlecloudplatform.github.io/spark-on-k8s-operator repository) to version `…
-
~/Data-Engineering-with-Databricks-Cookbook$ sh build.sh
Error response from daemon: page not found
Container name cannot be empty
Error response from daemon: page not found
Container name cannot …
-
## 📅 24/08/08
- 프로젝트 진행에 앞서, 전체적인 아키텍처를 구상.
- Rapid에서 사용할 수 있는 무료 실시간 api를 통한 파이프라인 구축 목표.
- Kafka, Spark, Airflow등을 중점으로 프로젝트를 구축.
![전체 아키텍처](https://github.com/user-attachments…
-
> 제가 자주 사용하는 Spark 라는 Framework를 소개해보았습니다.
> 이전 글들과는 다르게 사용해보시지 않으셨으면 조금 어려울 수도 있는 내용이 많습니다.
> 혹시나 관심있어서 들어오셨는데, 내용이 부실하여 이해가 잘 안되신다면 편하게 코멘트 부탁드립니다~
> 피드백은 언제나 환영입니다. 👋
![쉐보레 스파크](https://gith…
-
hi : kvs=KafkaUtils.createStream(ssc,zookeeper,self.app_conf["app_name"],topic,keyDecoder=utf8_decoder,valueDecoder=lambda v :Avro.AvroToJson(v,self.schema))
spark kafka start , but KafkaUtil…
-
I changed the base image from gcr.io spark:3.1.1 to spark:3.5.0 and built my dockerfile from there where it has my DataAnalyticsReporting.jar
This is the error when the spark-operator tries to star…
-
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
When …
lizc9 updated
3 weeks ago
-
Originally asked at https://stackoverflow.com/questions/76392397/got-org-apache-kafka-common-errors-timeoutexception-topic-xxx-not-present-in-m
I am wondering could it be related to this library th…