-
Hi folks 👋
I noted that there's an Apache Spark integration[1] for creating segments and batch-ingesting data. I'd be keen to explore creating a similar IO connector for Apache Beam[2], and I woul…
-
### What happened?
### Problem description
The function `apache_beam.io.gcp.pubsublite.ReadFromPubSubLite` is not working. This can be seen as a simple data processing pipeline for reading messages …
-
Hello,
I am running the following code with python 2 kernel:
`!pip install --upgrade setuptools
!pip install apache-beam
!pip install apache-beam[gcp]`
Installation is completing successfully…
-
Seems to be another pipeline execution system?
https://beam.apache.org/get-started/beam-overview/
-
### What happened?
I'm using "org.apache.beam:beam-runners-flink-1.18:2.57.0".
When I read from pubsub, I found it's not able to acknowledging messages that are generated before the job starts. As …
-
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
I'd like to deploy on GCP Dataflow, Apa…
-
### What happened?
I am trying to stream data from PubSub (with a throughput of 10-50 RPS) into an Iceberg Table (not partitioned) using the IcebergIO connector and Hive Metastore.
However, aft…
-
**Describe the solution you'd like**
Integration with Apache Beam and GCP Dataflow. Ability to use Google Dataflow templates and custom templates in addition to creating pipelines with Apache Beam.
…
-
### What happened?
Following the release of **PrismRunner** for Python SDK with Apache Beam 2.59.0, we are trying to adapt our code so that we can use the same code in GCP as in Local debugging (Dire…
-
### What would you like to happen?
### Summary
This feature request proposes adding asynchronous computation to Apache Beam's Interactive Beam API. This means allowing long-running tasks to execut…