apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.89k stars 4.27k forks source link

How to run typescript apache beam pipeline in Flink Cluster #27581

Open hilarudeensaktivo opened 1 year ago

hilarudeensaktivo commented 1 year ago

Hi Team,

I have tried starting apache flink in localhost and run typescript apache beam pipeline. I could not make succeed on this. is there any step by step document to achieve the same.

Used below docker compose to start cluster in localhost.

version: "3.8"
services:
  jobmanager:
    image: flink:latest
    hostname: jobmanager
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      FLINK_PROPERTIES: "jobmanager.rpc.address: jobmanager"
    volumes:  
        - .:/home
  taskmanager:
    image: flink:latest 
    hostname: taskmanager
    depends_on:
      - jobmanager
    command: taskmanager
    scale: 2
    volumes:
        - .:/home
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 4

Tried below command to run pipeline against flink cluster.

node dist/src/main.js --input_text="Greetings" \
--runner=flink --flinkMaster="http://0.0.0.0:8081" 

Any help would be appreciated!

ibzib commented 1 year ago

What was the error message you got?