[Bug]: typescript SDK - DirectRunner - readFromPubSub does not work

jesse-triplewhale commented 10 months ago

What happened?

The readFromPubSub function in typescript does not work on the DirectRunner, though it does work on DataflowRunner. In the course of debugging I tested writeToPubSub on the DirectRunner, which also does work.

My code is below. I do not see any messages in stdout, nor do I see any messages in the output topic. When I look at the subscription in the GCP UI, I see that messages are sent, but not acked, so it seems like the messages are making it to the java transform, but not to the typescript function.

    const msgs = await root.applyAsync(
      beam.withName(
        'readFromPubSub',
        beam.readFromPubSub({
          subscription: input_subscription,
        })
      )
    );
    msgs.map((element) => console.log(element));
    await msgs.applyAsync(
      beam.withName(
        'writeToPubSub',
        beam.writeToPubSub(output_topic)
      )
    );

Update: I tested beam.readFromText, which is a cross-language transform that invokes python, and I confirmed that it does work and the values make it back to typescript

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[X] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

jesse-triplewhale commented 10 months ago

@robertwb I saw that you wrote this TS code - thanks for that! Curious if you have any thoughts here?

robertwb commented 10 months ago

The problem is that neither the direct typescript runner nor the cross-langauge directly Python runner support streaming pipelines. We recently updated the error for that--is it not clear?

jesse-triplewhale commented 10 months ago

Hi Robert,

beam_run.txt I don't get an error when I run. I'll attach the output here

Thank you, Jesse

apache-beam@2.49.0 with the code I pasted in my original post

apache / beam