GoogleCloudPlatform / DataflowJavaSDK

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
http://cloud.google.com/dataflow
855 stars 325 forks source link

PubsubMessage readed form PubsubIO have no attributes #629

Open chikien276 opened 6 years ago

chikien276 commented 6 years ago

Dataflow SDK version: 2.4.0 I haven't tested with other attribute but when we use these attribute for de-duplication and timestaming we got null PubsubMessage's attribute For example

PubsubIO.readMessages()
                        .fromSubscription("mine")
                        .withIdAttribute("id"))
                .apply("name hehe", ParDo.of(new DoFn<PubsubMessage, String>() {

                    @ProcessElement
                    public void processElement(ProcessContext c) {
                        PubsubMessage message = c.element();
                        c.output(new String(message.getPayload(), Charset.forName("UTF-8"))

                                + " id: " + message.getAttribute("id")); // id will be null here
                    }
                }))

same with .withTimestampAttribute(str)

sx5640 commented 5 years ago

Has this problem been resolved?