dashbitco / broadway_cloud_pub_sub

A Broadway producer for Google Cloud Pub/Sub
Apache License 2.0
70 stars 24 forks source link

Allows messages to be maps with string keys #33

Closed Adzz closed 5 years ago

Adzz commented 5 years ago

For some reason that I can't figure out, when querying Pub Sub in production, line 71 in GoogleApiClient the pubsub_projects_subscriptions_pull/3 returns me something that looks like this:

 {:ok,
 %GoogleApi.PubSub.V1.Model.PullResponse{
   receivedMessages: [
     %{
       "ackId" => "ID",
       "message" => %{
         "attributes" => %{"space" => "vegans"},
         "data" => "VEhJUyBJUyBBIFRFU1Q=",
         "messageId" => "1234321",
         "publishTime" => "2019-09-05T17:41:31.520Z"
       }
     }
   ]
 }}

However I notice when I run the tests here the same function returns me:

{:ok,
 %GoogleApi.PubSub.V1.Model.PullResponse{
   receivedMessages: [
     %GoogleApi.PubSub.V1.Model.ReceivedMessage{
       ackId: "ACK ID",
       message: %GoogleApi.PubSub.V1.Model.PubsubMessage{
         attributes: %{"space" => "vegans"},
         data: "VEhJUyBJUyBBIFRFU1Q=",
         messageId: "708601755676243",
         publishTime: ~U[2019-09-05 17:41:31.520Z]
       }
     }
   ]
 }}

This allows me to continue even though my response from Pub Sub is not getting serialized into a GoogleApi.PubSub.V1.Model.ReceivedMessage

closes #34

mcrumm commented 5 years ago

I think this does not belong here, since the underlying Google library is ultimately responsible for (de)serialization.

Additionally, with this change a consumer can no longer be certain of the shape of the message data coming out of the producer, as they'll now have to match on both atom and string keys.

josevalim commented 5 years ago

Agreed. We should rather try to figure out exactly why you are getting a map of string keys. Are you running the latest? What is your broadway, broadway_cloud_pub_ssub and google API versions?

Adzz commented 5 years ago

Thanks all, I agree too, not proud of the code particularly just didn't know what else to do.

I updated the issue https://github.com/plataformatec/broadway_cloud_pub_sub/issues/34 , maybe lets continue the conversation there so we don't get confused! But am using latest versions of the libraries