Need a way send custom values to callback in Pubnub.publish()? Also, windowing?

[mm-gmbd]

I'm trying to follow Pubnub's best practices for Guaranteed Message Delivery, wherein it gives options on how to graciously handle large messages. The first method was to increase the windowing size, but I haven't seen that option implemented in the Pubnub Ruby API (is this coming?).

In the meantime, I'm chunking my message, sending it in pieces, and reconstructing on the client side, however when I'm chunking my messages a few of the chunks are getting lost - this problem is referenced directly in the best practices, however the solution for this is to not send all the chunks at once - sounds easy enough, but the method described is "waiting for the publish response before sending another message".

Because the structure of the callback message is predefined (somewhere under the hood the callback supplied to the published is getting .call()'ed), I cannot insert any of my own variables into the callback and am only left with the "what happened" status message which includes "sent", "message too large", etc...

I really would like to keep this contained within functions rather than using some global array to store the chunked pieces of the message, so there is a need for a way to pass in some extra variables to the callback function rather than just the status - is this achievable now and I've just overlooked it?

Thanks in advance,

-MM

[stephenlb]

Hi @mmuelle4 Good questions regarding message size. You will want to enable Elastic Message on your account: https://admin.pubnub.com/ - and increase the max message size.

[stephenlb]

Regarding the interface of Response Codes, can you show me code directly? Also Geremy Cohen will likely join in the conversation too.

[mm-gmbd]

Hey @stephenlb, I actually ended up finding a solution to the issue which I'd be happy to share. I'm somewhat new to Ruby/Rails (I've used Pubnub spottedly in the past) and the way I was handling the chunking/publishing code was OK, but not sufficient to have Guaranteed Message Delivery.

The code (before) in a Rails controller:

class MessagesController < ApplicationController def create message = JSON.parse(request_body) m_id = message["Message_ID"] pn_message = { some : "stuff", original_msg: message } pn_message_str = pn_message.to_json

max_size = 400 #bytes
if pn_message_str.bytesize < max_size
  #Go ahead and send
else
  chunked_message = #Build chunked message
  chunk_id = 0
  while chunk_id < chunked_message.length
    chunk = chunked_message[chunk_id]
      pn.publish({
        :channel => channel,
        :message => chunk,
        :callback => lambda { |msg| puts msg }
      })
    chunk_id += 1
  end
end

end end

My issue with this was, in order to have Guaranteed Message Delivery, publishing that fast was not allowed as sometimes messages are lost out there somewhere. So, I was trying to keep this structure while getting the desired functionality and thought that the following may (but probably wouldn't work):

First attempt:

class MessagesController < ApplicationController def create message = JSON.parse(request_body) m_id = message["Message_ID"] pn_message = { some : "stuff", original_msg: message } pn_message_str = pn_message.to_json

max_size = 400 #bytes
if pn_message_str.bytesize < max_size
  #Go ahead and send
else
  pn_do_send = true ##EDIT
  chunked_message = #Build chunked message
  chunk_id = 0
  while chunk_id < chunked_message.length
    if pn_do_send ##EDIT
      chunk = chunked_message[chunk_id]
      pn.publish({
        :channel => channel,
        :message => chunk,
        :callback => lambda { |msg| 
                                         puts msg 
                                         pn_do_send = true ##EDIT
                                       }
      })
      chunk_id += 1
      pn_do_send = false ##EDIT
    end ##END PN_DO_SEND IF
  end
end

end end

As mentioned, I figured this wouldn't work, but it was quick to write and maayybbee Ruby/Rails would allow - it didn't work. At this point, all I could think of is, "How can I insert my own values into the callback function?" This was the right track, but I couldn't figure out how to without refactoring the code. Eventually, I structured the code like this:

class MessagesController < ApplicationController def create message = JSON.parse(request_body) m_id = message["Message_ID"] pn_message = { some : "stuff", original_msg: message } pn_message_str = pn_message.to_json

max_size = 400 #bytes
handle_pubnub_message(pn, pn_channel, m_id, pn_message_str, max_size)

end

private

def handle_pubnub_message(pn, pn_channel, id, msg, max_size)
  if msg.bytesize < max_size
    #Send PN Message
  else
    chunked_message = #Chunk message
    send_chunked_pubnub_message(pn, pn_channel, id, chunked_message, 0)
  end
end

def send_chunked_pubnub_message(pn, pn_channel, id, chunked_message, chunk_id)
  pn.publish({
    :channel => pn_channel,
    :message => chunked_message[chunk_id]
    :callback => lambda { |status|
                                     chunk_id += 1
                                     if chunked_message.length > chunk_id
                                       send_chunked_pubnub_message(pn, pn_channel, id, chunked_message, chunk_id)
                                     end
                                  }
  })
end

end

So, looking back now the solution wasn't too hard to come by, it was accomplished by simply wrapping the pn.publish() call for the chunked message within a function that could keep track of the current chunk to send.

Maybe this can serve as an example somewhere - either way, thanks for the response and don't hesitate to any any questions!

-Max

[geremyCohen]

Very clever @mmuelle4 !

[mm-gmbd]

Thanks @geremyCohen, I have another question and not sure if it should be moved to another issue, but for now...

You can see in the code that the "max_size" var I'm setting is 400 (bytes) - I read on Pubnub's page that the max message size that can be send, save changing the limit with Elastic Message, was 1,800 bytes. I'm fairly certain the chunking I'm performing keeps the message size the "max_size" or less, so I had initially tried setting the max_size to half of Pubnub's maximum size (900 bytes) just to err on the side of caution, but I kept getting the callback status of "Message Too Large"!

After double-checking the bytesize of my messages, I couldn't figure out what the issue was that was causing the messages to be too large for Pubnub to handle so I figured I'd just dial back the "max_size", and I noticed that at around 400 I stopped receiving the "Message Too Large" response.

So, is this due to the headers associated with the Pubnub.publish()? If so, that seems to be a hefty amount of header size... just thought I'd bring it up!

[geremyCohen]

@mmuelle4 the max size is based on the entire URL-encoded message... not the plain URL before url-encoding. Does this perhaps explain things?

[mm-gmbd]

@geremyCohen, indeed it does. In that case, I guess it is just safer to find the "max_size" that works for my messages and be done with it, but just for the sake of curiosity is there a way to get the URL-encoded message without actually performing the request?

[mm-gmbd]

Not that I could determine the chunk size based off of that, I'd get stuck in a "which came first" battle, and I've already gone through too many of those headaches (just one for this app, but that's enough).

[geremyCohen]

let me know if https://github.com/pubnub/ruby/blob/master/3.3/lib/pubnub_request.rb#L246 points you in the right direction on this...

[mm-gmbd]

Looks like what I'm looking for -- thanks!