Support externalized messages

[shreeve]

I am hitting a BlazeDS server that is wrapping a Java based server. I see a lot of the messages with types of "DSK", "DSA", "DSC", etc. These look like a shorthand message type, that uses bitmapped fields instead of string-based keys. After looking through the reference implementation and also the source code for Charles (the web proxy), I added support so that RocketAMF can deserialize these messages. For certain fields, such as messageIdBytes (which is a byte array), I have adjusted the setter methods to also set a human-readable version called (for example) messageId (which is just a string version of the UUID from the byte array). For completeness, a reverse method should be added so that if you set messageId, it'll create a 16-byte byte array and store the bytes directly.

By the way, one other random helper (not sure if this is necessary, but I added it) is this:

convenience for instantiating TypedHash objects

Hash.class_eval { def +@; RocketAMF::Values::TypedHash.new("").update(self); end }

Let me know if any of this is helpful.

Thanks,

Steve Shreeve

[warhammerkid]

I'm trying to maintain a high test coverage ratio and I don't have access to BlazeDS, so I wouldn't be able to create tests for this. If you could write some test or at the very least add some binary data dump to spec/fixtures/object I can certainly investigate merging this in.

Is there any reason you didn't reuse the existing AbstractMessage class in lib/rocketamf/values/messages.rb?

[shreeve]

As I read your message this morning, I see that the github code is much newer than the gem that I was using when I made my changes. All of this may thus be irrelevant. :-)

Nevertheless, I'll try to describe what I did below. I'm certainly no expert on BlazeDS or AMF, but from what I gather there is both a "normal" way that messages are sent and a "shorthand" way. The normal way appears to indicate the message type by its class name (as a string) and then all of the fields follow (as strings) and then all of the actual values for the fields (as objects). In the shorthand way, instead of spelling out the whole class name, such as "flex.messaging.messages.AbstractMessage", a shorthand version called "DSK" is sent. Since this is a shorthand message, it tries to save space by not actually spelling out all of the field names as strings. Instead, following the "DSK", there are one or more bytes whose bits represent the presence of specific fields. So, for example if bits 1 and 4 of the first byte are on, then that means that two specific fields will follow. I assume the whole motivation is to shorten the length of the resulting message. From what I saw in RocketAMF, there is currently no way to deserialize these "shorthand" message types, which is what I added.

Here are the shorthand message mappings that I can see:

DSK - AcknowledgeDSMessage DSA - AsyncDSMessage DSC - CommandDSMessage DSD - DataDSMessage DSP - PagedDSMessage DSQ - SequencedDSMessage DSU - UpdateCollectionDSMessage

Here is a brief hierarchy of messages, along with the static fields defined in each class:

Abstract body clientId destination headers messageId timestamp timeToLive

Command < Abstract operation

Async < Abstract correlationId

Data < Async identity operation

UpdateCollection < Data collectionId replace updateMode

Acknowledge < Async

Sequenced < Acknowledge dataMessage dynamicSizing sequenceId sequenceProxies sequenceSize

Paged < Sequenced pageCount pageIndex

In my case, I needed to deserialize the "DSK" message, which needs the following minimal setup. The other classes above could follow the same pattern.

class AbstractMessage
  ExternalizedFields = [
    %w[ body clientId destination headers messageId timestamp 

timeToLive ], # byte 1: bits 1 through 7 %w[ clientIdBytes messageIdBytes ] # byte 2: bits 1 and 2 ]

class AsyncMessage
  ExternalizedFields = [
    %w[ correlationId correlationIdBytes ], # byte 1: bits 1 and 2
  ]

class AcknowledgeMessage
  ExternalizedFields = nil

In order to deserialize (which I called "internalize") a shorthand message, I modified the code in "../pure/deserializer.rb" to:

      if traits[:externalizable]
        if obj.respond_to?(:internalize)
          obj.internalize(source)
        else
          obj.externalized_data = deserialize(source)
        end
      else

Then each externalizable class has a method such as:

  def internalize(source)
    super
    populate(source, ExternalizedFields)
  end

So, to deserialize a message like DSK, we:

1) internalize AcknowledgeMessage, which will 2) internalize AsyncMessage, which will 3) internalize AbstractMessage, which will 4) read flags for AbstractMessage and deserialize the specified objects and then 5) read flags for AsyncMessage and deserialize the specified objects and then 6) read flags for AcknowledgeMessage and deserialize the specified objects

In order to read the flags and deserialize the specified objects, I use:

  def populate(source, fields)
    flags = []
    loop do
      flags << source.read(1).unpack('C').first
      break if flags.last < 128
    end

    if fields && !fields.empty?
      fields.each_with_index do |list, i|
        list.each_with_index do |name, j|
          if flags[i].ord[j] > 0
            data = RocketAMF.deserialize(source, 3)
            send("#{name}=", data)
          end
        end
      end
    end
  end

The flags[i].ord[j] syntax checks bit j of byte i in the flags array. If it's set, then that specific object needs to be deserialized.

Since some bit fields correspond to bytearrays that need to also be accessible as UUID's, some helper methods are needed, such as:

  def clientIdBytes=(obj)
    @clientId = to_pretty_uuid(obj)
    @clientIdBytes = obj
  end

In order to handle the other direction, I added others such as:

  def clientId=(obj)
    @clientIdBytes = to_binary_uuid(obj)
    @clientId = obj
  end

I'm not sure if this is still helpful or relevant, but I send it over just in case it's helpful. I've also attached a few

Cheers,

Steve Shreeve

ps - Here is the content of the message included in one of the images, if it's helpful...

00 03 00 00 00 01 00 0B 2F 33 2F 6F 6E 52 65 73 75 6C 74 00 00 FF FF FF FF 11 0A 0B 07 44 53 4B 09 62 6F 64 79 06 83 43 3C 65 6E 76 3A 45 6E 76 65 6C 6F 70 65 20 78 6D 6C 6E 73 3A 65 6E 76 3D 27 68 74 74 70 3A 2F 2F 73 63 68 65 6D 61 73 2E 78 6D 6C 73 6F 61 70 2E 6F 72 67 2F 73 6F 61 70 2F 65 6E 76 65 6C 6F 70 65 2F 27 3E 3C 65 6E 76 3A 48 65 61 64 65 72 3E 3C 2F 65 6E 76 3A 48 65 61 64 65 72 3E 3C 65 6E 76 3A 42 6F 64 79 3E 3C 76 61 6C 69 64 61 74 65 53 65 73 73 69 6F 6E 52 65 73 70 6F 6E 73 65 20 78 6D 6C 6E 73 3D 22 75 72 6E 3A 63 6F 6D 3A 6D 79 63 61 3A 73 69 22 3E 3C 72 65 73 75 6C 74 3E 74 72 75 65 3C 2F 72 65 73 75 6C 74 3E 3C 2F 76 61 6C 69 64 61 74 65 53 65 73 73 69 6F 6E 52 65 73 70 6F 6E 73 65 3E 3C 2F 65 6E 76 3A 42 6F 64 79 3E 3C 2F 65 6E 76 3A 45 6E 76 65 6C 6F 70 65 3E 11 63 6C 69 65 6E 74 49 64 06 49 33 39 35 39 34 44 41 41 2D 41 39 30 39 2D 35 32 30 42 2D 43 42 44 32 2D 44 41 41 41 35 37 39 46 45 30 43 33 1B 63 6F 72 72 65 6C 61 74 69 6F 6E 49 64 06 49 35 46 31 41 34 41 42 37 2D 38 36 43 41 2D 33 32 44 35 2D 39 46 41 46 2D 44 31 32 30 42 33 37 46 45 31 31 37 17 64 65 73 74 69 6E 61 74 69 6F 6E 01 0F 68 65 61 64 65 72 73 0A 0B 01 01 13 6D 65 73 73 61 67 65 49 64 06 49 33 39 35 39 35 37 34 43 2D 37 42 30 44 2D 34 37 31 36 2D 45 39 37 39 2D 41 36 36 43 45 45 39 31 34 30 42 41 15 74 69 6D 65 54 6F 4C 69 76 65 04 00 13 74 69 6D 65 73 74 61 6D 70 05 42 73 06 CA FD BA B0 00 01

----- Original Message ----- From: "warhammerkid" reply@reply.github.com To: steve.shreeve@gmail.com Sent: Tuesday, June 07, 2011 10:08 PM Subject: Re: [rocketamf] Support externalized messages (#1)

I'm trying to maintain a high test coverage ratio and I don't have access to BlazeDS, so I wouldn't be able to create tests for this. If you could write some test or at the very least add some binary data dump to spec/fixtures/object I can certainly investigate merging this in.

Is there any reason you didn't reuse the existing AbstractMessage class in lib/rocketamf/values/messages.rb?

Reply to this email directly or view it on GitHub: https://github.com/rubyamf/rocketamf/pull/1#issuecomment-1324049

[warhammerkid]

The provided binary dump doesn't mark class DSK as externalizable. When I converted it to a data file and deserialized it without any changes to RocketAMF, it properly identified all of the fields by name and deserialized it.

env = RocketAMF::Envelope.new.populate_from_stream(File.read('input.dat'))
puts "'"+env.messages[0].data.type+"'"
# => 'DSK'
puts env.messages[0].data.inspect
# => {:messageId=>"3959574C-7B0D-4716-E979-A66CEE9140BA", :timestamp=>1307493522347.0, :correlationId=>"5F1A4AB7-86CA-32D5-9FAF-D120B37FE117", :timeToLive=>0, :headers=>{}, :clientId=>"39594DAA-A909-520B-CBD2-DAAA579FE0C3", :body=>"<env:Envelope xmlns:env='http://schemas.xmlsoap.org/soap/envelope/'><env:Header></env:Header><env:Body><validateSessionResponse xmlns=\"urn:com:myca:si\"><result>true</result></validateSessionResponse></env:Body></env:Envelope>", :destination=>nil}

Do you have any other dump that actually contains the bit fields and externalizable flag?

[shreeve]

Oops! Try this message... it should be marked as externalizable:

00 03 00 00 00 01 00 0C 2F 33 33 2F 6F 6E 52 65 73 75 6C 74 00 00 FF FF FF FF 11 0A 07 07 44 53 4B A1 03 06 83 3F 3C 65 6E 76 3A 45 6E 76 65 6C 6F 70 65 20 78 6D 6C 6E 73 3A 65 6E 76 3D 27 68 74 74 70 3A 2F 2F 73 63 68 65 6D 61 73 2E 78 6D 6C 73 6F 61 70 2E 6F 72 67 2F 73 6F 61 70 2F 65 6E 76 65 6C 6F 70 65 2F 27 3E 3C 65 6E 76 3A 48 65 61 64 65 72 3E 3C 2F 65 6E 76 3A 48 65 61 64 65 72 3E 3C 65 6E 76 3A 42 6F 64 79 3E 3C 67 65 74 43 6F 6E 66 69 67 53 74 72 69 6E 67 52 65 73 70 6F 6E 73 65 20 78 6D 6C 6E 73 3D 22 75 72 6E 3A 63 6F 6D 3A 6D 79 63 61 3A 73 69 22 3E 3C 72 65 73 75 6C 74 3E 34 38 3C 2F 72 65 73 75 6C 74 3E 3C 2F 67 65 74 43 6F 6E 66 69 67 53 74 72 69 6E 67 52 65 73 70 6F 6E 73 65 3E 3C 2F 65 6E 76 3A 42 6F 64 79 3E 3C 2F 65 6E 76 3A 45 6E 76 65 6C 6F 70 65 3E 05 42 73 02 41 54 59 E0 00 0C 21 88 14 A0 67 FE 0D 3A 9C A2 74 4A AE D9 BD 7B 0B 0C 21 88 17 EE F6 BE 0D 84 62 17 F1 38 B6 A4 34 14 DE 02 0C 21 7B B0 1B C0 C8 36 8F 4D 7B 47 24 15 43 35 71 09 00

It should have the following YAML output when processed:

--- !ruby/object:RocketAMF::Envelope amf_version: 3 headers: []

messages:

• !ruby/object:RocketAMF::Message data: !ruby/object:RocketAMF::Values::AcknowledgeMessage body: <env:Envelope xmlns:env='http://schemas.xmlsoap.org/soap/envelope/'>env:Header/env:Headerenv:Body<getConfigStringResponse xmlns="urn:com:myca:si">48/env:Body/env:Envelope clientId: 8814A067-FE0D-3A9C-A274-4AAED9BD7B0B clientIdBytes: !ruby/object:StringIO {}

  correlationId: 7BB01BC0-C836-8F4D-7B47-241543357109 correlationIdBytes: !ruby/object:StringIO {}

  destination: headers: {}

  messageId: 8817EEF6-BE0D-8462-17F1-38B6A43414DE messageIdBytes: !ruby/object:StringIO {}

  timeToLive: 0 timestamp: 1306275431838.0 response_uri: "" target_uri: /33/onResult

[warhammerkid]

I downloaded the BlazeDS source to double check the implementation and added support for the shorthand AcknowledgeMessage, CommandMessage, and AsyncMessage to the existing message classes. Thanks for the data dump! Let me know if you run into issues with the new code.

[shreeve]

Great! Thanks, man.