Having a tookit for parsing Hexadecimal Data

[jchailloux]

Having a operator that can extract/decode data from it Hexadecimal format.

We can have thousands of sensors so we could not have one implementation per sensor. As the data extraction is usually the same, we could have a Descriptor for all sensor's data.

This data extraction is needed for example in the automotive industry.

Illustration

InputData

      sensorID,data in Hex
      001,"195E32004CFFFF1E"
      002,"00009779AF3B00"
      003,"0000000000000000"
      001,"195E32004CFFFF1E"
      001,"195E32004CFFFF1E"
      1234,"1000000000780000"
      003,"0000000000000000"
      001,"195E32004CFFFF1E"
      001,"197032004CFFFF1E"
      002,"00009779AF4A00"
      003,"0000000000000000"
      001,"197032004CFFFF1E"
      001,"197032004CFFFF1E"
      1600,"F320014300"

Each sensorId has it own way to extract data.

Descriptor definition

    <?xml version="1.0" encoding="ISO-8859-1"?>
    <raws>
      <raw id="001" name="ABC" length="8" >
        <packet id="1" name="Data_1" start="7" length="9" function="(0.125 * x) +     0"></packet>
      </raw>
      <raw id="1234" name="XXX" length="8" multiplex="a">
            <packet id="11" name="a" start="1" length="8" signed="true" ></packet>
            <packet id="12" name="b" multiplex="10" start="8" length="2" signed="true" function="(12 * x) + 4"></packet>
        <packet id="13" name="c" multiplex="20" start="8" length="2" signed="true" ></packet>
            <packet id="14" name="d" start="11" length="5" signed="true" function="pow(x,2) / 23.7658" ></packet>
      </raw>
    </raws>

[petenicholls]

I think this would be a useful addition to either a string or a byte toolkit where we could have additional parsing/formattting operators.

I will say +1 for streamsx.bytes and this toolkit will be used to provide operators and functions that work on the byte level.

[petenicholls]

pushed wrong button...

[JimGiles]

+1

[mikespicer]

+1 for a streamsx.string toolkit which can be used for simple conversions to and from strings. It could also include functions for checking validity of string literals, i.e. isValidHex or is ValidIPv4Address.

[jchailloux]

Where and when can I put the initial release ?

[jchailloux]

I already have the toolkit that parse a String binary or Hex (it runs with sensors at that time). It extracts data at the bit level according to the message definition.

Do I have to post it in Streams Exchange before having the opportunity to have it on GitHub ?

I also need the GitHub's discussion and feedback to improve it .

[ddebrunner]

No, StreamsExchange is not a staging area for IBMStreams at github, rather IBMStreams is a replacement for StreamsExchange.

[hildrum]

This has +1 from three of the five management committee members and no negative votes--what's it waiting on?

[petenicholls]

It was waiting for me....creating repos and added jchailloux as committer.now.

[petenicholls]

Created repos streamsx.bytes, added committer.

[ddebrunner]

As a reminder: Toolkit layout is described here:

https://github.com/IBMStreams/toolkits/wiki/proposal-for-IBMStreams-repository-guidelines