SBERepeatingGroupIterator missing

[tamboril]

Hi. I'm liking the prospects of using your library. Just ran into this and thought you might want to fix it:

    from sbedecoder.message import SBEMessage, SBERepeatingGroupIterator
ImportError: cannot import name SBERepeatingGroupIterator

It's in the tmpl file and so in the generated code.

[tamboril]

Also can you verify that self.iterators is supposed to be self.groups? I had to make this patch to make a simple test work with generated code:

diff --git a/generator/sbe_message.tmpl b/generator/sbe_message.tmpl
index 12aa5f6..dd16a01 100644
--- a/generator/sbe_message.tmpl
+++ b/generator/sbe_message.tmpl
@@ -10,7 +10,7 @@ ${docstring}
 '''
 %endif

-from sbedecoder.message import SBEMessage, SBERepeatingGroupIterator
+from sbedecoder.message import SBEMessage
 from sbedecoder.message import TypeMessageField, EnumMessageField, CompositeMessageField, SetMessageField

 ## Build a class definition for each message that is defined
@@ -141,12 +141,12 @@ class ${message_def.name}(${message_def.base_classes}):
                                            ])
         %endfor

-        self.iterators = [
+        self.groups = [
         %for group in message_def.iterators:
             self.${group.name},
         %endfor
         ]
         %else:
-        self.iterators = []
+        self.groups = []
         %endif
 </%def>

[tfgm-bud]

@tamboril Sorry about this. I haven't been using the generator and honestly just didn't verify that it still worked after making the group changes. I actually find the dynamically generated classes pretty easy to work with. Sort of ironic as I was the one who requested the class generator. Anyway, I'll take a look at it right now.

[tamboril]

Thanks, Bud. I almost have it fixed up, but I'll wait for your patch.

The class generator is a great idea, especially with the template idea. It's important to me, because I don't want to require other developers to have to have all the non-standard python libraries in order to build my project, and the CME templates rarely change.

[tfgm-bud]

@tamboril I just merged in a fix for the generator. It doesn't handle the case where there are nested groups yet. That's going to take a bit more work. The current MDP 3.0 templates don't have any nested groups so if that's what you are decoding, you should be fine. I'll leave this open to remind me to fix that.

[tamboril]

Bud, Thanks. There is some other problem with the latest. The error (code is attached): pydecoder.zip

Traceback (most recent call last):
  File "/home/jharris/tmp/pydecoder/pydecoder.py", line 43, in <module>
    for msg in parser.parse(buf2, offset):
  File "/home/jharris/repos/sbedecoder/sbedecoder/parser.py", line 8, in parse
    message = self.factory.build(message_buffer, msg_offset)
  File "/home/jharris/repos/sbedecoder/sbedecoder/message.py", line 309, in build
    message.wrap(msg_buffer, offset)
  File "/home/jharris/repos/sbedecoder/sbedecoder/message.py", line 294, in wrap
    group_offset += group_iterator.wrap(msg_buffer, msg_offset, group_offset)
  File "/home/jharris/repos/sbedecoder/sbedecoder/message.py", line 237, in wrap
    block_length = self.block_length_field.value
  File "/home/jharris/repos/sbedecoder/sbedecoder/message.py", line 52, in value
    _raw_value = self.raw_value
  File "/home/jharris/repos/sbedecoder/sbedecoder/message.py", line 72, in raw_value
    self.msg_offset + self.relative_offset + self.field_offset)[0]
struct.error: unpack_from requires a buffer of at least 2 bytes

[tfgm-bud]

@tamboril yeah, so not sure where you got that message from but remember how you said that the CME doesn't change the templates that often? Well it turns out they do change them. The current version is 8 and the message you are trying to decode is a version 5 message. Unfortunately we don't support versioning (issue #6). It is definitely something I need to figure out and get working...

Not sure why you couldn't get the tests to run. I was able to clone a new instance of the repo and run them with no issues:

bud@ws-bud ~/tmp $ [master|6] git clone https://github.com/tfgm/sbedecoder.git
Cloning into 'sbedecoder'...
remote: Counting objects: 138, done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 138 (delta 0), reused 4 (delta 0), pack-reused 128
Receiving objects: 100% (138/138), 49.74 KiB | 0 bytes/s, done.
Resolving deltas: 100% (57/57), done.
Checking connectivity... done.
bud@ws-bud ~/tmp $ [master|6] cd sbedecoder/
bud@ws-bud ~/tmp/sbedecoder $ [master] virtualenv venv
New python executable in /home/bud/tmp/sbedecoder/venv/bin/python
Installing setuptools, pip, wheel...done.
bud@ws-bud ~/tmp/sbedecoder $ [master] source venv/bin/activate
(venv) bud@ws-bud ~/tmp/sbedecoder $ [master] python setup.py install
[snip]
Finished processing dependencies for sbedecoder==0.1
(venv) bud@ws-bud ~/tmp/sbedecoder $ [master] nosetests -v tests/test_sbe_parser.py
test_sbe_parser.TestSBEParserLibrary.test_incremental_refresh_multiple_messages ... ok
test_sbe_parser.TestSBEParserLibrary.test_incremental_refresh_trade_summary ... ok
test_sbe_parser.TestSBEParserLibrary.test_incremental_refresh_verify_group_attributes ... ok
test_sbe_parser.TestSBEParserLibrary.test_incremental_refresh_verify_groups ... ok
test_sbe_parser.TestSBEParserLibrary.test_security_status ... ok
test_sbe_parser.TestSBEParserLibrary.test_security_status_reset_statistics ... ok

----------------------------------------------------------------------
Ran 6 tests in 0.966s

OK
(venv) bud@ws-bud ~/tmp/sbedecoder $ [master] cd generator/
(venv) bud@ws-bud ~/tmp/sbedecoder/generator $ [master] ./sbe_class_generator.py --schema /mnt/cme-data/2017/08/18/Templates/MDP3/templates_FixBinary.xml

(venv) bud@ws-bud ~/tmp/sbedecoder/generator $ [master] cp generated.py ../sbedecoder
(venv) bud@ws-bud ~/tmp/sbedecoder/generator $ [master] cd ..
(venv) bud@ws-bud ~/tmp/sbedecoder $ [master] nosetests -v tests/test_sbe_parser.py
test_sbe_parser.TestSBEParserLibrary.test_incremental_refresh_multiple_messages ... ok
test_sbe_parser.TestSBEParserLibrary.test_incremental_refresh_trade_summary ... ok
test_sbe_parser.TestSBEParserLibrary.test_incremental_refresh_verify_group_attributes ... ok
test_sbe_parser.TestSBEParserLibrary.test_incremental_refresh_verify_groups ... ok
test_sbe_parser.TestSBEParserLibrary.test_security_status ... ok
test_sbe_parser.TestSBEParserLibrary.test_security_status_reset_statistics ... ok

----------------------------------------------------------------------
Ran 6 tests in 0.572s

OK

[tamboril]

OK. You're right. That was an old message that my current C++ printers, which I'm trying to replace, handled by virtue of the RealLogic decoders handling the older version.

I put a big new message in there, and it doesn't crash.

I didn't have a problem with the unit tests.

Thanks for the quick patch.