multi-module asn files don't work

[aholtzma]

I've been trying to get asn1ate to digest the 3GPP RRC protocol, and I've had some success with taking snippets out and processing them individually. However the parser chokes on the whole thing:

# Auto-generated by asn1ate v.0.5.1.dev from rrc.asn1
# (last modified on 2016-09-08 23:03:47)

from pyasn1.type import univ, char, namedtype, namedval, tag, constraint, useful

class BCCH_BCH_Message(univ.Sequence):
    pass

Traceback (most recent call last):
  File "asn1ate/pyasn1gen.py", line 631, in <module>
    sys.exit(main())
  File "asn1ate/pyasn1gen.py", line 622, in main
    generate_pyasn1(module, output_file, modules)
  File "asn1ate/pyasn1gen.py", line 495, in generate_pyasn1
    return Pyasn1Backend(sema_module, out_stream, referenced_modules).generate_code()
  File "asn1ate/pyasn1gen.py", line 145, in generate_code
    details = self.generate_definition(assignment)
  File "asn1ate/pyasn1gen.py", line 159, in generate_definition
    return self.generate_defn(assigned_type, type_decl)
  File "asn1ate/pyasn1gen.py", line 171, in generate_defn
    return generator(class_name, t)
  File "asn1ate/pyasn1gen.py", line 210, in defn_constructed_type
    fragment.write_block(self.inline_component_types(t.components))
  File "asn1ate/pyasn1gen.py", line 321, in inline_component_types
    component_exprs.append(self.generate_expr(c))
  File "asn1ate/pyasn1gen.py", line 167, in generate_expr
    return generator(t)
  File "asn1ate/pyasn1gen.py", line 427, in inline_component_type
    return "namedtype.NamedType('%s', %s)" % (t.identifier, self.generate_expr(t.type_decl))
  File "asn1ate/pyasn1gen.py", line 167, in generate_expr
    return generator(t)
  File "asn1ate/pyasn1gen.py", line 337, in inline_tagged_type
    type_expr += '.subtype(%s=%s)' % (tag_implicitness, self.build_tag_expr(t))
  File "asn1ate/pyasn1gen.py", line 351, in build_tag_expr
    tagged_type_decl = self.sema_module.resolve_type_decl(tag_def.type_decl, self.referenced_modules)
  File "/Users/aholtzma/src/asn1ate/asn1ate/sema.py", line 303, in resolve_type_decl
    return module.resolve_type_decl(module.user_types()[type_decl.type_name], referenced_modules)
KeyError: 'SystemInformation-BCH'

The reason is that SystemInformation-BCH is defined in the PDU-definitions module, which gets completely dropped by the parser (confirmed by running test.py --parse, I won't paste it here).

[kimgr]

It could be that dependencies are not wired between modules -- I have quite elaborate logic in place to detect dependencies and generate assignments in dependency order, not sure if this carries over to multiple modules. Also not sure how hard it would be to make that happen.

[aholtzma]

Wouldn't the dependencies be handled at the semantic level? The parse tree doesn't even have these modules.

[kimgr]

Yes, at the semantic level.

Oh, now I see you say PDU-definitions disappears from the parse tree... I think that might be because --parse only dumps the parse tree of the first module, but I'm not sure. Take a look in test.py.

[aholtzma]

Attached is a minimal test case for multi-modules with a dependency. multi-module.asn1.txt

The result is actually different than what I see with the rrc.asn. In this case the parse sees both modules. It fails on resolving the cross module depenency though:

Boo-definitions DEFINITIONS AUTOMATIC TAGS ::=

BEGIN

  Yourenum ::= ENUMERATED { five, six, seven }

END

Foo-definitions DEFINITIONS AUTOMATIC TAGS ::=

BEGIN

IMPORTS
  Yourenum
FROM
  Boo-definitions;

  Mysequence-with-enum  ::=      SEQUENCE 
  {
    foo Yourenum DEFAULT two,
    bar ENUMERATED { a, b, c} DEFAULT c
  }

END

outputs:

WARNING: More than one module generated to the same stream.
# Auto-generated by asn1ate v.0.5.1.dev from multi-module.asn1.txt
# (last modified on 2016-09-09 16:01:43)

from pyasn1.type import univ, char, namedtype, namedval, tag, constraint, useful
import foo_definitions

class Yourenum(univ.Enumerated):
    pass

Yourenum.namedValues = namedval.NamedValues(
    ('five', 0),
    ('six', 1),
    ('seven', 2)
)

# Auto-generated by asn1ate v.0.5.1.dev from multi-module.asn1.txt
# (last modified on 2016-09-09 16:01:43)

from pyasn1.type import univ, char, namedtype, namedval, tag, constraint, useful
import boo_definitions

class Mysequence_with_enum(univ.Sequence):
    pass

Traceback (most recent call last):
  File "asn1ate/pyasn1gen.py", line 633, in <module>
    sys.exit(main())
  File "asn1ate/pyasn1gen.py", line 624, in main
    generate_pyasn1(module, output_file, modules)
  File "asn1ate/pyasn1gen.py", line 497, in generate_pyasn1
    return Pyasn1Backend(sema_module, out_stream, referenced_modules).generate_code()
  File "asn1ate/pyasn1gen.py", line 145, in generate_code
    details = self.generate_definition(assignment)
  File "asn1ate/pyasn1gen.py", line 159, in generate_definition
    return self.generate_defn(assigned_type, type_decl)
  File "asn1ate/pyasn1gen.py", line 171, in generate_defn
    return generator(class_name, t)
  File "asn1ate/pyasn1gen.py", line 210, in defn_constructed_type
    fragment.write_block(self.inline_component_types(t.components))
  File "asn1ate/pyasn1gen.py", line 321, in inline_component_types
    component_exprs.append(self.generate_expr(c))
  File "asn1ate/pyasn1gen.py", line 167, in generate_expr
    return generator(t)
  File "asn1ate/pyasn1gen.py", line 424, in inline_component_type
    type_expr = self.generate_expr(t.type_decl)
  File "asn1ate/pyasn1gen.py", line 167, in generate_expr
    return generator(t)
  File "asn1ate/pyasn1gen.py", line 337, in inline_tagged_type
    type_expr += '.subtype(%s=%s)' % (tag_implicitness, self.build_tag_expr(t))
  File "asn1ate/pyasn1gen.py", line 351, in build_tag_expr
    tagged_type_decl = self.sema_module.resolve_type_decl(tag_def.type_decl, self.referenced_modules)
  File "/Users/aholtzma/src/asn1ate/asn1ate/sema.py", line 303, in resolve_type_decl
    return module.resolve_type_decl(module.user_types()[type_decl.type_name], referenced_modules)
KeyError: 'Yourenum'

So I think this may be two different problems - first the parsing problem, second the inter-module hookup.

[kimgr]

Thanks for the contained repro.

I seem to recall this PR added support for emitting modules to individual files, maybe that would help here? Looks like there's a --split argument to pyasn1gen.py.

[kimgr]

Oh, now I see you say PDU-definitions disappears from the parse tree... I think that might be because --parse only dumps the parse tree of the first module, but I'm not sure.

This turned out to be caused by a bug in the grammar. which caused the rest of the module to be consumed by the IMPORTS parser rule. I have a fix/simplification in the works.

[mungayree]

I think is still seem to occur on on cloned tree (asn1ate v.0.6.1.dev0). Is it checked in yet ?

[kimgr]

No, I don't think I ever got around to solving this entirely. It was harder than I thought, and I ran out of time.