KeyError EXTERNAL

[danizen]

I'm trying to process the z3950 1995 asn spec using asn1ate, because I'm trying to revive the module PyZ3950 which implemented its own asn1 encoder/decoder, and I need to get good data before I choose to replace with pyasn1 and asn1ate.

The file https://github.com/danizen/PyZ3950/blob/master/z3950_asn/z3950_1995.asn holds the asn I am trying to parse, and I worked through problems where copy and paste from https://www.loc.gov/z3950/agency/asn1.html to that file failed, e.g. comments broken onto multiple lines.

Now that I've done that, I think it gets through parsing the file, but when it goes to export the pyasn1 schemas, it outputs this:

KeyError: 'EXTERNAL'

I understand its looking for that in a dict, but how can I address this?

[kimgr]

Is that all you see, or do you also get a python traceback?

[danizen]

I get the traceback, I'll provide it:

# Auto-generated by asn1ate v.0.6.0 from z3950_1995-2.asn
# (last modified on 2018-09-22 09:41:44.016003)

from pyasn1.type import univ, char, namedtype, namedval, tag, constraint, useful

class ReferenceId(univ.OctetString):
    pass

ReferenceId.tagSet = univ.OctetString.tagSet.tagImplicitly(tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 2))

class InfoCategory(univ.Sequence):
    pass

InfoCategory.componentType = namedtype.NamedTypes(
    namedtype.OptionalNamedType('categoryTypeId', univ.ObjectIdentifier().subtype(implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 1))),
    namedtype.NamedType('categoryValue', univ.Integer().subtype(implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 2)))
)

class InternationalString(char.GeneralString):
    pass

class OtherInformation(univ.SequenceOf):
    pass

Traceback (most recent call last):
  File "/home/dan/python-envs/z3950/bin/asn1ate", line 11, in <module>
    sys.exit(main())
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 640, in main
    generate_pyasn1(module, output_file, modules)
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 523, in generate_pyasn1
    return Pyasn1Backend(sema_module, out_stream, referenced_modules).generate_code()
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 145, in generate_code
    details = self.generate_definition(assignment)
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 159, in generate_definition
    return self.generate_defn(assigned_type, type_decl)
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 171, in generate_defn
    return generator(class_name, t)
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 231, in defn_tagged_type
    nested_dfn = self.generate_defn(class_name, t.type_decl)
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 171, in generate_defn
    return generator(class_name, t)
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 280, in defn_collection_type
    fragment.write_line('%s.componentType = %s' % (class_name, self.generate_expr(t.type_decl)))
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 167, in generate_expr
    return generator(t)
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 308, in inline_constructed_type
    fragment.write_block(self.inline_component_types(t.components))
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 321, in inline_component_types
    component_exprs.append(self.generate_expr(c))
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 167, in generate_expr
    return generator(t)
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 427, in inline_component_type
    return "namedtype.NamedType('%s', %s)" % (t.identifier, self.generate_expr(t.type_decl))
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 167, in generate_expr
    return generator(t)
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 308, in inline_constructed_type
    fragment.write_block(self.inline_component_types(t.components))
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 321, in inline_component_types
    component_exprs.append(self.generate_expr(c))
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 167, in generate_expr
    return generator(t)
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 430, in inline_named_type
    return "namedtype.NamedType('%s', %s)" % (t.identifier, self.generate_expr(t.type_decl))
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 167, in generate_expr
    return generator(t)
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 337, in inline_tagged_type
    type_expr += '.subtype(%s=%s)' % (tag_implicitness, self.build_tag_expr(t))
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/pyasn1gen.py", line 351, in build_tag_expr
    tagged_type_decl = self.sema_module.resolve_type_decl(tag_def.type_decl, self.referenced_modules)
  File "/home/dan/python-envs/z3950/local/lib/python2.7/site-packages/asn1ate/sema.py", line 305, in resolve_type_decl
    return module.resolve_type_decl(module.user_types()[type_decl.type_name], referenced_modules)
KeyError: 'EXTERNAL'

Found more official definition file at ftp://ftp.loc.gov/pub/z3950/official/asn1.txt, no copy and paste.

[danizen]

This section of RFC 3641 shows that EXTERNAL is a built-in ASN1 type. I wish they'd repeated the definition rather than referring to X.690.

I think I can lift the definition of EXTERNAL from there, and compile it along with the rest of the definitions.

[danizen]

So, https://www.obj-sys.com/asn1tutorial/node16.html very usefully provides this schema for external:

EXTERNAL  ::=  [UNIVERSAL 8] IMPLICIT SEQUENCE
     {
      direct-reference  OBJECT IDENTIFIER OPTIONAL,
      indirect-reference  INTEGER OPTIONAL,
      data-value-descriptor  ObjectDescriptor  OPTIONAL,
      encoding  CHOICE
                  {single-ASN1-type  [0] ANY,
                   octet-aligned     [1] IMPLICIT OCTET STRING,
                   arbitrary         [2] IMPLICIT BIT STRING}
     }

As a work-around, I will directly add this to my ASN file and try to compile to pyasn1 again. However, EXTERNAL is clearly a basic type supported by the spec and should be supported by asn1ate.

[kimgr]

Most of asn1ate is based on X.680 and this is actually mentioned in clause 37. I hadn't seen that before, and we don't yet have any handling of such "well-known" types in asn1ate.

I think it would need to be introduced as a builtin type with a special sema node so code generators could produce the structure you showed above (or the X.680 equivalent).

I'm not sure if/when I'll find time to do so, so any contributions are welcome here.

[danizen]

X.860 is interesting reading. I last dealt with binary encoding rules with XDR and a version of rpcgen that compiled ONC RPC's idl into json encoding. I'm surprised to see that ASN.1 is actually a much more recent and modern standard, relatively. That makes it more complicated, but I ought to be capable of learning it given time. That said, I spend much of my time encouraging web developers to follow basic software engineering precepts such as to use the build system to do builds, and to check in only the stuff needed to build (i.e. an ASN1 file, rather than its pyasn1 equivalent python) - not in my current role.

This is a project on my own anyway; I just find it a bit of an afront that there isn't a good Python 3 z39.50 implementation.