search

kaitai-io / kaitai_struct

Kaitai Struct: declarative language to generate binary data parsers in C++ / C# / Go / Java / JavaScript / Lua / Nim / Perl / PHP / Python / Ruby
https://kaitai.io
3.99k stars 194 forks source link

Unable to find Enum in Instance when Imported #857

Open Werni2A opened 3 years ago

Werni2A commented 3 years ago

I'm not sure whether this is the right repository or not, feel free to move the issue.

Enums don't work when they are used under the condition, that they are

I run Kaitai Struct locally in its WebIDE with latest commit 3a8cbe0d50596546fc05946bb27c6a04ec404ce9.

See the following example which produces the error message for bar while foo works fine.

Parse error: undefined
Call stack: undefined io.kaitai.struct.precompile.ErrorInInput: (main): /types/bar/instances/get_some_value/value: unable to find enum 'my_enum::foo', searching from repreoducer::bar

My current workaround is to define the enum inside reproducer.ksy then it works fine.

reproducer.ksy

meta:
  id: repreoducer
  file-extension: repreoducer
  endian: le
  imports: my_enum

types:

  foo:
    seq:
      - id: bitmap
        type: u2
        enum: my_enum::foo

  bar:
    seq:
      - id: bitmap
        type: u2
    instances:
      get_some_value:
        value: bitmap & 0x03
        enum: my_enum::foo

my_enum.ksy

meta:
  id: my_enum
  file-extension: my_enum

enums:
  foo:
    0: test0
    1: test1

This might be related to #651 but it seems like this bug has already been fixed, at least my type foo works without any problems.

Mingun commented 1 week ago

Probably the problem lies in those lines, because I do not see any other differences between values instances and seq fields: https://github.com/kaitai-io/kaitai_struct_compiler/blob/542b24124a95f5308df625d667f8776dd023c696/shared/src/main/scala/io/kaitai/struct/precompile/ResolveTypes.scala#L26-L35

For values instances ResolveTypes pass are never run, that means that neither type, neither enums will be resolved. When type field is forbidden in value instances, enum is not.

Trivial fix by just running this pass for value instances as well leads to another problem:

> .\ksc.bat --ksc-exceptions --verbose enum_resolve -d 857 -t java .\857\reproducer.ksy
Exception in thread "main" io.kaitai.struct.precompile.TypeUndecidedError: get_some_value
        at io.kaitai.struct.format.ValueInstanceSpec.dataType(InstanceSpec.scala:22)
        at io.kaitai.struct.precompile.ResolveTypes.resolveUserTypeForMember(ResolveTypes.scala:36)
        at io.kaitai.struct.precompile.ResolveTypes.$anonfun$resolveUserTypes$2(ResolveTypes.scala:27)
        at scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:118)
        at scala.collection.StrictOptimizedIterableOps.flatMap$(StrictOptimizedIterableOps.scala:105)
        at scala.collection.immutable.TreeMap.flatMap(TreeMap.scala:73)
        at io.kaitai.struct.precompile.ResolveTypes.resolveUserTypes(ResolveTypes.scala:27)
        at io.kaitai.struct.precompile.ResolveTypes.$anonfun$run$1(ResolveTypes.scala:15)
        at io.kaitai.struct.format.ClassSpec.mapRec(ClassSpec.scala:123)
        at io.kaitai.struct.format.ClassSpec.$anonfun$mapRec$1(ClassSpec.scala:125)
        at scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:118)
        at scala.collection.StrictOptimizedIterableOps.flatMap$(StrictOptimizedIterableOps.scala:105)
        at scala.collection.immutable.TreeMap.flatMap(TreeMap.scala:73)
        at io.kaitai.struct.format.ClassSpec.mapRec(ClassSpec.scala:124)
        at io.kaitai.struct.precompile.ResolveTypes.run(ResolveTypes.scala:15)
        at io.kaitai.struct.Main$.$anonfun$precompile$1(Main.scala:45)
        at scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:118)
        at scala.collection.StrictOptimizedIterableOps.flatMap$(StrictOptimizedIterableOps.scala:105)
        at scala.collection.mutable.HashMap.flatMap(HashMap.scala:35)
        at io.kaitai.struct.Main$.precompile(Main.scala:43)
        at io.kaitai.struct.Main$.$anonfun$importAndPrecompile$1(Main.scala:28)
        at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

That exception means that ValueInstanceSpec.dataTypeOpt contains None. This value is filled by SpecsValueTypeDerive pass which is runned after ResolveTypes pass:

> .\ksc.bat --ksc-exceptions --verbose value -d 857 -t java .\857\reproducer.ksy
### SpecsValueTypeDerive: iteration #1
#### reproducer
deriveValueType(reproducer)
deriveValueType(reproducer::bar)
get_some_value try to detect type of: EnumById(identifier(my_enum::foo),BinOp(Name(identifier(bitmap)),BitAnd,IntNum(3)),typeId(false,List(),false))
expression error: io.kaitai.struct.precompile.EnumNotFoundError: unable to find enum 'my_enum::foo', searching from reproducer::bar
.\857\reproducer.ksy: /types/bar/instances/get_some_value/value:
        error: unable to find enum 'my_enum::foo', searching from reproducer::bar

(--ksc-exceptions flag not always work. It seems that new CompilationProblem system is not aware of it. I have to add additional logging (see above) and comment catching of ExpressionError to get the stacktrace below) (btw, why I cannot specify multiple or all --verbose flags?)

> .\ksc.bat --ksc-exceptions --verbose value -d 857 -t java .\857\reproducer.ksy
### SpecsValueTypeDerive: iteration #1
#### reproducer
deriveValueType(reproducer)
deriveValueType(reproducer::bar)
get_some_value try to detect type of: EnumById(identifier(my_enum::foo),BinOp(Name(identifier(bitmap)),BitAnd,IntNum(3)),typeId(false,List(),false))
Exception in thread "main" io.kaitai.struct.precompile.EnumNotFoundError: unable to find enum 'my_enum::foo', searching from reproducer::bar
        at io.kaitai.struct.ClassTypeProvider.resolveEnum(ClassTypeProvider.scala:105)
        at io.kaitai.struct.ClassTypeProvider.resolveEnum(ClassTypeProvider.scala:103)
        at io.kaitai.struct.ClassTypeProvider.resolveEnum(ClassTypeProvider.scala:94)
        at io.kaitai.struct.translators.TypeDetector.detectTypeRaw(TypeDetector.scala:59)
        at io.kaitai.struct.translators.TypeDetector.detectType(TypeDetector.scala:25)
        at io.kaitai.struct.precompile.ValueTypesDeriver.$anonfun$deriveValueType$2(ValueTypesDeriver.scala:29)
        at io.kaitai.struct.precompile.ValueTypesDeriver.$anonfun$deriveValueType$2$adapted(ValueTypesDeriver.scala:21)
        at scala.collection.immutable.RedBlackTree$.foreach(RedBlackTree.scala:291)
        at scala.collection.immutable.TreeMap.foreach(TreeMap.scala:195)
        at io.kaitai.struct.precompile.ValueTypesDeriver.deriveValueType(ValueTypesDeriver.scala:21)
        at io.kaitai.struct.precompile.ValueTypesDeriver.$anonfun$deriveValueType$6(ValueTypesDeriver.scala:53)
        at io.kaitai.struct.precompile.ValueTypesDeriver.$anonfun$deriveValueType$6$adapted(ValueTypesDeriver.scala:51)
        at scala.collection.immutable.RedBlackTree$._foreach(RedBlackTree.scala:291)
        at scala.collection.immutable.RedBlackTree$.foreach(RedBlackTree.scala:290)
        at scala.collection.immutable.TreeMap.foreach(TreeMap.scala:195)
        at io.kaitai.struct.precompile.ValueTypesDeriver.deriveValueType(ValueTypesDeriver.scala:51)
        at io.kaitai.struct.precompile.ValueTypesDeriver.run(ValueTypesDeriver.scala:13)
        at io.kaitai.struct.precompile.SpecsValueTypeDerive.$anonfun$run$2(SpecsValueTypeDerive.scala:15)
        at io.kaitai.struct.precompile.SpecsValueTypeDerive.$anonfun$run$2$adapted(SpecsValueTypeDerive.scala:13)
        at scala.collection.mutable.HashMap$Node.foreach(HashMap.scala:642)
        at scala.collection.mutable.HashMap.foreach(HashMap.scala:504)
        at io.kaitai.struct.precompile.SpecsValueTypeDerive.run(SpecsValueTypeDerive.scala:13)
        at io.kaitai.struct.Main$.precompile(Main.scala:54)
        at io.kaitai.struct.Main$.$anonfun$importAndPrecompile$1(Main.scala:28)
        at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

Obviously, those two passes depends on each other and it is not trivial to split them.

It does not looks like that ClassTypeProvider cannot resolve imported types (although I do not see obvious code where imports is handled), because branch https://github.com/kaitai-io/kaitai_struct_compiler/pull/309 (which uses only ClassTypeProvider implementation) is able to resolve types in foo in reproducer.

Mingun commented 1 week ago

It seems that the real problem is here:

EnumById(identifier(my_enum::foo),...

Somehow the enum name contains the full name together with type (my_enum::foo) instead of just foo

Mingun commented 1 week ago

Of course... https://github.com/kaitai-io/kaitai_struct_compiler/blob/542b24124a95f5308df625d667f8776dd023c696/shared/src/main/scala/io/kaitai/struct/format/InstanceSpec.scala#L59-L65 another cutting of corners....