search

etingof / pyasn1

Generic ASN.1 library for Python
http://snmplabs.com/pyasn1
BSD 2-Clause "Simplified" License
242 stars 118 forks source link

how to fix asn spec given pyasn1.error.PyAsn1Error exception details #151

Open kaxap opened 5 years ago

kaxap commented 5 years ago

I am using a CDR spec from asn1ate and have several CDR files from Huawei hardware I want to decode.

I have translated the spec using asn1ate and using pyasn1 from the master branch as following:

from cdr import CallEventDataFile
from pyasn1.codec.native.encoder import encode
from pyasn1.codec.ber.decoder import decode as ber_decoder
import json

bytes_read = open("test-data/1546606507938713068", "rb").read()

data = ber_decoder(bytes_read, asn1Spec=CallEventDataFile())
n = encode(data)
with open("dump.json", "w", encoding="utf-8") as f:
    json.dump(n, f, ensure_ascii=False)

This throws an exception

Traceback (most recent call last):
  File "/home/r/dev/cdr-py/run.py", line 10, in <module>
    data = ber_decoder(bytes_read, asn1Spec=CallEventDataFile())
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 1498, in __call__
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 608, in valueDecoder
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 1498, in __call__
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 694, in valueDecoder
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 1498, in __call__
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 939, in valueDecoder
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 1498, in __call__
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 608, in valueDecoder
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 1530, in __call__
pyasn1.error.PyAsn1Error: <TagSet object at 0x28ca3a50b70 tags 128:0:160> not in asn1Spec: <TagMap object at 0x28cfaddf898 present {<TagSet object at 0x28cfadc6550 tags 128:0:0>: <CallEventRecordType schema object at 0x28cfadc64e0 tagSet <TagSet object at 0x28cfadc6550 tags 128:0:0> namedValues <NamedValues object 0x28cf9b2f1d0 enums moCallRecord=0, mtCallRecord=1, ...Record=25, forwardCallRecord=100>>, <TagSet object at 0x28cfadc66a0 tags 128:0:1>: <RecordingEntity schema object at 0x28cfadc6630 tagSet <TagSet object at 0x28cfadc66a0 tags 128:0:1> subtypeSpec <ValueSizeConstraint object at 0x28cf9abd2b0 consts 1, <Integer value object at 0x28cf9abdda0 tagSet <TagSet object at 0x28cf9c90208 tags 0:0:2> payload [20]>> encoding iso-8859-1>, <TagSet object at 0x28cfadc67b8 tags 128:32:2>: <ROUTE schema object at 0x28cfadc6748 tagSet=<TagSet object at 0x28cfadc67b8 tags 128:32:2> subtypeSpec=<ConstraintsIntersection object at 0x28cf9aa7668> componentType=<NamedTypes object at 0x28cf9af04a8 types <NamedType object at 0x28cf9af03c8 type rOUTENumber=<Integer schema object at 0x28cf9af0438 tagSet <TagSet object at 0x28cf9af0470 tags 128:0:0>>>, <NamedType object at 0x28cf9af0358 type rOUTEName=<GraphicString schema object at 0x28cf9af0518 tagSet <TagSet object at 0x28cf9af0550 tags 128:0:1> encoding iso-8859-1>>> sizeSpec=<ConstraintsIntersection object at 0x28cf9aa76d8 consts <ValueSizeConstraint object at 0x28cf9aa76a0 consts 1, 1>>>, <TagSet object at 0x28cfadc6908 tags 128:32:3>: <ROUTE schema object at 0x28cfadc6898 tagSet=<TagSet object at 0x28cfadc6908 tags 128:32:3> subtypeSpec=<ConstraintsIntersection object at 0x28cf9aa7668> componentType=<NamedTypes object at 0x28cf9af04a8 types <NamedType object at 0x28cf9af03c8 type rOUTENumber=<Integer schema object at 0x28cf9af0438 tagSet <TagSet object at 0x28cf9af0470 tags 128:0:0>>>, <NamedType object at 0x28cf9af0358 type rOUTEName=<GraphicString schema object at 0x28cf9af0518 tagSet <TagSet object at 0x28cf9af0550 tags 128:0:1> encoding iso-8859-1>>> sizeSpec=<ConstraintsIntersection object at 0x28cf9aa76d8 consts <ValueSizeConstraint object at 0x28cf9aa76a0 consts 1, 1>>>, <TagSet object at 0x28cfadc6a90 tags 128:0:4>: <CallingNumber schema object at 0x28cfadc6a20 tagSet <TagSet object at 0x28cfadc6a90 tags 128:0:4> encoding iso-8859-1>, <TagSet object at 0x28cfadc6be0 tags 128:0:5>: <CalledNumber schema object at 0x28cfadc6b70 tagSet <TagSet object at 0x28cfadc6be0 tags 128:0:5> encoding iso-8859-1>, <TagSet object at 0x28cfadc6d30 tags 128:0:6>: <BasicService schema object at 0x28cfadc6cc0 tagSet <TagSet object at 0x28cfadc6d30 tags 128:0:6> subtypeSpec <ValueSizeConstraint object at 0x28cf9abde48 consts 1, 1> encoding iso-8859-1>, <TagSet object at 0x28cfadc6e80 tags 128:0:7>: <TimeStamp schema object at 0x28cfadc6e10 tagSet <TagSet object at 0x28cfadc6e80 tags 128:0:7> subtypeSpec <ValueSizeConstraint object at 0x28cf9aafda0 consts 9, 10> encoding iso-8859-1>, <TagSet object at 0x28cfadc6fd0 tags 128:0:8>: <TimeStamp schema object at 0x28cfadc6f60 tagSet <TagSet object at 0x28cfadc6fd0 tags 128:0:8> subtypeSpec <ValueSizeConstraint object at 0x28cf9aafda0 consts 9, 10> encoding iso-8859-1>, <TagSet object at 0x28cfadcf160 tags 128:0:9>: <TimeStamp schema object at 0x28cfadcf0f0 tagSet <TagSet object at 0x28cfadcf160 tags 128:0:9> subtypeSpec <ValueSizeConstraint object at 0x28cf9aafda0 consts 9, 10> encoding iso-8859-1>, <TagSet object at 0x28cfadcf2b0 tags 128:0:10>: <CallDuration schema object at 0x28cfadcf240 tagSet <TagSet object at 0x28cfadcf2b0 tags 128:0:10>>, <TagSet object at 0x28cfadcf400 tags 128:0:12>: <CauseForTerm schema object at 0x28cfadcf390 tagSet <TagSet object at 0x28cfadcf400 tags 128:0:12> namedValues <NamedValues object 0x28cf9af07b8 enums normalRelease=0, partialRecord=1...unknownOrUnreachableLCSClient=58>>, <TagSet object at 0x28cfadcf518 tags 128:32:13>: <Diagnostics schema object at 0x28cfadcf4a8 tagSet=<TagSet object at 0x28cfadcf518 tags 128:32:13> subtypeSpec=<ConstraintsIntersection object at 0x28cf9aa7668> componentType=<NamedTypes object at 0x28cf9afb128 types <NamedType object at 0x28cf9af0d30 type gsm0408Cause=<Integer schema object at 0x28cf9af0dd8 tagSet <TagSet object at 0x28cf9af0e10 tags 128:0:0>>>, <NamedType object at 0x28cf9af0d68 type gsm0902MapErrorValue=<Integer schema object at 0x28cf9af0eb8 tagSet <TagSet object at 0x28cf9af0ef0 tags 128:0:1>>>, <NamedType object at 0x28cf9af0e48 type ccittQ767Cause=<Integer schema object at 0x28cf9af0f98 tagSet <TagSet object at 0x28cf9af0fd0 tags 128:0:2>>>, <NamedType object at 0x28cf9af0f28 type networkSpecificCause=<ManagementExtension schema object at 0x28cf9afb0b8 tagSet=<TagSet object at 0x28cf9afb0f0 tags 128:32:3> subtypeSpec=<ConstraintsIntersection object at 0x28cf9aa7160> componentType=<NamedTypes object at 0x28cf9af09e8 types <NamedType object at 0x28cf9af0898 type identifier=<ObjectIdentifier schema object at 0x28cf9af0860 tagSet <TagSet object at 0x28cf9a9a780 tags 0:0:6>>>, <DefaultedNamedType object at 0x28cf9af0978 type significance=<Boolean value object at 0x28cf9af08d0 tagSet <TagSet object at 0x28cf9af09b0 tags 128:0:1> subtypeSpec <ConstraintsIntersection object at 0x28cf9a95fd0 consts <SingleValueConstraint object at 0x28cf9a95f28 consts 0, 1>> namedValues <NamedValues object 0x28cf9a9a048 enums False=0, True=1> payload [False]>>, <NamedType object at 0x28cf9af0908 type information=<OctetString schema object at 0x28cf9af0a58 tagSet <TagSet object at 0x28cf9af0a90 tags 128:0:2> encoding iso-8859-1>>> sizeSpec=<ConstraintsIntersection object at 0x28cf9aa7198>>>, <NamedType object at 0x28cf9afb048 type manufacturerSpecificCause=<ManagementExtension schema object at 0x28cf9afb198 tagSet=<TagSet object at 0x28cf9afb1d0 tags 128:32:4> subtypeSpec=<ConstraintsIntersection object at 0x28cf9aa7160> componentType=<NamedTypes object at 0x28cf9af09e8 types <NamedType object at 0x28cf9af0898 type identifier=<ObjectIdentifier schema object at 0x28cf9af0860 tagSet <TagSet object at 0x28cf9a9a780 tags 0:0:6>>>, <DefaultedNamedType object at 0x28cf9af0978 type significance=<Boolean value object at 0x28cf9af08d0 tagSet <TagSet object at 0x28cf9af09b0 tags 128:0:1> subtypeSpec <ConstraintsIntersection object at 0x28cf9a95fd0 consts <SingleValueConstraint object at 0x28cf9a95f28 consts 0, 1>> namedValues <NamedValues object 0x28cf9a9a048 enums False=0, True=1> payload [False]>>, <NamedType object at 0x28cf9af0908 type information=<OctetString schema object at 0x28cf9af0a58 tagSet <TagSet object at 0x28cf9af0a90 tags 128:0:2> encoding iso-8859-1>>> sizeSpec=<ConstraintsIntersection object at 0x28cf9aa7198>>>> sizeSpec=<ConstraintsIntersection object at 0x28cf9aa76d8 consts <ValueSizeConstraint object at 0x28cf9aa76a0 consts 1, 1>>>, <TagSet object at 0x28cfadcf6a0 tags 128:0:14>: <CallReference schema object at 0x28cfadcf630 tagSet <TagSet object at 0x28cfadcf6a0 tags 128:0:14> subtypeSpec <ValueSizeConstraint object at 0x28cf9b2fe48 consts 1, 8> encoding iso-8859-1>, <TagSet object at 0x28cfadcf7f0 tags 128:0:15>: <SequenceNumber schema object at 0x28cfadcf780 tagSet <TagSet object at 0x28cfadcf7f0 tags 128:0:15>>, <TagSet object at 0x28cfadcf940 tags 128:32:16>: <ManagementExtensions value object at 0x28cfadcf8d0 tagSet=<TagSet object at 0x28cfadcf940 tags 128:32:16> subtypeSpec=<ConstraintsIntersection object at 0x28cf9a9ae48> componentType=<ManagementExtension schema object at 0x28cf9b2f358 tagSet=<TagSet object at 0x28cf9aa7080 tags 0:32:16> subtypeSpec=<ConstraintsIntersection object at 0x28cf9aa7160> componentType=<NamedTypes object at 0x28cf9af09e8 types <NamedType object at 0x28cf9af0898 type identifier=<ObjectIdentifier schema object at 0x28cf9af0860 tagSet <TagSet object at 0x28cf9a9a780 tags 0:0:6>>>, <DefaultedNamedType object at 0x28cf9af0978 type significance=<Boolean value object at 0x28cf9af08d0 tagSet <TagSet object at 0x28cf9af09b0 tags 128:0:1> subtypeSpec <ConstraintsIntersection object at 0x28cf9a95fd0 consts <SingleValueConstraint object at 0x28cf9a95f28 consts 0, 1>> namedValues <NamedValues object 0x28cf9a9a048 enums False=0, True=1> payload [False]>>, <NamedType object at 0x28cf9af0908 type information=<OctetString schema object at 0x28cf9af0a58 tagSet <TagSet object at 0x28cf9af0a90 tags 128:0:2> encoding iso-8859-1>>> sizeSpec=<ConstraintsIntersection object at 0x28cf9aa7198>> sizeSpec=<ConstraintsIntersection object at 0x28cf9a9ae80>>, <TagSet object at 0x28cfadcfa90 tags 128:0:23>: <PartialRecordType schema object at 0x28cfadcfa20 tagSet <TagSet object at 0x28cfadcfa90 tags 128:0:23> namedValues <NamedValues object 0x28cf9ab62e8 enums timeLimit=0, serviceChange=1, lo...CS1Shortnumber=26, roLinkFail=27>>, <TagSet object at 0x28cfadcfba8 tags 128:32:130>: <BasicServiceCode schema object at 0x28cfadcfb38 tagSet=<TagSet object at 0x28cfadcfba8 tags 128:32:130> subtypeSpec=<ConstraintsIntersection object at 0x28cf9aa7668> componentType=<NamedTypes object at 0x28cf9ac7940 types <NamedType object at 0x28cf9abd630 type bearerService=<BearerServiceCode schema object at 0x28cf9abd390 tagSet <TagSet object at 0x28cf9abd2e8 tags 128:0:2> subtypeSpec <ValueSizeConstraint object at 0x28cf9abd710 consts 1, 1> encoding iso-8859-1>>, <NamedType object at 0x28cf9abd668 type teleservice=<TeleserviceCode schema object at 0x28cf9abd198 tagSet <TagSet object at 0x28cf9abd080 tags 128:0:3> subtypeSpec <ValueSizeConstraint object at 0x28cf9abde10 consts 1, 1> encoding iso-8859-1>>, <NamedType object at 0x28cf9abd1d0 type isdnService=<IsdnServiceCode schema object at 0x28cf9ac7550 tagSet <TagSet object at 0x28cf9ac7518 tags 128:0:4> subtypeSpec <ValueSizeConstraint object at 0x28cf9abddd8 consts 1, 1> encoding iso-8859-1>>> sizeSpec=<ConstraintsIntersection object at 0x28cf9aa76d8 consts <ValueSizeConstraint object at 0x28cf9aa76a0 consts 1, 1>>>, <TagSet object at 0x28cfadcfd30 tags 128:32:133>: <AdditionalChgInfo value object at 0x28cfadcfcc0 tagSet=<TagSet object at 0x28cfadcfd30 tags 128:32:133> subtypeSpec=<ConstraintsIntersection object at 0x28cf9aa7160> componentType=<NamedTypes object at 0x28cf9ac2208 types <OptionalNamedType object at 0x28cf9ac2860 type chargeIndicator=<ChargeIndicator schema object at 0x28cf9ac2908 tagSet <TagSet object at 0x28cf9ac2240 tags 128:0:0> namedValues <NamedValues object 0x28cf9ac2b38 enums noIndication=0, noCharge=1, charge=2>>>, <OptionalNamedType object at 0x28cf9ac2898 type chargeParameters=<OctetString schema object at 0x28cf9ac20b8 tagSet <TagSet object at 0x28cf9abd6a0 tags 128:0:1> encoding iso-8859-1>>> sizeSpec=<ConstraintsIntersection object at 0x28cf9aa7198>>, <TagSet object at 0x28cfadcfe80 tags 128:0:138>: <Null schema object at 0x28cfadcfe10 tagSet <TagSet object at 0x28cfadcfe80 tags 128:0:138> subtypeSpec <ConstraintsIntersection object at 0x28cf9a9a6d8 consts <SingleValueConstraint object at 0x28cf9a9a5f8 consts b''>> encoding iso-8859-1>, <TagSet object at 0x28cfadcffd0 tags 128:0:142>: <OriginalCalledNumber schema object at 0x28cfadcff60 tagSet <TagSet object at 0x28cfadcffd0 tags 128:0:142> encoding iso-8859-1>, <TagSet object at 0x28cfadd4160 tags 128:0:145>: <ChargeAreaCode schema object at 0x28cfadd40f0 tagSet <TagSet object at 0x28cfadd4160 tags 128:0:145> subtypeSpec <ValueSizeConstraint object at 0x28cf9b2f470 consts 1, 3> encoding iso-8859-1>, <TagSet object at 0x28cfadd42b0 tags 128:0:159>: <RateIndication schema object at 0x28cfadd4240 tagSet <TagSet object at 0x28cfadd42b0 tags 128:0:159> subtypeSpec <ValueSizeConstraint object at 0x28cf9ec8ef0 consts 1, 1> encoding iso-8859-1>, <TagSet object at 0x28cfadd4400 tags 128:0:166>: <MSCCIC schema object at 0x28cfadd4390 tagSet <TagSet object at 0x28cfadd4400 tags 128:0:166>>, <TagSet object at 0x28cfadd4550 tags 128:0:167>: <MSCCIC schema object at 0x28cfadd44e0 tagSet <TagSet object at 0x28cfadd4550 tags 128:0:167>>, <TagSet object at 0x28cfadd46a0 tags 128:0:168>: <MSCId schema object at 0x28cfadd4630 tagSet <TagSet object at 0x28cfadd46a0 tags 128:0:168> subtypeSpec <ValueSizeConstraint object at 0x28cf9ab6128 consts 3, 3> encoding iso-8859-1>, <TagSet object at 0x28cfadd47f0 tags 128:0:170>: <EmlppPriority schema object at 0x28cfadd4780 tagSet <TagSet object at 0x28cfadd47f0 tags 128:0:170> subtypeSpec <ValueSizeConstraint object at 0x28cf9eef208 consts 1, 1> encoding iso-8859-1>, <TagSet object at 0x28cfadd4940 tags 128:0:174>: <EASubscriberInfo schema object at 0x28cfadd48d0 tagSet <TagSet object at 0x28cfadd4940 tags 128:0:174> subtypeSpec <ValueSizeConstraint object at 0x28cf9eeff28 consts 3, 3> encoding iso-8859-1>, <TagSet object at 0x28cfadd4a90 tags 128:0:175>: <SelectedCIC schema object at 0x28cfadd4a20 tagSet <TagSet object at 0x28cfadd4a90 tags 128:0:175> subtypeSpec <ValueSizeConstraint object at 0x28cf9eefeb8 consts 3, 3> encoding iso-8859-1>, <TagSet object at 0x28cfadd4be0 tags 128:0:180>: <PortedFlag schema object at 0x28cfadd4b70 tagSet <TagSet object at 0x28cfadd4be0 tags 128:0:180> namedValues <NamedValues object 0x28cf9bccda0 enums numberNotPorted=0, numberPortedIn=1, numberPortedOut=2>>, <TagSet object at 0x28cfadd4d30 tags 128:0:195>: <CUGOutgoingAccessIndicator schema object at 0x28cfadd4cc0 tagSet <TagSet object at 0x28cfadd4d30 tags 128:0:195> namedValues <NamedValues object 0x28cf9b1fbe0 enums notCUGCall=0, cUGCall=1>>, <TagSet object at 0x28cfadd4e80 tags 128:0:196>: <CUGInterlockCode schema object at 0x28cfadd4e10 tagSet <TagSet object at 0x28cfadd4e80 tags 128:0:196> subtypeSpec <ValueSizeConstraint object at 0x28cf9b1fb70 consts 4, 4> encoding iso-8859-1>, <TagSet object at 0x28cfadd4fd0 tags 128:0:197>: <CUGIncomingAccessUsed schema object at 0x28cfadd4f60 tagSet <TagSet object at 0x28cfadd4fd0 tags 128:0:197> namedValues <NamedValues object 0x28cf9b1fa90 enums callInTheSameCUGGroup=0, callNotInTheSameCUGGroup=1>>, <TagSet object at 0x28cfadd6160 tags 128:0:198>: <RouteAttribute schema object at 0x28cfadd60f0 tagSet <TagSet object at 0x28cfadd6160 tags 128:0:198> namedValues <NamedValues object 0x28cfa3c4908 enums cas=0, tup=1, isup=2, pra=3, bicc=4, sip=5, others=255>>, <TagSet object at 0x28cfadd62b0 tags 128:0:199>: <RouteAttribute schema object at 0x28cfadd6240 tagSet <TagSet object at 0x28cfadd62b0 tags 128:0:199> namedValues <NamedValues object 0x28cfa3c4908 enums cas=0, tup=1, isup=2, pra=3, bicc=4, sip=5, others=255>>, <TagSet object at 0x28cfadd6400 tags 128:0:200>: <NetworkCallReference schema object at 0x28cfadd6390 tagSet <TagSet object at 0x28cfadd6400 tags 128:0:200> subtypeSpec <ValueSizeConstraint object at 0x28cf9b12828 consts 1, 8> encoding iso-8859-1>, <TagSet object at 0x28cfadd6550 tags 128:0:201>: <TimeStamp schema object at 0x28cfadd64e0 tagSet <TagSet object at 0x28cfadd6550 tags 128:0:201> subtypeSpec <ValueSizeConstraint object at 0x28cf9aafda0 consts 9, 10> encoding iso-8859-1>, <TagSet object at 0x28cfadd66a0 tags 128:0:202>: <TimeStamp schema object at 0x28cfadd6630 tagSet <TagSet object at 0x28cfadd66a0 tags 128:0:202> subtypeSpec <ValueSizeConstraint object at 0x28cf9aafda0 consts 9, 10> encoding iso-8859-1>, <TagSet object at 0x28cfadd67f0 tags 128:0:203>: <VoiceIndicator schema object at 0x28cfadd6780 tagSet <TagSet object at 0x28cfadd67f0 tags 128:0:203> namedValues <NamedValues object 0x28cf9eef0b8 enums sendToneByLocalMsc=0, sendToneByOtherMsc=1, voiceNoIndication=3>>, <TagSet object at 0x28cfadd6940 tags 128:0:204>: <BCategory schema object at 0x28cfadd68d0 tagSet <TagSet object at 0x28cfadd6940 tags 128:0:204> namedValues <NamedValues object 0x28cf9abdfd0 enums subscriberFree=0, subscriberBusy=1, subscriberNoIndication=3>>, <TagSet object at 0x28cfadd6a90 tags 128:0:205>: <CallType schema object at 0x28cfadd6a20 tagSet <TagSet object at 0x28cfadd6a90 tags 128:0:205> namedValues <NamedValues object 0x28cf9eef198 enums unknown=0, internal=1, incoming=2, outgoing=3, tandem=4>>, <TagSet object at 0x28cfadd6be0 tags 128:0:208>: <ChargePulseNumber schema object at 0x28cfadd6b70 tagSet <TagSet object at 0x28cfadd6be0 tags 128:0:208>>, <TagSet object at 0x28cfadd6d30 tags 128:0:218>: <DisconnectParty schema object at 0x28cfadd6cc0 tagSet <TagSet object at 0x28cfadd6d30 tags 128:0:218> namedValues <NamedValues object 0x28cf9eeff98 enums unknowparty=0, callingPartyRelea...PartyRelease=2, networkRelease=3>>, <TagSet object at 0x28cfadd6e80 tags 128:0:219>: <ChargePulseNumber schema object at 0x28cfadd6e10 tagSet <TagSet object at 0x28cfadd6e80 tags 128:0:219>>, <TagSet object at 0x28cfadd6fd0 tags 128:0:223>: <NetworkOperatorId schema object at 0x28cfadd6f60 tagSet <TagSet object at 0x28cfadd6fd0 tags 128:0:223> subtypeSpec <ValueSizeConstraint object at 0x28cf9b2f160 consts 1, 1> encoding iso-8859-1>, <TagSet object at 0x28cfaddb160 tags 128:0:225>: <AudioDataType schema object at 0x28cfaddb0f0 tagSet <TagSet object at 0x28cfaddb160 tags 128:0:225> namedValues <NamedValues object 0x28cf9abdd30 enums audio=0, fax=1>>, <TagSet object at 0x28cfaddb2b0 tags 128:0:232>: <RecordNumber schema object at 0x28cfaddb240 tagSet <TagSet object at 0x28cfaddb2b0 tags 128:0:232>>, <TagSet object at 0x28cfaddb400 tags 128:32:236>: <PartyRelCause schema object at 0x28cfaddb390 tagSet=<TagSet object at 0x28cfaddb400 tags 128:32:236> subtypeSpec=<ConstraintsIntersection object at 0x28cf9aa7160> componentType=<NamedTypes object at 0x28cf9be4198 types <NamedType object at 0x28cf9bccf28 type releaseParty=<ReleaseParty schema object at 0x28cf9be40b8 tagSet <TagSet object at 0x28cf9be4128 tags 128:0:0> namedValues <NamedValues object 0x28cf9bcceb8 enums bssmap=0, ranap=1, dtap=2, isup=3, relbutt=255>>>, <NamedType object at 0x28cf9bccfd0 type releaseCause=<Integer schema object at 0x28cf9be4208 tagSet <TagSet object at 0x28cf9be4278 tags 128:0:1>>>> sizeSpec=<ConstraintsIntersection object at 0x28cf9aa7198>>, <TagSet object at 0x28cfaddb550 tags 128:0:237>: <ChargeLevel schema object at 0x28cfaddb4e0 tagSet <TagSet object at 0x28cfaddb550 tags 128:0:237> namedValues <NamedValues object 0x28cf9ac77f0 enums chargeBySecond=0, chargeBy10ms=1>>, <TagSet object at 0x28cfaddb6a0 tags 128:0:238>: <LocationNumber schema object at 0x28cfaddb630 tagSet <TagSet object at 0x28cfaddb6a0 tags 128:0:238> encoding iso-8859-1>, <TagSet object at 0x28cfaddb7f0 tags 128:0:241>: <LocationNumberNai schema object at 0x28cfaddb780 tagSet <TagSet object at 0x28cfaddb7f0 tags 128:0:241> namedValues <NamedValues object 0x28cf9be4588 enums iAddressInfoNull=0, iUserNumber=...Number=3, iInternationalNumber=4>>, <TagSet object at 0x28cfaddb940 tags 128:0:245>: <TranslatedNumber schema object at 0x28cfaddb8d0 tagSet <TagSet object at 0x28cfaddb940 tags 128:0:245> encoding iso-8859-1>, <TagSet object at 0x28cfaddba90 tags 128:0:251>: <CmnFlag schema object at 0x28cfaddba20 tagSet <TagSet object at 0x28cfaddba90 tags 128:0:251> namedValues <NamedValues object 0x28cfadc62b0 enums notCmnCall=0, isCmnCall=1>>, <TagSet object at 0x28cfaddbbe0 tags 128:0:252>: <IMSCHARGE schema object at 0x28cfaddbb70 tagSet <TagSet object at 0x28cfaddbbe0 tags 128:0:252> subtypeSpec <ValueSizeConstraint object at 0x28cfa3c4a20 consts 128, 128> encoding iso-8859-1>, <TagSet object at 0x28cfaddbd30 tags 128:0:253>: <IMSCHARGE schema object at 0x28cfaddbcc0 tagSet <TagSet object at 0x28cfaddbd30 tags 128:0:253> subtypeSpec <ValueSizeConstraint object at 0x28cfa3c4a20 consts 128, 128> encoding iso-8859-1>, <TagSet object at 0x28cfaddbe80 tags 128:0:254>: <IMSCHARGE schema object at 0x28cfaddbe10 tagSet <TagSet object at 0x28cfaddbe80 tags 128:0:254> subtypeSpec <ValueSizeConstraint object at 0x28cfa3c4a20 consts 128, 128> encoding iso-8859-1>, <TagSet object at 0x28cfaddbfd0 tags 128:0:255>: <PortedFlag schema object at 0x28cfaddbf60 tagSet <TagSet object at 0x28cfaddbfd0 tags 128:0:255> namedValues <NamedValues object 0x28cf9bccda0 enums numberNotPorted=0, numberPortedIn=1, numberPortedOut=2>>, <TagSet object at 0x28cfaddf160 tags 128:0:256>: <LocationRoutingNumber schema object at 0x28cfaddf0f0 tagSet <TagSet object at 0x28cfaddf160 tags 128:0:256> encoding iso-8859-1>, <TagSet object at 0x28cfaddf2b0 tags 128:0:258>: <IntermediateChargingInd schema object at 0x28cfaddf240 tagSet <TagSet object at 0x28cfaddf2b0 tags 128:0:258> namedValues <NamedValues object 0x28cf9be45f8 enums sINGLE-BILL=0, fIRST-BILL=1, iNTERMEDIATE-BILL=2, lAST-BILL=3>>, <TagSet object at 0x28cfaddf400 tags 128:0:261>: <MscOutgoingROUTENumber schema object at 0x28cfaddf390 tagSet <TagSet object at 0x28cfaddf400 tags 128:0:261> subtypeSpec <ValueRangeConstraint object at 0x28cf9be4828 consts 0, 65535>>, <TagSet object at 0x28cfaddf550 tags 128:0:262>: <MscIncomingROUTENumber schema object at 0x28cfaddf4e0 tagSet <TagSet object at 0x28cfaddf550 tags 128:0:262> subtypeSpec <ValueRangeConstraint object at 0x28cf9be4710 consts 0, 65535>>, <TagSet object at 0x28cfaddf6a0 tags 128:0:266>: <DrcCallNumber schema object at 0x28cfaddf630 tagSet <TagSet object at 0x28cfaddf6a0 tags 128:0:266> encoding iso-8859-1>, <TagSet object at 0x28cfaddf7f0 tags 128:0:267>: <DrcCallNumber schema object at 0x28cfaddf780 tagSet <TagSet object at 0x28cfaddf7f0 tags 128:0:267> encoding iso-8859-1>}>

Now, is it possible to fix the asn spec given the traceback? Or maybe there is an option to ignore records with unknown tags?

etingof commented 5 years ago

My guess would be one of those structures in the CallEventRecord which does not have tag 160 defined, though it is present in the serialization.

The other way to approach this is to enable pyasn1 debugging:

from pyasn1 import debug

debug.setLogger(debug.Debug('all'))

It will reveal more information on the decoding process.

It should be doable to define the missing element in the ASN.1 spec to get the decoder going over it.

You can try to set defaultErrorState to stDumpRawValue, though I am not certain it won't fail somewhere later along the way.

kaxap commented 5 years ago

So the first approach with debug is really slow and it's still executing (has been executing for last ~20 hours).

The second approach with setting defaultErrorState to stDumpRawValue raises the following exception:

Traceback (most recent call last):
  File "/home/r/dev/cdr-py/run2.py", line 12, in <module>
    data = ber_decoder(bytes_read, asn1Spec=CallEventDataFile())
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 1498, in __call__
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 608, in valueDecoder
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 1498, in __call__
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 694, in valueDecoder
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 1498, in __call__
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 939, in valueDecoder
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 1498, in __call__
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 608, in valueDecoder
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 1498, in __call__
  File "/home/r/Anaconda3/lib/site-packages/pyasn1-0.4.6-py3.6.egg/pyasn1/codec/ber/decoder.py", line 1015, in valueDecoder
AttributeError: 'TagMap' object has no attribute 'tagSet'

while executing the following code:

from cdr import CallEventDataFile
from pyasn1.codec.native.encoder import encode
from pyasn1.codec.ber.decoder import decode as ber_decoder, stDumpRawValue

bytes_read = open("test-data/1546606507938713068", "rb").read()
ber_decoder.defaultErrorState = stDumpRawValue
data = ber_decoder(bytes_read, asn1Spec=CallEventDataFile())
etingof commented 5 years ago

Well, the first approach is meant for a single run, to figure out what's missing. That's not for production use of course.

Anyway, let's try #152 with the second approach? If it still does not help, I'd appreciate a reproducer e.g. the test data blob that causes decoder failure.

kaxap commented 5 years ago

I am not using it in production, it took 50+ hours to process a single 10MB ANS.1 encoded file. Anyways, I was able to find the missing ASN.1 notation by looking at local variables at the exception time.

I will try #152 with the missing annotation and see if it helps. Thank you for you help!

salano commented 3 years ago

@etingof I am having the same issue. How you interpret the debugged information. Below is my debug info.

2020-11-11 16:41:38,361 pyasn1: decoder called at scope with state 0, working with up to 1088 octets of substrate: 00000: 00 00 04 40 00 00 41 E8 E8 94 56 A1 80 94 56 A1 00016: 80 FF FF FF FF FF FF FF FF 01 FF FF FF 13 FF FF 00032: FF 00 00 00 00 20 FF FF FF FF FF FF FF FF FF FF 00048: FF FF FF FF FF FF FF FF FF 80 00 01 00 FF FF FF 00064: FF 03 FA E8 46 00 BF 53 80 80 01 53 82 06 49 4E 00080: 56 49 54 45 83 01 00 A4 80 81 32 64 65 6E 63 6F 00096: 30 30 6D 74 61 73 2D 52 46 52 4F 2E 69 6D 73 2E 00112: 6D 6E 63 30 34 30 2E 6D 63 63 33 31 31 2E 33 67 00128: 70 70 6E 65 74 77 6F 72 6B 2E 6F 72 67 00 00 85 00144: 7F 64 65 6E 63 6F 30 30 6D 74 61 73 2D 52 46 52 00160: 4F 2E 69 6D 73 2E 6D 6E 63 30 34 30 2E 6D 63 63 00176: 33 31 31 2E 33 67 70 70 6E 65 74 77 6F 72 6B 2E 00192: 6F 72 67 3B 33 37 32 34 34 31 3B 38 36 31 34 32 00208: 30 38 38 33 3B 31 36 38 3B 37 37 3B 30 3B 73 69 00224: 70 3A 2B 31 37 32 30 37 33 34 30 39 39 38 40 69 00240: 6D 73 2E 6D 6E 63 30 34 30 2E 6D 63 63 33 31 31 00256: 2E 33 67 70 70 6E 65 74 77 6F 72 6B 2E 6F 72 67 00272: A6 80 80 32 73 69 70 3A 2B 31 37 32 30 37 33 34 00288: 30 39 39 38 40 69 6D 73 2E 6D 6E 63 30 34 30 2E 00304: 6D 63 63 33 31 31 2E 33 67 70 70 6E 65 74 77 6F 00320: 72 6B 2E 6F 72 67 81 10 74 65 6C 3A 2B 31 37 32 00336: 30 37 33 34 30 39 39 38 00 00 A7 80 80 3D 73 69 00352: 70 3A 2B 31 35 32 30 34 37 39 30 39 39 39 40 69 00368: 6D 73 2E 6D 6E 63 30 34 30 2E 6D 63 63 33 31 31 00384: 2E 33 67 70 70 6E 65 74 77 6F 72 6B 2E 6F 72 67 00400: 3B 75 73 65 72 3D 70 68 6F 6E 65 00 00 89 09 20 00416: 09 08 21 39 36 2D 06 00 8B 07 14 14 09 08 15 29 00432: 24 8C 09 20 09 08 21 39 36 2D 06 00 8D 09 20 09 00448: 08 21 39 36 2D 06 00 AE 80 30 80 80 21 69 6D 73 00464: 2E 6D 6E 63 30 34 30 2E 6D 63 63 33 31 31 2E 33 00480: 67 70 70 6E 65 74 77 6F 72 6B 2E 6F 72 67 00 00 00496: 00 00 91 01 01 93 34 64 65 6E 63 6F 30 30 73 62 00512: 67 2E 69 6D 73 2E 6D 6E 63 30 34 30 2E 6D 63 63 00528: 33 31 31 2E 33 2D 31 35 39 39 2D 36 32 32 37 37 00544: 36 2D 32 30 31 39 34 2D 33 30 39 B5 80 30 80 80 00560: 09 20 09 08 21 39 36 2D 06 00 81 09 20 09 08 21 00576: 39 36 2D 06 00 86 01 5D 87 02 00 AB 00 00 00 00 00592: 97 03 34 30 33 B9 80 30 80 06 0D 04 00 7F 00 05 00608: 02 02 00 00 00 02 00 01 81 01 FF A2 80 A0 80 31 00624: 80 A0 80 0C 0F 2B 67 2E 65 72 69 63 73 73 6F 6E 00640: 2E 6D 6D 74 0C 29 75 72 6E 25 33 41 75 72 6E 2D 00656: 37 25 33 41 33 67 70 70 2D 73 65 72 76 69 63 65 00672: 2E 69 6D 73 2E 69 63 73 69 2E 6D 6D 74 65 6C 00 00688: 00 B6 80 83 01 00 87 32 73 69 70 3A 2B 31 37 32 00704: 30 37 33 34 30 39 39 38 40 69 6D 73 2E 6D 6E 63 00720: 30 34 30 2E 6D 63 63 33 31 31 2E 33 67 70 70 6E 00736: 65 74 77 6F 72 6B 2E 6F 72 67 88 01 00 8C 01 02 00752: 8E 32 73 69 70 3A 2B 31 37 32 30 37 33 34 30 39 00768: 39 38 40 69 6D 73 2E 6D 6E 63 30 34 30 2E 6D 63 00784: 63 33 31 31 2E 33 67 70 70 6E 65 74 77 6F 72 6B 00800: 2E 6F 72 67 B3 80 30 80 81 01 00 82 02 0A F0 00 00816: 00 00 00 00 00 00 00 00 00 89 01 01 00 00 00 00 00832: 00 00 9D 35 33 47 50 50 2D 45 2D 55 54 52 41 4E 00848: 2D 46 44 44 3B 75 74 72 61 6E 2D 63 65 6C 6C 2D 00864: 69 64 2D 33 67 70 70 3D 33 31 31 30 34 30 30 33 00880: 65 34 30 32 39 36 35 30 31 9E 11 31 32 2E 33 32 00896: 32 36 30 40 33 67 70 70 2E 6F 72 67 BF 1F 80 31 00912: 80 80 01 02 81 32 73 69 70 3A 2B 31 37 32 30 37 00928: 33 34 30 39 39 38 40 69 6D 73 2E 6D 6E 63 30 34 00944: 30 2E 6D 63 63 33 31 31 2E 33 67 70 70 6E 65 74 00960: 77 6F 72 6B 2E 6F 72 67 00 00 00 00 BF 65 80 80 00976: 6B 73 69 70 3A 35 32 30 34 37 39 30 39 39 39 3B 00992: 70 68 6F 6E 65 2D 63 6F 6E 74 65 78 74 3D 69 6D 01008: 73 2E 6D 6E 63 30 34 30 2E 6D 63 63 33 31 31 2E 01024: 33 67 70 70 6E 65 74 77 6F 72 6B 2E 6F 72 67 40 01040: 69 6D 73 2E 6D 6E 63 30 34 30 2E 6D 63 63 33 31 01056: 31 2E 33 67 70 70 6E 65 74 77 6F 72 6B 2E 6F 72 01072: 67 3B 75 73 65 72 3D 70 68 6F 6E 65 00 00 00 00 2020-11-11 16:41:38,361 pyasn1: tag decoded into <TagSet object at 0x7600668 tags 0:0:0>, decoding length 2020-11-11 16:41:38,361 pyasn1: value length decoded into 0, payload substrate is: 2020-11-11 16:41:38,362 pyasn1: codec chosen by ASN.1 spec, decoding as explicit tag 2020-11-11 16:41:38,362 pyasn1: codec chosen, decoding as failure

mikem23 commented 2 years ago

This looks very similar to #143 ?