search

P1sec / pycrate

A Python library to ease the development of encoders and decoders for various protocols and file formats; contains ASN.1 and CSN.1 compilers.
GNU Lesser General Public License v2.1
380 stars 130 forks source link

ASN1 Update (RRCLTE) Fails due to missing key GLOBAL.COMP['NS']['obj'][typeref] #96

Closed aliasgherman closed 3 years ago

aliasgherman commented 3 years ago

Steps Followed:

  1. Download 3GPP Rel16 LTE RRC specifications [3GPP TS 36.331 V16.0.0 (2020-03)]. File attached too.
  2. Converted it into text format using MS Word and placed it into pycrate_asn1dir/3GPP_EUTRAN_RRC_36331/36331-g00.txt
  3. Modified the script in the same directory to operate on the file 36331-g00.txt and created *.asn files
  4. Ran command python -m pycrate_asn1c.asnproc
  5. Command fails while trying to recompile the LTE RRC asn.

Please can you help me understand more on the dependencies for updating the ASN files. Specifically the 3GPP ones. 36331-g00.txt

Error :

[GEN] RRCLTE [proc] starting with ASN.1 specification: 3GPP_EUTRAN_RRC_36331 [proc] [EUTRA-UE-Variables.asn] module EUTRA-UE-Variables (oid: []): 31 ASN.1 assignments found [proc] [EUTRA-Sidelink-Preconf.asn] module EUTRA-Sidelink-Preconf (oid: []): 28 ASN.1 assignments found [proc] [NBIOT-RRC-Definitions.asn] module NBIOT-RRC-Definitions (oid: []): 341 ASN.1 assignments found [proc] [EUTRA-RRC-Definitions.asn] module EUTRA-RRC-Definitions (oid: []): 2269 ASN.1 assignments found [proc] [PC5-RRC-Definitions.asn] module PC5-RRC-Definitions (oid: []): 7 ASN.1 assignments found [proc] [EUTRA-InterNodeDefinitions.asn] module EUTRA-InterNodeDefinitions (oid: []): 72 ASN.1 assignments found [proc] [NBIOT-InterNodeDefinitions.asn] module NBIOT-InterNodeDefinitions (oid: []): 17 ASN.1 assignments found [proc] [NBIOT-UE-Variables.asn] module NBIOT-UE-Variables (oid: []): 6 ASN.1 assignments found --- compilation cycle --- Traceback (most recent call last): File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 1901, in _parse_type refmod = GLOBAL.COMP['NS']['obj'][typeref] KeyError: 'FailureInformation2-r16'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/aamhabiby/Desktop/software/miniconda3/envs/condaml2/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/aamhabiby/Desktop/software/miniconda3/envs/condaml2/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnproc.py", line 1272, in generate_all() File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnproc.py", line 1254, in generate_all compile_spec(kwargs) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnproc.py", line 88, in compile_spec compile_text(spec_texts, kwargs) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnproc.py", line 257, in compile_text compile_modules(remain) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnproc.py", line 940, in compile_modules ObjNew = asnobj_compile(Obj) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnproc.py", line 874, in asnobj_compile text = Obj.parse_def(Obj._text_def) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 1682, in parse_def text = self._parse_cont(text) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 2246, in _parse_cont text = getattr(self, self._PARSE_CONT_DISPATCH[self._type])(text) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 3205, in _parse_cont_choice self.parse_cont_comps(text_comps, self._type, False) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 3395, in parse_cont_comps comp_t = Comp.parse_def(comp_t) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 1682, in parse_def text = self._parse_cont(text) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 2246, in _parse_cont text = getattr(self, self._PARSE_CONT_DISPATCH[self._type])(text) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 3205, in _parse_cont_choice self.parse_cont_comps(text_comps, self._type, False) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 3395, in parse_cont_comps comp_t = Comp.parse_def(comp_t) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 1682, in parse_def text = self._parse_cont(text) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 2246, in _parse_cont text = getattr(self, self._PARSE_CONT_DISPATCH[self._type])(text) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 3205, in _parse_cont_choice self.__parse_cont_comps(text_comps, self._type, False) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 3395, in __parse_cont_comps comp_t = Comp.parse_def(comp_t) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 1680, in parse_def text = self._parse_type(text) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 1904, in _parse_type .format(self.fullname(), typeref))) pycrate_asn1c.err.ASN1ProcTextErr: UL-DCCH-MessageType.messageClassExtension.c2.failureInformation2-r16: ASN1RefType to FailureInformation2-r16, undefined

mitshell commented 3 years ago

You need to check the error log: pycrate_asn1c.err.ASN1ProcTextErr: UL-DCCH-MessageType.messageClassExtension.c2.failureInformation2-r16: ASN1RefType to FailureInformation2-r16, undefined Looking at your file, it seems there is a typo here at line 10022:

--- ASN1START

FailureInformation2-r16 ::=     SEQUENCE {

where it should be -- ASN1START. Or you patch the extraction script to handle this case, or you patch the txt file of the 3GPP TS.

When you make it working, do not hesitate to do a clean pull request. Thanks

aliasgherman commented 3 years ago

Hi Mitshell, Indeed you caught it right. I modified the extract.py script to handle similar cases and the script for updating went further however, it ended up with another error.

I have not yet tried to explore the code for pycrate yet so the exceptions seem alien to me. If you have any directions for me here then I would be glad to work on it. Else, I may give it another go later this week to understand more.

[proc] starting with ASN.1 specification: 3GPP_EUTRAN_RRC_36331 [proc] [EUTRA-UE-Variables.asn] module EUTRA-UE-Variables (oid: []): 31 ASN.1 assignments found [proc] [EUTRA-Sidelink-Preconf.asn] module EUTRA-Sidelink-Preconf (oid: []): 28 ASN.1 assignments found [proc] [NBIOT-RRC-Definitions.asn] module NBIOT-RRC-Definitions (oid: []): 341 ASN.1 assignments found [proc] [EUTRA-RRC-Definitions.asn] module EUTRA-RRC-Definitions (oid: []): 2271 ASN.1 assignments found [proc] [PC5-RRC-Definitions.asn] module PC5-RRC-Definitions (oid: []): 7 ASN.1 assignments found [proc] [EUTRA-InterNodeDefinitions.asn] module EUTRA-InterNodeDefinitions (oid: []): 72 ASN.1 assignments found [proc] [NBIOT-InterNodeDefinitions.asn] module NBIOT-InterNodeDefinitions (oid: []): 17 ASN.1 assignments found [proc] [NBIOT-UE-Variables.asn] module NBIOT-UE-Variables (oid: []): 6 ASN.1 assignments found --- compilation cycle --- --- verifications --- Traceback (most recent call last): File "/home/aamhabiby/Desktop/software/miniconda3/envs/condaml2/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/aamhabiby/Desktop/software/miniconda3/envs/condaml2/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnproc.py", line 1272, in generate_all() File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnproc.py", line 1254, in generate_all compile_spec(kwargs) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnproc.py", line 88, in compile_spec compile_text(spec_texts, kwargs) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnproc.py", line 290, in compile_text verify_modules(**kwargs) File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnproc.py", line 1029, in verify_modules void = O.get_cont() File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 1279, in get_cont self._verif_typeref() File "/home/aamhabiby/Desktop/software/pycrate/pycrate_asn1c/asnobj.py", line 1069, in _verif_typeref .format(self.fullname()))) pycrate_asn1c.err.ASN1ProcLinkErr: VarMeasReport.poolsTriggeredListNR-r16: typeref not yet compiled

p1-bmu commented 3 years ago

Except the issue with FailureInformation2-r16, this RRC LTE g00 release compiles fine on my laptop:

[proc] [3GPP_EUTRAN_RRC_36331_g00/EUTRA-UE-Variables.asn] module EUTRA-UE-Variables (oid: []): 31 ASN.1 assignments found
[proc] [3GPP_EUTRAN_RRC_36331_g00/EUTRA-Sidelink-Preconf.asn] module EUTRA-Sidelink-Preconf (oid: []): 28 ASN.1 assignments found
[proc] [3GPP_EUTRAN_RRC_36331_g00/NBIOT-RRC-Definitions.asn] module NBIOT-RRC-Definitions (oid: []): 341 ASN.1 assignments found
[proc] [3GPP_EUTRAN_RRC_36331_g00/EUTRA-RRC-Definitions.asn] module EUTRA-RRC-Definitions (oid: []): 2271 ASN.1 assignments found
[proc] [3GPP_EUTRAN_RRC_36331_g00/PC5-RRC-Definitions.asn] module PC5-RRC-Definitions (oid: []): 7 ASN.1 assignments found
[proc] [3GPP_EUTRAN_RRC_36331_g00/EUTRA-InterNodeDefinitions.asn] module EUTRA-InterNodeDefinitions (oid: []): 72 ASN.1 assignments found
[proc] [3GPP_EUTRAN_RRC_36331_g00/NBIOT-InterNodeDefinitions.asn] module NBIOT-InterNodeDefinitions (oid: []): 17 ASN.1 assignments found
[proc] [3GPP_EUTRAN_RRC_36331_g00/NBIOT-UE-Variables.asn] module NBIOT-UE-Variables (oid: []): 6 ASN.1 assignments found
--- compilation cycle ---
--- compilation cycle ---
--- verifications ---
[proc] ASN.1 modules processed: ['EUTRA-UE-Variables', 'EUTRA-Sidelink-Preconf', 'NBIOT-RRC-Definitions', 'EUTRA-RRC-Definitions', 'PC5-RRC-Definitions', 'EUTRA-InterNodeDefinitions', 'NBIOT-InterNodeDefinitions', 'NBIOT-UE-Variables']
[proc] ASN.1 objects compiled: 2561 types, 0 sets, 204 values
[proc] done

Did you change any other thing on your side ?

aliasgherman commented 3 years ago

Hi, I have attached the 3gpp_f00 file from my side.

  1. After your response, to eliminate other variables. I downloaded the file in first post of this issue.
  2. Hand modified the ASN Start header on line 10022
  3. Saved the file in directory.
  4. Ran the original extract.py script with just the filename changed
  5. Tried compiling using the command "python -m pycrate_asn1c.asnproc"
  6. Faced the same error as pasted above.

Then, I removed pycrate completely. Cloned the repo again, repeated the steps as listed above. The file compiles without issues. I am not sure if the previous failed verifications were the cause of this.

I am on Ubuntu 18.04 with anaconda installation.

36331-g00.txt

Closing the issue from my side.