Question: Encoding issue

satyabalaanusha commented 3 years ago

I am using pycrate with asn specifications as 3GPP 38.413 (from 3GPP_NR_NGAP_38413) to encode and decode the ngap packet on one side and on the other side there is similar C library asn1c which is used for asn encoding and decoding. While I am trying to send the encoded pdu session modify request message packet by using to_aper() function at the runtime and encoding the entire initiating message structure and sending it, it is not getting decoded by C library on the other side. But the decoded structure is the same on both sides and only after encoding that packet at pycrate, it is not getting decoded on the other side. And I am passing the nas pdu as byte array into the structure to encode. One more question I have is, are there any length restrictions on the parameters which are to be encoded?

The example structure that I am passing to to_aper() function is like this : (after filling appropriate values in this structure, passing it to to_aper() function):
{'procedureCode': , 'criticality': ' ', 'value': ('PDUSessionResourceModifyRequest', {'protocolIEs': [{'id': , 'criticality': 'reject', 'value': ('AMF-UE-NGAP-ID', )}, {'id': , 'criticality': ' ', 'value': ('RAN-UE-NGAP-ID', )}, {'id': , 'criticality': 'reject', 'value': ('PDUSessionResourceModifyListModReq', [{'pDUSessionID': , 'nAS-PDU': b' ', 'pDUSessionResourceModifyRequestTransfer': ('PDUSessionResourceModifyRequestTransfer', {'protocolIEs': [{'id': , 'criticality': 'reject', 'value': ('UL-NGU-UP-TNLModifyList', [{'uL-NGU-UP-TNLInformation': ('gTPTunnel', {'transportLayerAddress': ( , ), 'gTP-TEID': b' ' }), 'dL-NGU-UP-TNLInformation': ('gTPTunnel', {'transportLayerAddress': ( , ), 'gTP-TEID': b' '})}])}]})}])}]})}

p1-bmu commented 3 years ago

You need to give me all the information so that I can fully understand and eventually reproduce your issue, al least:

your python version
the commit level of the pycrate library you are using
which object is used for the encoding (NGAP_PDU_Descriptions.NGAP_PDU I presume)
the exact and complete Python value to be encoded in this object
the corresponding buffer generated by pycrate
the error or log message provided by asn1c when trying to decode the buffer
the buffer generated by asn1c for an similar value

Otherwise I cannot help you.

satyabalaanusha commented 3 years ago

Python version- 3.6.9
Latest commit used i.e, d03673ef1f3655de143ac12c42736d46b6379026

Code snippet used for encoding in pycrate


import json
from pycrate_asn1dir.NGAP import *

init_msg_pdu_modify_req = NGAP_PDU_Descriptions.InitiatingMessage

struct = {'procedureCode': 26, 'criticality': 'reject', 'value': ('PDUSessionResourceModifyRequest', {'protocolIEs': [{'id': 10, 'criticality': 'reject', 'value': ('AMF-UE-NGAP-ID', 1)}, {'id': 85, 'criticality': 'reject', 'value': ('RAN-UE-NGAP-ID', 0)}, {'id': 64, 'criticality': 'reject', 'value': ('PDUSessionResourceModifyListModReq', [{'pDUSessionID': 1, 'nAS-PDU': b'~\x02\x00\x00\x00\x00\x04~\x00h\x01\x00\x04.\x01\x01\xcb\x12\x01', 'pDUSessionResourceModifyRequestTransfer': ('PDUSessionResourceModifyRequestTransfer', {'protocolIEs': [{'id': 140, 'criticality': 'reject', 'value': ('UL-NGU-UP-TNLModifyList', [{'uL-NGU-UP-TNLInformation': ('gTPTunnel', {'transportLayerAddress': (32895, 16), 'gTP-TEID': b'\x00\x00\x02\xf0'}), 'dL-NGU-UP-TNLInformation': ('gTPTunnel', {'transportLayerAddress': (0, 1), 'gTP-TEID': b'\x01\xf0\x7f\x00'})}])}]})}])}]})}

init_msg_pdu_modify_req.set_val(struct) print(init_msg_pdu_modify_req.to_aper())


- Corresponding buffer generated by pycrate

b'\x1a\x00A\x00\x00\x03\x00\n\x00\x02\x00\x01\x00U\x00\x02\x00\x00\x00@\x00.\x00@\x01\x13~\x02\x00\x00\x00\x00\x04~\x00h\x01\x00\x04.\x01\x01\xcb\x12\x01\x16\x00\x00\x01\x00\x8c\x00\x0f\x00\x0f\x80\x7f\x00\x00\x02\xf0\x00\x00\x00\x01\xf0\x7f\x00'


- the error or log message provided by asn1c when trying to decode the buffer
![decode_error](https://user-images.githubusercontent.com/72131599/94782460-e12c3980-03e8-11eb-983d-cf7ac8136955.png)

- the buffer generated by asn1c for an similar value

b'\x1a\x00G\x00\x00\x03\x00\n\x00\x02\x00\x01\x00U\x00\x02\x00\x00\x00@\x004\x00@\x01\x13~\x02\x00\x00\x00\x00\x04~\x00h\x01\x00\x04.\x01\x01\xcb\x12\x01\x1c\x00\x00\x01\x00\x8c\x00\x15 \x0f\x80\x7f\x00\x00\x02\xf0\x00\x00\x01\x01\xf0\x7f\x00\x00\x02\xf0\x00\x00\x01'



This is the actual issue and even if I try to increase the nas pdu length in the form of octet string value of the struct in the above mentioned code snippet, then also I am facing this decode error. So further, wanted to know whether there are any specific length constraints on the parameters I am using.

p1-bmu commented 3 years ago

As you can see, the software generated with asn1c produces a longer buffer compared to pycrate. Even if it decodes well into pycrate, there are 6 more bytes, and some of the bytes related to the pDUSessionResourceModifyRequestTransfer components are not the same.

Digging into this, with a special focus on the PDUSessionResourceModifyRequestTransfer object:

In [18]: I = NGAP_PDU_Descriptions.InitiatingMessage                                                                                                          

In [19]: v = {'procedureCode': 26, 'criticality': 'reject', 'value': ('PDUSessionResourceModifyRequest', {'protocolIEs': [{'id': 10, 'criticality': 'reject', 'value': ('AMF-UE-NGAP-ID', 1)}, {'id': 85, 'criticality': 'reject', 'value': ('RAN-UE-NGAP-ID', 0)}, {'id': 64, 'criticality': 'reject', 'value': ('PDUSessionResourceModifyListModReq', [{'pDUSessionID': 1, 'nAS-PDU': b'~\x02\x00\x00\x00\x00\x04~\x00h\x01\x00\x04.\x01\x01\xcb\x12\x01', 'pDUSessionResourceModifyRequestTransfer': ('PDUSessionResourceModifyRequestTransfer', {'protocolIEs': [{'id': 140, 'criticality': 'reject', 'value': ('UL-NGU-UP-TNLModifyList', [{'uL-NGU-UP-TNLInformation': ('gTPTunnel', {'transportLayerAddress': (32895, 16), 'gTP-TEID': b'\x00\x00\x02\xf0'}), 'dL-NGU-UP-TNLInformation': ('gTPTunnel', {'transportLayerAddress': (0, 1), 'gTP-TEID': b'\x01\xf0\x7f\x00'})}])}]})}])}]})}                                      

In [20]: I.set_val(v)                                                                                                                                         

In [21]: buf = I.to_aper_ws()                                                                                                                                 

In [22]: len(buf)                                                                                                                                             
Out[22]: 68

In [23]: I.from_aper_ws(buf)                                                                                                                                  

In [24]: I.get_val() == v                                                                                                                                     
Out[24]: True

In [25]: print(I._struct.show())                                                                                                                              
### InitiatingMessage ###
 ### procedureCode ###
  <V : 26>
 ### criticality ###
  <I : 0>
 ### value ###
  <P : 0b000000>
  <C_form : 0 (short)>
  <C : 65>
  ### PDUSessionResourceModifyRequest ###
   <E : 0>
   ### protocolIEs ###
    <P : 0b0000000>
    <C : 3>
    ### _item_ ###
     ### id ###
      <V : 10>
     ### criticality ###
      <I : 0>
     ### value ###
      <P : 0b000000>
      <C_form : 0 (short)>
      <C : 2>
      ### AMF-UE-NGAP-ID ###
       <C : 0>
       <P : 0b00000>
       <V : 1>
    ### _item_ ###
     ### id ###
      <V : 85>
     ### criticality ###
      <I : 0>
     ### value ###
      <P : 0b000000>
      <C_form : 0 (short)>
      <C : 2>
      ### RAN-UE-NGAP-ID ###
       <C : 0>
       <P : 0b000000>
       <V : 0>
    ### _item_ ###
     ### id ###
      <V : 64>
     ### criticality ###
      <I : 0>
     ### value ###
      <P : 0b000000>
      <C_form : 0 (short)>
      <C : 46>
      ### PDUSessionResourceModifyListModReq ###
       <C : 0>
       ### _item_ ###
        <E : 0>
        <B : 0b10>
        ### pDUSessionID ###
         <P : 0b00000>
         <V : 1>
        ### nAS-PDU ###
         <C_form : 0 (short)>
         <C : 19>
         <V : 0x7e0200000000047e00680100042e0101cb1201>
        ### pDUSessionResourceModifyRequestTransfer ###
         <C_form : 0 (short)>
         <C : 22>
         ### _cont_pDUSessionResourceModifyRequestTransfer ###
          <E : 0>
          ### protocolIEs ###
           <P : 0b0000000>
           <C : 1>
           ### _item_ ###
            ### id ###
             <V : 140>
            ### criticality ###
             <I : 0>
            ### value ###
             <P : 0b000000>
             <C_form : 0 (short)>
             <C : 15>
             ### UL-NGU-UP-TNLModifyList ###
              <C : 0>
              ### _item_ ###
               <E : 0>
               <B : 0b0>
               ### uL-NGU-UP-TNLInformation ###
                <I : 0>
                ### gTPTunnel ###
                 <E : 0>
                 <B : 0b0>
                 ### transportLayerAddress ###
                  <E : 0>
                  <C : 15>
                  <V : 0b1000000001111111>
                 ### gTP-TEID ###
                  <V : 0x000002f0>
               ### dL-NGU-UP-TNLInformation ###
                <I : 0>
                ### gTPTunnel ###
                 <E : 0>
                 <B : 0b0>
                 ### transportLayerAddress ###
                  <E : 0>
                  <C : 0>
                  <P : 0b0000>
                  <V : 0b0>
                 ### gTP-TEID ###
                  <P : 0b0000000>
                  <V : 0x01f07f00>

The complete buffer generated by pycrate is 68 bytes. The complete structure associated with the buffer generated shows that the length prefix (called C) for the object UL-NGU-UP-TNLModifyList is 15 bytes, encoded into its short format. Everything looks fine to me, here.

Now looking at the buffer generated by your asn1c-based software:

In [26]: buf = b'\x1a\x00G\x00\x00\x03\x00\n\x00\x02\x00\x01\x00U\x00\x02\x00\x00\x00@\x004\x00@\x01\x13~\x02\x00\x00\x00\x00\x04~\x00h\x01\x00\x04.\x01\x01\xcb\x12\x01\x1c\x00\x00\x01\x00\x8c\x00\x15 \x0f\x80\x7f\x00\x00\x02\xf0\x00\x00\x01\x01\xf0\x7f\x00\x00\x02\xf0\x00\x00\x01'                         

In [27]: len(buf)                                                                                                                                             
Out[27]: 74

In [28]: I.from_aper_ws(buf)                                                                                                                                  

In [29]: I.get_val() == v                                                                                                                                     
Out[29]: True

In [30]: print(I._struct.show())                                                                                                                              
### InitiatingMessage ###
 ### procedureCode ###
  <V : 26>
 ### criticality ###
  <I : 0>
 ### value ###
  <P : 0b000000>
  <C_form : 0 (short)>
  <C : 71>
  ### PDUSessionResourceModifyRequest ###
   <E : 0>
   ### protocolIEs ###
    <P : 0b0000000>
    <C : 3>
    ### _item_ ###
     ### id ###
      <V : 10>
     ### criticality ###
      <I : 0>
     ### value ###
      <P : 0b000000>
      <C_form : 0 (short)>
      <C : 2>
      ### AMF-UE-NGAP-ID ###
       <C : 0>
       <P : 0b00000>
       <V : 1>
    ### _item_ ###
     ### id ###
      <V : 85>
     ### criticality ###
      <I : 0>
     ### value ###
      <P : 0b000000>
      <C_form : 0 (short)>
      <C : 2>
      ### RAN-UE-NGAP-ID ###
       <C : 0>
       <P : 0b000000>
       <V : 0>
    ### _item_ ###
     ### id ###
      <V : 64>
     ### criticality ###
      <I : 0>
     ### value ###
      <P : 0b000000>
      <C_form : 0 (short)>
      <C : 52>
      ### PDUSessionResourceModifyListModReq ###
       <C : 0>
       ### _item_ ###
        <E : 0>
        <B : 0b10>
        ### pDUSessionID ###
         <P : 0b00000>
         <V : 1>
        ### nAS-PDU ###
         <C_form : 0 (short)>
         <C : 19>
         <V : 0x7e0200000000047e00680100042e0101cb1201>
        ### pDUSessionResourceModifyRequestTransfer ###
         <C_form : 0 (short)>
         <C : 28>
         ### _cont_pDUSessionResourceModifyRequestTransfer ###
          <E : 0>
          ### protocolIEs ###
           <P : 0b0000000>
           <C : 1>
           ### _item_ ###
            ### id ###
             <V : 140>
            ### criticality ###
             <I : 0>
            ### value ###
             <P : 0b000000>
             <C_form : 0 (short)>
             <C : 21>
             ### UL-NGU-UP-TNLModifyList ###
              <C : 0>
              ### _item_ ###
               <E : 1>
               <B : 0b0>
               ### uL-NGU-UP-TNLInformation ###
                <I : 0>
                ### gTPTunnel ###
                 <E : 0>
                 <B : 0b0>
                 ### transportLayerAddress ###
                  <E : 0>
                  <C : 15>
                  <V : 0b1000000001111111>
                 ### gTP-TEID ###
                  <V : 0x000002f0>
               ### dL-NGU-UP-TNLInformation ###
                <I : 0>
                ### gTPTunnel ###
                 <E : 0>
                 <B : 0b0>
                 ### transportLayerAddress ###
                  <E : 0>
                  <C : 0>
                  <P : 0b0000>
                  <V : 0b0>
                 ### gTP-TEID ###
                  <P : 0b0000001>
                  <V : 0x01f07f00>
               <big : 0>
               <C : 0>
               <B : 0b0>

In [31]: len(I.to_aper())                                                                                                                                     
Out[31]: 68

As a 1st remark, you can see the buffer generated by asn1c is 74 bytes (and not 68): the trailing bytes are b'\x00\x02\xf0\x00\x00\x01' . Pycrate can still decode it back to the original value. However, if we look carefully at the decoded structure, we can see that the length count for UL-NGU-UP-TNLModifyList is 21 bytes here, and not 15. When pycrate decodes the buffer, it tries to decode further after the component dL-NGU-UP-TNLInformation of the object UL-NGU-UP-TNLModifyList, but the bitmap (last field called B) indicates no extension are to be decoded... The trailing bytes generated by asn1c are actually considered as garbage by pycrate.

To me, the root cause may be one of the following: 1) Your asn1c-based software does not use the same NGAP specification as pycrate (therefore some formal definition are not matching between both). Pycrate's NGAP is currently based on Rel. f60. Which NGAP release does your asn1c-based software uses ? 2) Maybe there is an issue in the way asn1c handles the encoding / decoding of the object UL-NGU-UP-TNLModifyList, which is a SEQUENCE contained within the OCTET STRING pDUSessionResourceModifyRequestTransfer. Maybe you should try to further debug what is done by asn1c, that produces those trailing bytes.

Please let me know if you come to any conclusion.

satyabalaanusha commented 3 years ago

Thanks for the detailed response. Yes, the asn1c is using asn definitions according to 38.413 spec Rel 15. So, tried to eliminate the mismatch by replacing the asn files in folder 3GPP_NR_NGAP_38413 with the exact asn files which the asn1c is using. And after that, recompiled the asn files with the following command: ./tools/pycrate_asn1compile.py -i ./pycrate_asn1dir/3GPP_NR_NGAP_38413/ -o NGAP.py -j

Then tried the previously mentioned code snippet and those trailing bytes which are not coming before from the pycrate are getting encoded and the packet is also getting decoded properly on the asn1c side.

P1sec / pycrate

Question: Encoding issue #113