search

pycrate-org / pycrate

A Python library to ease the development of encoders and decoders for various protocols and file formats, especially telecom ones. Provides an ASN.1 compiler and a CSN.1 runtime.
https://github.com/pycrate-org/pycrate
GNU Lesser General Public License v2.1
45 stars 9 forks source link

Wiki needs more information on compiling custom ASN.1 modules #14

Open James-E-A opened 2 months ago

James-E-A commented 2 months ago

I'm trying to compile this very small ASN.1 definition as a test, which only declares a new and extremely boring signature algorithm:

HelloX509 DEFINITIONS ::=

BEGIN

-- EXPORTS All

IMPORTS
  authenticationFramework
    FROM UsefulDefinitions {joint-iso-itu-t ds(5) module(1) usefulDefinitions(0) 7}

  ALGORITHM
    FROM AuthenticationFramework authenticationFramework;

  null-with-null OBJECT IDENTIFIER ::= {iso(1) standard(0) 20248 digital-signature-methods(1) null-with-null(1)}

  null-with-null-Algorithm ALGORITHM ::= {
    -- PARMS      ABSENT
    IDENTIFIED BY null-with-null
  }

END -- HelloX509

However, when I try to use the compilation script with -i as the Wiki recommends, I get this error:

(venv) C:\…>python .venv\Scripts\pycrate_asn1compile.py -i HelloX509.asn1
[proc] [HelloX509.asn1] module HelloX509 (oid: []): 3 ASN.1 assignments found
--- compilation cycle ---
Traceback (most recent call last):
  File "C:\…\.venv\Lib\site-packages\pycrate_asn1c\asnobj.py", line 92, in get_asnobj
    mod = GLOBAL.MOD[mod_name]
          ~~~~~~~~~~^^^^^^^^^^
  File "C:\…\.venv\Lib\site-packages\pycrate_asn1c\dictobj.py", line 69, in __getitem__
    return self._dict[key]
           ~~~~~~~~~~^^^^^
KeyError: 'AuthenticationFramework'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\…\.venv\Lib\site-packages\pycrate_asn1c\asnobj.py", line 675, in get_typeref
    tr = get_asnobj(ref.called[0], ref.called[1])
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\…\.venv\Lib\site-packages\pycrate_asn1c\asnobj.py", line 94, in get_asnobj
    raise(ASN1Err('module {0}, undefined'.format(mod_name)))
pycrate_asn1c.err.ASN1Err: module AuthenticationFramework, undefined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\…\.venv\Scripts\pycrate_asn1compile.py", line 264, in <module>
    sys.exit(main())
             ^^^^^^
  File "C:\…\.venv\Scripts\pycrate_asn1compile.py", line 215, in main
    compile_text(txt, **ckw)
  File "C:\…\.venv\Lib\site-packages\pycrate_asn1c\asnproc.py", line 254, in compile_text
    compile_modules(remain)
  File "C:\…\.venv\Lib\site-packages\pycrate_asn1c\asnproc.py", line 942, in compile_modules
    ObjNew = asnobj_compile(Obj)
             ^^^^^^^^^^^^^^^^^^^
  File "C:\…\.venv\Lib\site-packages\pycrate_asn1c\asnproc.py", line 866, in asnobj_compile
    text = Obj.parse_def(text)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\…\.venv\Lib\site-packages\pycrate_asn1c\asnobj.py", line 1703, in parse_def
    text = self._parse_type(text)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\…\.venv\Lib\site-packages\pycrate_asn1c\asnobj.py", line 1929, in _parse_type
    tr = self.get_typeref()
         ^^^^^^^^^^^^^^^^^^
  File "C:\…\.venv\Lib\site-packages\pycrate_asn1c\asnobj.py", line 677, in get_typeref
    raise(ASN1ProcTextErr('{0}: {1}'\
pycrate_asn1c.err.ASN1ProcTextErr: null-with-null-Algorithm: module AuthenticationFramework, undefined

Why isn't this import working?

mitshell commented 2 months ago

You need to provide all the required ASN.1 definitions to the compiler for your module to compile. You can check the existing x.509 related ASN.1 definitions already provided in the pycrate_asn1dir subdirectory, and see if one already has all the definitions for compiling the ALGORITHM object.

James-E-A commented 2 months ago

You need to provide all the required ASN.1 definitions to the compiler for your module to compile.

Even when I include the file that exports the definitions I need manually with -i, I'm still getting a problem:

python .venv\Scripts\pycrate_asn1compile.py -i pycrate\pycrate_asn1dir\IETF_PKI_RFC5958\AlgorithmInformation-2009.asn -i HelloX509.asn1 HelloX509 -o HelloX509

pycrate_asn1c.err.ASN1ProcTextErr: sa-null-with-null-Algorithm: ASN1RefType to SIGNATURE-ALGORITHM, undefined

-- File based on https://github.com/pycrate-org/pycrate/blob/9093d3875caf2adc7833724edfc71a2321909d74/pycrate_asn1dir/IETF_PKI_RFC5958/CryptographicMessageSyntaxAlgorithms-2009.asn
--           and https://github.com/pycrate-org/pycrate/blob/9093d3875caf2adc7833724edfc71a2321909d74/pycrate_asn1dir/IETF_PKI_RFC5958/AlgorithmInformation-2009.asn#L89
HelloX509 DEFINITIONS ::=

BEGIN

-- EXPORTS All

IMPORTS

  ParamOptions, DIGEST-ALGORITHM, SIGNATURE-ALGORITHM,
      PUBLIC-KEY, KEY-DERIVATION, KEY-WRAP, MAC-ALGORITHM,
      KEY-AGREE, KEY-TRANSPORT, CONTENT-ENCRYPTION, ALGORITHM,
      AlgorithmIdentifier
  FROM AlgorithmInformation-2009 {
      iso(1) identified-organization(3) dod(6) internet(1) security(5)
      mechanisms(5) pkix(7) id-mod(0)
      id-mod-algorithmInformation-02(58)}

  id-null-with-null OBJECT IDENTIFIER ::= {iso(1) standard(0) 20248 digital-signature-methods(1) null-with-null(1)}

  sa-null-with-null SIGNATURE-ALGORITHM ::= {
    IDENTIFIER id-null-with-null
    PARAMS ARE absent
  }

END -- HelloX509
James-E-A commented 2 months ago

And to defuse any possible X-Y problem that might be here: my ultimate goal is to use pycrate as a library to serialize and deserialize CMS messages that have been encrypted with algorithms that have not been standardized yet; I'm only trying to start with the simplest possible "custom algorithm" as a test to make sure I've got the fundamentals of extending pycrate down first.

mitshell commented 2 months ago

OK, my recommendation would be to copy the entire IETF_PKI_RFC5958 subdirectory in your home. Add your new file into it, delete the load_mod.txt and load_obj.txt files, and compile the entire directory with:

pycrate_asn1compile.py -i $subdir -j

If it succeeds, your new value null-with-null-Algorithm should be available in the module out.py produced. Let me know if it works on your side, thx.

James-E-A commented 2 months ago

copy the entire IETF_PKI_RFC5958 subdirectory in your home. Add your new file into it, delete the load_mod.txt and load_obj.txt files, and compile the entire directory with [-i $subdir -j]

When I try exactly that, I get this error:

(venv) C:\…>python .venv\Scripts\pycrate_asn1compile.py -i IETF_PKI_RFC5958 -j
.venv\Scripts\pycrate_asn1compile.py, args error: unable to open input file IETF_PKI_RFC5958AlgorithmInformation-2009.asn
[Errno 2] No such file or directory: 'IETF_PKI_RFC5958AlgorithmInformation-2009.asn'

and out.py is created, but left completely empty.

(I tested, and I get this same error even if I don't delete the TXT files...)

mitshell commented 2 months ago

Maybe add a / after the subdir name passed as argument.

On my side, I get:

mich@HPMitch:~/src$ cp -r pycrate/pycrate_asn1dir/IETF_PKI_RFC5958/ .
mich@HPMitch:~/src$ rm IETF_PKI_RFC5958/*.txt
mich@HPMitch:~/src$ pycrate_asn1compile.py -j -i IETF_PKI_RFC5958/
[proc] [IETF_PKI_RFC5958/PKIX1Implicit-2009.asn] module PKIX1Implicit-2009 (oid: [1, 3, 6, 1, 5, 5, 7, 0, 59]): 108 ASN.1 assignments found
[proc] [IETF_PKI_RFC5958/AlgorithmInformation-2009.asn] module AlgorithmInformation-2009 (oid: [1, 3, 6, 1, 5, 5, 7, 0, 58]): 16 ASN.1 assignments found
[proc] [IETF_PKI_RFC5958/CryptographicMessageSyntax-2009.asn] module CryptographicMessageSyntax-2009 (oid: [1, 2, 840, 113549, 1, 9, 16, 0, 41]): 108 ASN.1 assignments found
[proc] [IETF_PKI_RFC5958/PKIX1Explicit-2009.asn] module PKIX1Explicit-2009 (oid: [1, 3, 6, 1, 5, 5, 7, 0, 51]): 84 ASN.1 assignments found
[proc] [IETF_PKI_RFC5958/PKIXAttributeCertificate-2009.asn] module PKIXAttributeCertificate-2009 (oid: [1, 3, 6, 1, 5, 5, 7, 0, 47]): 54 ASN.1 assignments found
[proc] [IETF_PKI_RFC5958/PKIX-CommonTypes-2009.asn] module PKIX-CommonTypes-2009 (oid: [1, 3, 6, 1, 5, 5, 7, 0, 57]): 10 ASN.1 assignments found
[proc] [IETF_PKI_RFC5958/AttributeCertificateVersion1-2009.asn] module AttributeCertificateVersion1-2009 (oid: [1, 2, 840, 113549, 1, 9, 16, 0, 49]): 6 ASN.1 assignments found
[proc] [IETF_PKI_RFC5958/CryptographicMessageSyntaxAlgorithms-2009.asn] module CryptographicMessageSyntaxAlgorithms-2009 (oid: [1, 2, 840, 113549, 1, 9, 16, 0, 37]): 44 ASN.1 assignments found
[proc] [IETF_PKI_RFC5958/SecureMimeMessageV3dot1-2009.asn] module SecureMimeMessageV3dot1-2009 (oid: [1, 2, 840, 113549, 1, 9, 16, 0, 39]): 15 ASN.1 assignments found
[proc] [IETF_PKI_RFC5958/PKIX-X400Address-2009.asn] module PKIX-X400Address-2009 (oid: [1, 3, 6, 1, 5, 5, 7, 0, 60]): 74 ASN.1 assignments found
[proc] [IETF_PKI_RFC5958/PKIXAlgs-2009.asn] module PKIXAlgs-2009 (oid: [1, 3, 6, 1, 5, 5, 7, 0, 56]): 75 ASN.1 assignments found
[proc] [IETF_PKI_RFC5958/PKIX1-PSS-OAEP-Algorithms-2009.asn] module PKIX1-PSS-OAEP-Algorithms-2009 (oid: [1, 3, 6, 1, 5, 5, 7, 0, 54]): 45 ASN.1 assignments found
[proc] [IETF_PKI_RFC5958/AsymmetricKeyPackageModuleV1.asn] module AsymmetricKeyPackageModuleV1 (oid: [1, 2, 840, 113549, 1, 9, 16, 0, 50]): 18 ASN.1 assignments found
--- compilation cycle ---
WNG: AlgorithmInformation-2009.AlgorithmIdentifier, untagged OPEN / ANY in SEQUENCE with parameters
WNG: AlgorithmInformation-2009.SMIMECapability, untagged OPEN / ANY in SEQUENCE with parameters
--- compilation cycle ---
--- compilation cycle ---
--- compilation cycle ---
--- compilation cycle ---
WNG: PKIX1Explicit-2009.SIGNED.algorithmIdentifier, untagged OPEN / ANY in SEQUENCE with parameters
--- compilation cycle ---
--- compilation cycle ---
--- verifications ---
[proc] ASN.1 modules processed: ['PKIX1Implicit-2009', 'AlgorithmInformation-2009', 'CryptographicMessageSyntax-2009', 'PKIX1Explicit-2009', 'PKIXAttributeCertificate-2009', 'PKIX-CommonTypes-2009', 'AttributeCertificateVersion1-2009', 'CryptographicMessageSyntaxAlgorithms-2009', 'SecureMimeMessageV3dot1-2009', 'PKIX-X400Address-2009', 'PKIXAlgs-2009', 'PKIX1-PSS-OAEP-Algorithms-2009', 'AsymmetricKeyPackageModuleV1']
[proc] ASN.1 objects compiled: 241 types, 62 sets, 341 values
[proc] done
mich@HPMitch:~/src$ ls -l out.*
-rw-r--r-- 1 mich mich 185038 Sep 28 09:36 out.json
-rw-r--r-- 1 mich mich 800737 Sep 28 09:36 out.py