Failure to install DSDL using example command

TomDeRybel commented 3 years ago

Using the latest pypi version of pyuavcan (1.1.0-dev3), I encountered the following problem trying to exactly reproduce the example described here: https://pyuavcan.readthedocs.io/en/latest/pages/cli.html

Issues:

Generating the local DSDL structure fails, with an error regarding a missing (target) folder.
When I manually generate the target folder (second run, below), the same error occurs, and the folder is deleted.

Below are the exact commands I used, with the resulting output. To make sure I was running the newest code, I removed and then re-installed pyuavcan as well.

(base) tomr@Feynman:~$ pip uninstall pyuavcan
Found existing installation: pyuavcan 1.1.0.dev3
Uninstalling pyuavcan-1.1.0.dev3:
  Would remove:
    /home/tomr/anaconda3/bin/pyuavcan
    /home/tomr/anaconda3/bin/uvc
    /home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan-1.1.0.dev3.dist-info/*
    /home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/*
Proceed (y/n)? y
  Successfully uninstalled pyuavcan-1.1.0.dev3

(base) tomr@Feynman:~$ pip install pyuavcan
Collecting pyuavcan
  Using cached pyuavcan-1.1.0.dev3-py3-none-any.whl (304 kB)
Requirement already satisfied: nunavut~=0.4 in ./anaconda3/lib/python3.8/site-packages (from pyuavcan) (0.5.0)
Requirement already satisfied: numpy~=1.19 in ./anaconda3/lib/python3.8/site-packages (from pyuavcan) (1.19.4)
Requirement already satisfied: setuptools in ./anaconda3/lib/python3.8/site-packages (from nunavut~=0.4->pyuavcan) (49.2.0.post20200714)
Requirement already satisfied: pydsdl~=1.4 in ./anaconda3/lib/python3.8/site-packages (from nunavut~=0.4->pyuavcan) (1.9.1)
Installing collected packages: pyuavcan
Successfully installed pyuavcan-1.1.0.dev3

(base) tomr@Feynman:~$ pyuavcan -v dsdl-gen-pkg --lookup https://github.com/UAVCAN/public_regulated_data_types/archive/master.zip ~/namespace
2020-11-17 13:37:26  2004 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Input DSDL root namespace directories: ['/home/tomr/namespace']
2020-11-17 13:37:26  2004 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Downloading the archive from 'https://github.com/UAVCAN/public_regulated_data_types/archive/master.zip' into '/tmp/pyuavcan-cli-dsdl1eddc9e6/dsdl.zip'...
2020-11-17 13:37:27  2004 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Extracting the archive into '/tmp/pyuavcan-cli-dsdl1eddc9e6'...
2020-11-17 13:37:27  2004 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Resource 'https://github.com/UAVCAN/public_regulated_data_types/archive/master.zip' contains the following root namespace directories: ['/tmp/pyuavcan-cli-dsdl1eddc9e6/public_regulated_data_types-master/uavcan', '/tmp/pyuavcan-cli-dsdl1eddc9e6/public_regulated_data_types-master/.vscode', '/tmp/pyuavcan-cli-dsdl1eddc9e6/public_regulated_data_types-master/reg']
2020-11-17 13:37:27  2004 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Lookup DSDL root namespace directories: ['/tmp/pyuavcan-cli-dsdl1eddc9e6/public_regulated_data_types-master/uavcan', '/tmp/pyuavcan-cli-dsdl1eddc9e6/public_regulated_data_types-master/.vscode', '/tmp/pyuavcan-cli-dsdl1eddc9e6/public_regulated_data_types-master/reg']
2020-11-17 13:37:27  2004 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Generating DSDL package PosixPath('/home/tomr/namespace') from root namespace PosixPath('/home/tomr/namespace') with lookup dirs: ['/tmp/pyuavcan-cli-dsdl1eddc9e6/public_regulated_data_types-master/reg', '/tmp/pyuavcan-cli-dsdl1eddc9e6/public_regulated_data_types-master/uavcan', '/home/tomr/namespace', '/tmp/pyuavcan-cli-dsdl1eddc9e6/public_regulated_data_types-master/.vscode']
Error: FileNotFoundError: [Errno 2] No such file or directory: '/home/tomr/namespace'
2020-11-17 13:37:27  2004 INFO     pyuavcan._cli._main: Unhandled exception: [Errno 2] No such file or directory: '/home/tomr/namespace'
Traceback (most recent call last):
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/_main.py", line 25, in main
    exit(_main_impl())
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/_main.py", line 56, in _main_impl
    result = args.func(args)
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/_main.py", line 152, in execute
    return cmd.execute(args, subsystems)
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/commands/dsdl_generate_packages.py", line 113, in execute
    gpi_list = self._generate_dsdl_packages(source_root_namespace_dirs=inputs,
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/commands/dsdl_generate_packages.py", line 184, in _generate_dsdl_packages
    gpi = pyuavcan.dsdl.generate_package(root_namespace_directory=ns,
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/dsdl/_compiler.py", line 188, in generate_package
    composite_types = pydsdl.read_namespace(root_namespace_directory=str(root_namespace_directory),
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_namespace.py", line 157, in read_namespace
    target_dsdl_definitions = _construct_dsdl_definitions_from_namespace(root_namespace_directory)
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_namespace.py", line 407, in _construct_dsdl_definitions_from_namespace
    for root, _dirnames, filenames in walker:
  File "/home/tomr/anaconda3/lib/python3.8/os.py", line 355, in walk
    onerror(error)
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_namespace.py", line 400, in on_walk_error
    raise os_ex     # pragma: no cover
  File "/home/tomr/anaconda3/lib/python3.8/os.py", line 352, in walk
    scandir_it = scandir(top)
FileNotFoundError: [Errno 2] No such file or directory: '/home/tomr/namespace'

(base) tomr@Feynman:~$ mkdir namespace

(base) tomr@Feynman:~$ pyuavcan -v dsdl-gen-pkg --lookup https://github.com/UAVCAN/public_regulated_data_types/archive/master.zip ~/namespace
2020-11-17 13:38:11  2018 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Input DSDL root namespace directories: ['/home/tomr/namespace']
2020-11-17 13:38:11  2018 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Downloading the archive from 'https://github.com/UAVCAN/public_regulated_data_types/archive/master.zip' into '/tmp/pyuavcan-cli-dsdlz85u6ze9/dsdl.zip'...
2020-11-17 13:38:12  2018 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Extracting the archive into '/tmp/pyuavcan-cli-dsdlz85u6ze9'...
2020-11-17 13:38:12  2018 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Resource 'https://github.com/UAVCAN/public_regulated_data_types/archive/master.zip' contains the following root namespace directories: ['/tmp/pyuavcan-cli-dsdlz85u6ze9/public_regulated_data_types-master/uavcan', '/tmp/pyuavcan-cli-dsdlz85u6ze9/public_regulated_data_types-master/.vscode', '/tmp/pyuavcan-cli-dsdlz85u6ze9/public_regulated_data_types-master/reg']
2020-11-17 13:38:12  2018 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Lookup DSDL root namespace directories: ['/tmp/pyuavcan-cli-dsdlz85u6ze9/public_regulated_data_types-master/uavcan', '/tmp/pyuavcan-cli-dsdlz85u6ze9/public_regulated_data_types-master/.vscode', '/tmp/pyuavcan-cli-dsdlz85u6ze9/public_regulated_data_types-master/reg']
2020-11-17 13:38:12  2018 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Generating DSDL package PosixPath('/home/tomr/namespace') from root namespace PosixPath('/home/tomr/namespace') with lookup dirs: ['/tmp/pyuavcan-cli-dsdlz85u6ze9/public_regulated_data_types-master/uavcan', '/tmp/pyuavcan-cli-dsdlz85u6ze9/public_regulated_data_types-master/reg', '/home/tomr/namespace', '/tmp/pyuavcan-cli-dsdlz85u6ze9/public_regulated_data_types-master/.vscode']
Error: FileNotFoundError: [Errno 2] No such file or directory: '/home/tomr/namespace'
2020-11-17 13:38:12  2018 INFO     pyuavcan._cli._main: Unhandled exception: [Errno 2] No such file or directory: '/home/tomr/namespace'
Traceback (most recent call last):
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/_main.py", line 25, in main
    exit(_main_impl())
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/_main.py", line 56, in _main_impl
    result = args.func(args)
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/_main.py", line 152, in execute
    return cmd.execute(args, subsystems)
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/commands/dsdl_generate_packages.py", line 113, in execute
    gpi_list = self._generate_dsdl_packages(source_root_namespace_dirs=inputs,
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/commands/dsdl_generate_packages.py", line 184, in _generate_dsdl_packages
    gpi = pyuavcan.dsdl.generate_package(root_namespace_directory=ns,
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/dsdl/_compiler.py", line 188, in generate_package
    composite_types = pydsdl.read_namespace(root_namespace_directory=str(root_namespace_directory),
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_namespace.py", line 157, in read_namespace
    target_dsdl_definitions = _construct_dsdl_definitions_from_namespace(root_namespace_directory)
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_namespace.py", line 407, in _construct_dsdl_definitions_from_namespace
    for root, _dirnames, filenames in walker:
  File "/home/tomr/anaconda3/lib/python3.8/os.py", line 355, in walk
    onerror(error)
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_namespace.py", line 400, in on_walk_error
    raise os_ex     # pragma: no cover
  File "/home/tomr/anaconda3/lib/python3.8/os.py", line 352, in walk
    scandir_it = scandir(top)
FileNotFoundError: [Errno 2] No such file or directory: '/home/tomr/namespace'

(base) tomr@Feynman:~$ ls namespace
ls: cannot access 'namespace': No such file or directory

pavel-kirienko commented 3 years ago

This is kind of a hilarious problem actually that I stumbled upon myself and it was also briefly mentioned in https://github.com/UAVCAN/pyuavcan/issues/127. Strictly speaking, it does behave exactly as documented; the docs say:

Existing packages will be overwritten entirely.

Meaning that the output directory is removed before the package is generated. The name of the output directory matches the input one (unless stropping is applied), so it follows that if the input is in the current working directory, the tool diligently deletes it and then complains that it can't find the source files.

I guess we just need to postpone deleting the directory and emit an adequate error message here.

TomDeRybel commented 3 years ago

Now I have the mental image of the snake that bites itself in the tail ;-)

Though, now I'm kinda stuck: the old GUI tool doesn't work for V1 and I clearly fail to compile the DSDL stuff for pyuavcan-cli. Testing my own implementation against my own implementation also has that circular thing going on... (Internally-consistent, but ultimately wrong.)

I was hoping to use the Python tool to help develop the PnP implementation in my code. Any thoughts?

pavel-kirienko commented 3 years ago

Just move your DSDL definitions into a subdirectory, or override the default destination directory.

TomDeRybel commented 3 years ago

Getting closer, but now this: Seems there is something going on with the "8184.Record.1.1.uavcan" DSDL definition.

(base) tomr@Feynman:~$ pyuavcan -v dsdl-gen-pkg --output ~/dsdl/pyuavcan/ --lookup https://github.com/UAVCAN/public_regulated_data_types/archive/master.zip ~/dsdl/public_regulated_data_types/       
2020-11-19 11:12:26 11961 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Input DSDL root namespace directories: ['/home/tomr/dsdl/public_regulated_data_types']
2020-11-19 11:12:26 11961 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Downloading the archive from 'https://github.com/UAVCAN/public_regulated_data_types/archive/master.zip' into '/tmp/pyuavcan-cli-dsdl9asbimd4/dsdl.zip'...
2020-11-19 11:12:27 11961 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Extracting the archive into '/tmp/pyuavcan-cli-dsdl9asbimd4'...
2020-11-19 11:12:27 11961 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Resource 'https://github.com/UAVCAN/public_regulated_data_types/archive/master.zip' contains the following root namespace directories: ['/tmp/pyuavcan-cli-dsdl9asbimd4/public_regulated_data_types-master/uavcan', '/tmp/pyuavcan-cli-dsdl9asbimd4/public_regulated_data_types-master/.vscode', '/tmp/pyuavcan-cli-dsdl9asbimd4/public_regulated_data_types-master/reg']
2020-11-19 11:12:27 11961 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Lookup DSDL root namespace directories: ['/tmp/pyuavcan-cli-dsdl9asbimd4/public_regulated_data_types-master/uavcan', '/tmp/pyuavcan-cli-dsdl9asbimd4/public_regulated_data_types-master/.vscode', '/tmp/pyuavcan-cli-dsdl9asbimd4/public_regulated_data_types-master/reg']
2020-11-19 11:12:27 11961 INFO     pyuavcan._cli.commands.dsdl_generate_packages: Generating DSDL package PosixPath('/home/tomr/dsdl/pyuavcan/public_regulated_data_types') from root namespace PosixPath('/home/tomr/dsdl/public_regulated_data_types') with lookup dirs: ['/tmp/pyuavcan-cli-dsdl9asbimd4/public_regulated_data_types-master/.vscode', '/tmp/pyuavcan-cli-dsdl9asbimd4/public_regulated_data_types-master/uavcan', '/tmp/pyuavcan-cli-dsdl9asbimd4/public_regulated_data_types-master/reg', '/home/tomr/dsdl/public_regulated_data_types']
2020-11-19 11:12:27 11961 INFO     pydsdl._namespace: Reading 194 definitions from the root namespace 'public_regulated_data_types', with 388 lookup definitions located in root namespaces: reg, public_regulated_data_types, uavcan
2020-11-19 11:12:27 11961 INFO     pydsdl._dsdl_definition: public_regulated_data_types.reg.drone.phy.acoustics.Note.0.1: Starting processing with 387 lookup definitions located in root namespaces: reg, public_regulated_data_types, uavcan
///--------------------------------------------------///
///----- CUT A LOT OF SUCCESSES ----///
///--------------------------------------------------///
2020-11-19 11:12:27 11961 INFO     pydsdl._dsdl_definition: public_regulated_data_types.uavcan.diagnostic.Record.1.1: Starting processing with 387 lookup definitions located in root namespaces: reg, public_regulated_data_types, uavcan
2020-11-19 11:12:27 11961 INFO     pydsdl._dsdl_definition: public_regulated_data_types.uavcan.diagnostic.Severity.1.0: Starting processing with 386 lookup definitions located in root namespaces: reg, public_regulated_data_types, uavcan
2020-11-19 11:12:27 11961 INFO     pydsdl._dsdl_definition: public_regulated_data_types.uavcan.diagnostic.Severity.1.0: Processed in 9 ms; category: StructureType, fixed port ID: None
Error: UnregulatedFixedPortIDError: /home/tomr/dsdl/public_regulated_data_types/uavcan/diagnostic/8184.Record.1.1.uavcan: Regulated port ID 8184 for message type 'public_regulated_data_types.uavcan.diagnostic.Record' is not valid. Consider using allow_unregulated_fixed_port_id.
2020-11-19 11:12:27 11961 INFO     pyuavcan._cli._main: Unhandled exception: /home/tomr/dsdl/public_regulated_data_types/uavcan/diagnostic/8184.Record.1.1.uavcan: Regulated port ID 8184 for message type 'public_regulated_data_types.uavcan.diagnostic.Record' is not valid. Consider using allow_unregulated_fixed_port_id.
Traceback (most recent call last):
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/_main.py", line 25, in main
    exit(_main_impl())
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/_main.py", line 56, in _main_impl
    result = args.func(args)
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/_main.py", line 152, in execute
    return cmd.execute(args, subsystems)
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/commands/dsdl_generate_packages.py", line 113, in execute
    gpi_list = self._generate_dsdl_packages(source_root_namespace_dirs=inputs,
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/commands/dsdl_generate_packages.py", line 184, in _generate_dsdl_packages
    gpi = pyuavcan.dsdl.generate_package(root_namespace_directory=ns,
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/dsdl/_compiler.py", line 188, in generate_package
    composite_types = pydsdl.read_namespace(root_namespace_directory=str(root_namespace_directory),
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_namespace.py", line 185, in read_namespace
    types = _read_namespace_definitions(target_dsdl_definitions,
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_namespace.py", line 236, in _read_namespace_definitions
    raise ex
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_namespace.py", line 231, in _read_namespace_definitions
    dt = tdd.read(lookup_definitions,
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_dsdl_definition.py", line 149, in read
    raise ex
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_dsdl_definition.py", line 139, in read
    self._cached_type = builder.finalize()
  File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_data_type_builder.py", line 108, in finalize
    raise UnregulatedFixedPortIDError(
pydsdl._data_type_builder.UnregulatedFixedPortIDError: /home/tomr/dsdl/public_regulated_data_types/uavcan/diagnostic/8184.Record.1.1.uavcan: Regulated port ID 8184 for message type 'public_regulated_data_types.uavcan.diagnostic.Record' is not valid. Consider using allow_unregulated_fixed_port_id.
(base) tomr@Feynman:~$

pavel-kirienko commented 3 years ago

$ pyuavcan -v dsdl-gen-pkg --output ~/dsdl/pyuavcan/ --lookup https://github.com/UAVCAN/public_regulated_data_types/archive/master.zip ~/dsdl/public_regulated_data_types/

Here's the culprit: ~/dsdl/public_regulated_data_types/. You are saying that the root namespace is public_regulated_data_types, but what you mean is ~/dsdl/public_regulated_data_types/uavcan/.

Also, you don't need any lookup definitions when processing the standard namespace. Providing them will actually trigger an error because of the namespace name conflict --- you would have uavcan both in your target namespace and in the look-up directory.

The correct commands are:

$ pyuavcan -v dsdl-gen-pkg --output ~/dsdl/pyuavcan/ ~/dsdl/public_regulated_data_types/uavcan
$ pyuavcan -v dsdl-gen-pkg --output ~/dsdl/pyuavcan/ ~/dsdl/public_regulated_data_types/reg

TomDeRybel commented 3 years ago

That is entirely not how I thought that worked...

The first command worked, and I now have all the Python packages in the target.
The second one didn't. It's not finding the base types from uavcan. (For good measure, and to make sure, I exported the path, as explained by the previous command's output.) I can see it't not getting the uavcan namespace.

Sorry I'm being so useless at this. I'm one of those people who needs a worked example to really grok things, and then builds on that.

(base) tomr@Feynman:~$ export PYTHONPATH="/home/tomr/dsdl/pyuavcan" (base) tomr@Feynman:~$ pyuavcan -v dsdl-gen-pkg --output ~/dsdl/pyuavcan/ ~/dsdl/public_regulated_data_types/reg 2020-11-19 12:16:45 12708 INFO pyuavcan._cli.commands.dsdl_generate_packages: Input DSDL root namespace directories: ['/home/tomr/dsdl/public_regulated_data_types/reg'] 2020-11-19 12:16:45 12708 INFO pyuavcan._cli.commands.dsdl_generate_packages: Lookup DSDL root namespace directories: [] 2020-11-19 12:16:45 12708 INFO pyuavcan._cli.commands.dsdl_generate_packages: Generating DSDL package PosixPath('/home/tomr/dsdl/pyuavcan/reg') from root namespace PosixPath('/home/tomr/dsdl/public_regulated_data_types/reg') with lookup dirs: ['/home/tomr/dsdl/public_regulated_data_types/reg'] 2020-11-19 12:16:45 12708 INFO pydsdl._namespace: Reading 40 definitions from the root namespace 'reg', with 40 lookup definitions located in root namespaces: reg 2020-11-19 12:16:45 12708 INFO pydsdl._dsdl_definition: reg.drone.phy.acoustics.Note.0.1: Starting processing with 39 lookup definitions located in root namespaces: reg Error: UndefinedDataTypeError: /home/tomr/dsdl/public_regulated_data_types/reg/drone/phy/acoustics/Note.0.1.uavcan:6: Data type uavcan.si.unit.frequency.Scalar.1.0 could not be found in the following root namespaces: {'reg'}. Please make sure that you specified the directories correctly. 2020-11-19 12:16:45 12708 INFO pyuavcan._cli._main: Unhandled exception: /home/tomr/dsdl/public_regulated_data_types/reg/drone/phy/acoustics/Note.0.1.uavcan:6: Data type uavcan.si.unit.frequency.Scalar.1.0 could not be found in the following root namespaces: {'reg'}. Please make sure that you specified the directories correctly. Traceback (most recent call last): File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/_main.py", line 25, in main exit(_main_impl()) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/_main.py", line 56, in _main_impl result = args.func(args) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/_main.py", line 152, in execute return cmd.execute(args, subsystems) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/commands/dsdl_generate_packages.py", line 113, in execute gpi_list = self._generate_dsdl_packages(source_root_namespace_dirs=inputs, File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/_cli/commands/dsdl_generate_packages.py", line 184, in _generate_dsdl_packages gpi = pyuavcan.dsdl.generate_package(root_namespace_directory=ns, File "/home/tomr/anaconda3/lib/python3.8/site-packages/pyuavcan/dsdl/_compiler.py", line 188, in generate_package composite_types = pydsdl.read_namespace(root_namespace_directory=str(root_namespace_directory), File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_namespace.py", line 185, in read_namespace types = _read_namespace_definitions(target_dsdl_definitions, File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_namespace.py", line 236, in _read_namespace_definitions raise ex File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_namespace.py", line 231, in _read_namespace_definitions dt = tdd.read(lookup_definitions, File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_dsdl_definition.py", line 149, in read raise ex File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_dsdl_definition.py", line 137, in read _parser.parse(f.read(), builder) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_parser.py", line 36, in parse raise ex File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_parser.py", line 30, in parse pr.parse(text) # type: ignore File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 254, in parse return self._parse_or_match(text, pos, 'parse') File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 289, in _parse_or_match return self.visit(getattr(self.grammar, method_name)(text, pos=pos)) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/third_party/parsimonious/nodes.py", line 217, in visit return method(node, [self.visit(n) for n in node]) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_parser.py", line 235, in visit_type_versioned return self._statement_stream_processor.resolve_versioned_data_type(name, version) File "/home/tomr/anaconda3/lib/python3.8/site-packages/pydsdl/_data_type_builder.py", line 192, in resolve_versioned_data_type raise UndefinedDataTypeError(error_description) pydsdl._data_type_builder.UndefinedDataTypeError: /home/tomr/dsdl/public_regulated_data_types/reg/drone/phy/acoustics/Note.0.1.uavcan:6: Data type uavcan.si.unit.frequency.Scalar.1.0 could not be found in the following root namespaces: {'reg'}. Please make sure that you specified the directories correctly. (base) tomr@Feynman:~$

pavel-kirienko commented 3 years ago

Ah, sorry, it's my fault -- forgot the --lookup. The second command is:

$ pyuavcan -v dsdl-gen-pkg --output ~/dsdl/pyuavcan/ ~/dsdl/public_regulated_data_types/reg --lookup ~/dsdl/public_regulated_data_types/uavcan

TomDeRybel commented 3 years ago

Cool! That all worked now. And now the synthax makes much more sense to me. Especially the second command showcases them all: where to put the compiled results; from which dsdl definitions; using these public regulated types.

May I propose to add a version of both these lines as an example to the documentation? Right now, they show how to generate a custom type. It would have helped if they also showed how to set-up the basics. Perhaps here: https://pyuavcan.readthedocs.io/en/latest/pages/usage.html#basic-usage-demo or https://pyuavcan.readthedocs.io/en/latest/pages/usage.html#generating-data-type-packages-from-dsdl or https://pyuavcan.readthedocs.io/en/latest/pages/cli.html#subcommand-dsdl-generate-packages

Thanks again for your time!

pavel-kirienko commented 3 years ago

There, I fixed it. Now you get helpful suggestions:

TomDeRybel commented 3 years ago

Nice! I like helpful suggestions :-)

Also, having finally understood enough of the syntax, I was able to use Nunavut to generate the DSDL serialization headers for C and C++. Though, I noticed these are a work in progress, so I'll stick to my home-grown all-manual serialization for now.

pavel-kirienko commented 3 years ago

Please keep it open. Someone should address the confusing behavior where the DSDL namespace directory is unexpectedly deleted.

TomDeRybel commented 3 years ago

Ok

OpenCyphal / pycyphal

Failure to install DSDL using example command #133