Vanuan
commented
3 years ago Here's a simple generator I wrote to generate python stubs for *Py.xml files:
import sys
import xml.etree.ElementTree as ET
def indent(block):
indented_lines = []
for line in block.splitlines():
indented_lines.append(' ' * 4 + line)
return '\n'.join(indented_lines)
def gen_name(name, module=None):
_name = name.rstrip('Py')
_name = _name.lstrip('Topo')
if module:
_name = module + '.' + _name
return _name
def gen_parent_name(child):
name = child.attrib['Father']
return gen_name(name, child.attrib['FatherNamespace'])
def gen_class_name(child):
name = child.attrib['Name']
return gen_name(name)
def gen_doc(doc_str):
return '"""' + doc_str + '"""\n'
def gen_class(child):
class_str = 'class '
name = gen_class_name(child)
class_str += name
parent = gen_parent_name(child)
if parent:
class_str += '(' + parent + ')'
class_str += ':\n '
doc_str = child.find('./Documentation/UserDocu').text
class_str += gen_doc(doc_str)
methods = child.findall('Methode')
for method_node in methods:
method_str = gen_method(method_node)
class_str += indent(method_str) + '\n\n'
attributes = child.findall('Attribute')
for attribute_node in attributes:
attr_str = gen_attr(attribute_node)
class_str += indent(attr_str) + '\n\n'
return class_str
def gen_attr(node):
doc_str = node.find('./Documentation/UserDocu').text
return '@property\ndef ' + node.attrib['Name'] + '(self):\n ' + gen_doc(doc_str) + ' ...'
def gen_method(node):
doc_str = node.find('./Documentation/UserDocu').text
return 'def ' + node.attrib['Name'] + '(self):\n ' + gen_doc(doc_str) + ' ...'
if __name__ == '__main__':
filename = sys.argv[1]
tree = ET.parse(filename)
root = tree.getroot()
for child in root:
if child.tag == 'PythonExport':
class_definition = gen_class(child)
print(class_definition)usage:
python3 generate.py src/Mod/Part/App/TopoShapePy.xml > src/python-stubs/Part/TopoShape.pyiUnfortunately xml files don't have a list of method arguments and don't specify arguments and return types. Also, special care should be put to Property types as those are kind of confusing. So not sure how to proceed here: whether I should push for XML -> Py conversion and add the information required or insist on the more high-level YAML-based format from which to generate those XML files.
luzpaz
Abstract
Keeping up to date with the Object API, verifying its usage correctness and being productive while using it in the IDE needs changing the way we define and document properties. This proposal suggests a way to move to the static definition of properties and object types. The primary focus is on the Python API, keeping in mind finding a way to integrate it into C++ workflow.
Motivation
Part 1
Being interested in CI to improve overall quality and frustrated with a lot of seemingly easy preventable Python exceptions, I've started looking into how python tooling can help to find bugs before users report them. Taking a look into the pylint proposal and doing some research, I've came to conclusion that static analysis tools mainly focus on the type correctness (keeping track of None values and object attributes and methods).
Among static analysis tools the most advanced ones are pyright and mypy. Pyright is used in the Microsoft's Python extension for VSCode (read the article) and mypy is used for CI scripting, to catch bugs before they're merged.
While trying to integrate it (see PR) I soon found that the DocumentObject class which is a return type of a
FreeCAD.ActiveDocument.addObjectmethod has a type system which is not exposed to static analysis tools. In fact, it is completely hidden from any sort of static analysis. The type on an object can only be inferred in runtime.Part 2
Being interested in how different object types fit together and how their properties work, I started reading wiki. E.g. Arch Wall. Soon I've came across the Object API page which pointed me to the autogenerated documentation. Finding a corresponding page for ArchWall I was baffled that it doesn't list any of the properties listed on the Wiki page.
Digging further, I found this is generated by a combination of sphinx and Doxygen. Those tools only understand the Python and C++ documentation syntax. They cannot look into the dynamic types and property system that is the core of FreeCAD.
So, there are multiple documentation sources. Some of them are manual, some are automatic. There's a need for consolidation of the Object API into a cohesive system driven by a single source of truth.
Thinking of a way to implement it, I've came to conclusion that you need to create the document with all the possible object types and their properties and iterate though them to build documentation. This approach is very expensive as you need to build the whole FreeCAD first, and run it on some performant server. Besides writing a script to generate the docs in the first place.
Instead of runtime type inference, it seems more reasonable to define types statically and generate the code which adds them in runtime. As it is fast, doesn't require C++ toolchain, so can be integrated into automated documentation generation CI quite cheaply.
Current C++ to Python XML bindings already do quite a good job to define Objects statically. But they are not enough. There needs to be a Python counterpart as the bulk of objects are defined using the App::FeaturePython Scripted Objects.
Specification
YAML-based properties in Python
Here's an attempt of capturing the definition of properties added using the
addPropertymethod.Arch.Structureis given as an example.This already allows to generate the following class that allows you to add type information in runtime:
And the corresponding class that adds the static python type:
Rationale
XML and JSON might be to verbose for this task. YAML appears to be more human friendly.
YAML could be used to generate C++ XML bindings as well. It would need to be enriched with C++ and Python imports/headers to reference dependencies.
So that there would be a single source of truth of objects defined in C++ and Python.
Alternatives
Alternatively, the current XML schema needs to be enriched with Python types. But it would mean the existing generator needs to be changed, so there's a threat of breaking backward compatibility.
Backwards Compatibility
To preserve compatibility, generated code has exactly the same behavior. In future versions we might want to change API to consolidate C++ and Python defined objects. For example, methods to manipulate objects might be divorced from the objects themselves. So that
FCStdfile format doesn't imply any dynamic behavior of objects but rather defines only the structure. This allows for versioning the file format separately from FreeCAD releases.Sample Implementation
Haven't published yet, but here's a gist. Usage:
Implementation:
It relies on the Python metaprogramming by inspecting
__class__.__dict__and usingDocumentObject.addPropertymethod to making it available to FreeCAD's runtime type system.FAQ
Copyright
All FEPs are explicitly CC0 1.0 Universal.