arvidfm commented 12 months ago

Fixes #9, fixes #14, fixes #16, fixes #17

Hi there! I bring you a boatload of improvements to the stubs, both for the Python commands and the Python API 1.0/2.0. It's a chonky PR, so if you'd like me to split it up into smaller parts I'd be happy to oblige, but I've tried to keep each commit as small and self-contained as possible to make it easier to review. Some of the changes that relate to the development environment are quite opinionated, so I'm very much expecting that you might disagree with some of them - don't be afraid to tell me to revert/change some things!

The overall goals for these changes were:

Add Maya 2024 support
Ensure that all stubs are syntactically valid and that all references resolve correctly
Make Python command documentation parsing more complete
Improve stubs generated by BuiltinParser by implementing various low-hanging strategies that were still easier than writing a full documentation parser for the Python API modules

NOTE: I haven't actually tested these changes with Maya 2023 as I don't have a copy installed on my machine! I'd be very grateful if you or someone could give the generation a spin on Maya 2023, if you think compatibility is important (though I'm not sure how you'd handle versioning of the maya-stubs package to support different stubs for different versions of Maya).

A full description of the changes follows below.

Development environment

I made various changes to make the maya-stubgen project cross platform, and to make it easier to use it with different Maya versions. This does mean a few changes in workflow.

The base interpreter for the virtual environment is now required to be mayapy, meaning that you should initialise your environment by running poetry env use /path/to/mayapy[.exe] before running poetry install
- This makes it easier to locate all required module paths, so the Poetry venv no longer needs to exist in a predefined location, so the poetry.toml overrides were no longer needed
- The correct way to run maya-stubgen is now to launch it via Poetry (e.g. poetry run python -m ... or poetry run generate-stubs ...), after which the Maya libraries are added to the path by finding the system library path of the base interpreter
- This change also makes maya-stubgen more platform agnostic, as the original setup used a few Windows-specific paths
The generate-stubs script now gets generated correctly on poetry install and can be run with poetry run generate-stubs --help
The entire project, including the stubs, now passes strict type checking with both Pyright and Mypy
- As part of this, I removed the seemingly old, unused and unfinished maya_stubgen/cli/generate_stubs/common.py as it was causing headaches when attempting to type hint properly
maya.standalone is now only initialised when needed - specifically when generating the docspec files - to reduce startup time when working with the template files
The generate-stubs and generate-docspec commands now take two flags to allow you to filter what modules and members to generate stubs for, making it a lot easier to test changes on a small number of functions
- -m specifies a module that you want to generate stubs for, and can be included multiple times, e.g. -m maya.cmds -m maya.mel
- --members takes a regex pattern which filters members inside a module, e.g. --members "(currentTime|playblast)$" will only generate stubs for currentTime and playblast
The stubgen project is now explicitly marked as only supporting Python 3.9 and above - this was already effectively the case due to the use of some 3.9-specific syntax like list[str]
- maya-stubs itself still works with 3.7
Some dependencies were updated and some unused ones were removed (particularly ones that were causing trouble for me on Linux)
CONTRIBUTING.md has been updated to reflect the above changes

Python command stubs

Command documentation is now fetched for the version of Maya that the stubs are being generated using
Positional arguments of commands now have default values of ..., as there are many commands where the default arguments are required in some modes but not others
- Trying to detect when a positional argument is always required is left as an exercise for someone braver than I am
query and/or edit flags are now included for commands detected as queryable and/or editable based on the description
- In query mode, any queryable argument takes a bool unless explicitly specified to only take a value in query mode as well, so these flags are now type hinted as X | bool
- The return type of queryable commands is now a union of the original return type as well as all queryable types
- In certain cases (in fact, only for optionVar as far as I could find), the query flag actually takes types other than bool, which is documented in the help text for the command, so if available the query and edit arguments are copied over from the synopsis parser
The command parser now handles commands that list multiple different possible return types in the synopsis by expressing them as a union
Flags marked as multiuse are now type hinted as X | List[X]
The MEL type to Python parser has been made more robust, and to my knowledge is now able to satisfactorily parse all MEL types in the online documentation and the help texts
- Tuples with optional elements, e.g. [String Float [ String ]] are now type hinted as Tuple[str, float] | Tuple[str, float, str]
- Types like linear are just converted to float for now even though they apparently also support strings like "3in", as I couldn't find any proper documentation on valid values for these types
- The correct types for timerange and floatrange were discovered through experimentation
The maya.cmds stub now includes various type aliases to make the function signatures a bit easier to read, e.g. Queryable[str] for str | bool and Range[float] for Tuple[float] | Tuple[float, float]

Python API stubs

Literals are now only printed to the stub files if they are simple literals that can actually be represented in Python, e.g. numbers, strings and empty lists/dicts, avoiding syntax errors from sequences like <property object at 0x7fc18a818400> in the stubs
The parser now keeps track of the fully qualified path of symbols imported from other modules, so e.g. MFnBase will be referenced as maya.OpenMaya.MFnBase when used in OpenMayaUI
- docspec-to-jinja then searches for any non-local symbols and adds any imports required to the stub file
The parser now attempt to guess the types of descriptor class members (property and getset_descriptor attributes) by instantiating the class and retrieving the member via the instance
- This is a bit of a volatile strategy, as there are some classes and attributes that crash Maya on class instantiation/attribute access, so I've had to explicitly exclude a number of items
- Even so, it does result in a surprising number of attributes getting resolved to the correct type
The parser also tries to guess the type of parameters using the default value imported by inspect.signature
Members imported from other modules are no longer included
Members are now looked up using inspect.getattr_static, allowing better introspection for e.g. static methods
- This also makes filtering of members inherited from parent classes more robust
@staticmethod and @classmethod decorators are now detected and included in the stubs
Module-level global variables are now included in the stubs
Add special cases for __init__ and __cls__ so that the former always returns None, and the latter always has a cls argument
- The type checkers would complain otherwise
Synopsis header parsing has been improved as it was breaking on some commands

Other various changes

Stubs are now also generated for the top-level maya package in maya.__init__.pyi (only contains the stringTable global)
__init__.pyi files are now included in all directories so that type checkers correctly detect them as Python packages
We no longer include symbols like *, ** or / as part of the name when constructing a docspec.Argument; instead, the symbols are added as needed by docspec-to-jinja based on the type of each argument
Any use of Callable is now type hinted as Callable[..., Any] to prevent type checkers for complaining about missing type parameters

Muream commented 11 months ago

Oh wow! I'm only seeing this now, I'll try and review this this week 👍

Muream commented 11 months ago

Hey, did you try this on windows or Linux only?

I'm running into this issue when running poetry env use 'C:\Program Files\Autodesk\Maya2024\bin\mayapy.exe'

Creating virtualenv maya-stubs-distrib-d06cmien-py3.10 in C:\Users\muream\AppData\Local\pypoetry\Cache\virtualenvs

Command ['C:\\Users\\muream\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\maya-stubs-distrib-d06cmien-py3.10\\Scripts\\python.exe', '-I', '-W', 'ignore', '-'] errored with the following return code 103

Error output:
No Python at '"C:\Program Files\Autodesk\Maya2024\bin\python.exe'

Input:
import sys

if hasattr(sys, "real_prefix"):
    print(sys.real_prefix)
elif hasattr(sys, "base_prefix"):
    print(sys.base_prefix)
else:
    print(sys.prefix)

I had a read through most of your commits and it all looks pretty good so I'll be happy to merge this once I can build things on my end

arvidfm commented 11 months ago

@Muream I only tested on Linux myself as I don't have ready access to a Windows machine unfortunately. I did just test on macOS though and got a somewhat similar error. I'll play around a bit until I can figure something out, it looks like mayapy is doing some PYTHONPATH magic that breaks virtual environments on non-Linux. I might have to give up on my dream of being able to do poetry env use and have it Just Work™ , but I'll try to come up with something that's at least cross platform, will keep you posted!

Muream commented 11 months ago

No problem at all, give me a shout if you want me to test something on windows 👍

If it ends up being too tricky we can also revert the dev environment changes from this PR and open another one that just focuses on that

arvidfm commented 11 months ago

@Muream OK, I managed to get... something... working. It turns out that mayapy does a lot more than just adding the site-packages path containing its Python modules, and on macOS and Windows e.g. PySide2 errors due to missing shared libraries and Qt resources if you don't run mayapy directly. Even if you manage to get a virtual environment going using mayapy as the base interpreter, running the symlinked python from Poetry's venv won't result in the environment being correctly set up.

My solution is a bit scuffed, but the idea is that if the maya module isn't available, we attempt to rerun the command in a subprocess using the direct path to mayapy to ensure that the environment is setup correctly, while adding the path to Poetry's site-packages via sitecustomize.py.

The location of mayapy is automatically detected, and to enable this it still assumes that mayapy is used as the base interpreter for the Poetry environment, but there's a caveat in that on macOS and Windows you can't just use poetry env use with mayapy as is. I've updated CONTRIBUTING.md with instructions (Windows instructions originally from here), see if that works for you.

A bit convoluted, but the result is something that's still cross platform, and still allows you to (somewhat) easily switch between Maya versions by just running poetry env use, as long as you take care to specify the path correctly depending on your OS. No need to define any additional environment variables or ensure that mayapy is on your PATH.

Muream commented 11 months ago

Cool that works for me, especially since making that symlink is recommended on maya's docs

The only error I ran into was that running poetry run generate-stubs generate-stubs maya-stubs --reuse-cache failed with this traceback

Traceback (most recent call last):
  File "C:\Program Files\Autodesk\Maya2024\Python\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Autodesk\Maya2024\Python\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\muream\projects\maya\maya-stubs\maya-stubgen\maya_stubgen\__main__.py", line 63, in <module>
    main()
  File "C:\Users\muream\projects\maya\maya-stubs\maya-stubgen\maya_stubgen\__main__.py", line 59, in main
    cli()
  File "C:\Users\muream\AppData\Local\pypoetry\Cache\virtualenvs\maya-stubs-distrib-d06cmien-py3.10\lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\muream\AppData\Local\pypoetry\Cache\virtualenvs\maya-stubs-distrib-d06cmien-py3.10\lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "C:\Users\muream\AppData\Local\pypoetry\Cache\virtualenvs\maya-stubs-distrib-d06cmien-py3.10\lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\muream\AppData\Local\pypoetry\Cache\virtualenvs\maya-stubs-distrib-d06cmien-py3.10\lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\muream\AppData\Local\pypoetry\Cache\virtualenvs\maya-stubs-distrib-d06cmien-py3.10\lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\muream\projects\maya\maya-stubs\maya-stubgen\maya_stubgen\cli\__init__.py", line 119, in generate_stubs
    build_stubs(path, reuse_cache, whitelist=module, member_pattern=members)
  File "C:\Users\muream\projects\maya\maya-stubs\maya-stubgen\maya_stubgen\cli\generate_stubs\__init__.py", line 234, in build_stubs
    docspec_cache = cache_dir() / "docspec"
  File "C:\Users\muream\projects\maya\maya-stubs\maya-stubgen\maya_stubgen\utils.py", line 112, in cache_dir
    return Path().resolve() / ".cache" / maya_version()
  File "C:\Users\muream\projects\maya\maya-stubs\maya-stubgen\maya_stubgen\utils.py", line 106, in maya_version
    _maya_version = maya.cmds.about(majorVersion=True)
AttributeError: module 'maya.cmds' has no attribute 'about'

This happens because maya.standalone.initialize() only ends up being called from dump_docspec which only runs when --reuse-cache isn't specified

But then down the line there's a call to cmds.about() to get the maya version that's only useful to get the cached docspec files

I feel like ideally nothing but the dump_docspec function should depend on maya calls. I was even hoping to down the line extract out all the maya specific stuff in its own "sandbox" (getting the synopsis from cmds.help and running the builtin parser on the OpenMaya modules) and then be able to just run the rest from any regular python interpreter

We don't need to take this this far at all right now though but I'd still like --reuse-cache to work :)

Oh and the generated stubs look good, I like the Type Aliases you did for Queryable and MultiUse and the OpenMaya stubs look much cleaner

Muream commented 11 months ago

Ah, and I just compared sys.path between mayapy and the venv created from it and the venv seems to just be missing these two paths from the PYTHONPATH

This works just fine, maybe we can skip the subprocess part and just do some PYTHONPATH fuckery as well?

$ poetry run python
# Python 3.10.8 (tags/v3.10.8:aaaf517, Oct 11 2022, 16:50:30) [MSC v.1933 64 bit (AMD64)] on win32
# Type "help", "copyright", "credits" or "license" for more information.
>>> import site
>>> site.addsitedir('C:\\Program Files\\Autodesk\\Maya2024\\Python')
>>> site.addsitedir('C:\\Program Files\\Autodesk\\Maya2024\\Python\\lib\\site-packages')
>>> import maya.standalone
>>> maya.standalone.initialize()
# Qt WebEngine seems to be initialized from a plugin. Please set Qt::AA_ShareOpenGLContexts using QCoreApplication::setAttribute before constructing QGuiApplication.
>>> from maya import cmds
>>> cmds.createNode("transform")
'transform1'
>>>

arvidfm commented 11 months ago

I updated the caching so that instead of saving to a version-specific directory, it records what Maya version it was generated using and clears the cache if you run it against another Maya version (so you don't accidentally use outdated synopsis and documentation caches after upgrading to a new version). That way there's no need to get the Maya version just to access the existing cache when running with --reuse-cache.

This works just fine, maybe we can skip the subprocess part and just do some PYTHONPATH fuckery as well?

I'm surprised that works on Windows, actually! I tried something similar on macOS, and importing both maya.standalone and PySide2 fails unless you set DYLD_LIBRARY_PATH to Maya.app/Contents/MacOS (where the shared library files are):

% poetry run python
Python 3.10.8 (v3.10.8:aaaf517424, Oct 11 2022, 10:14:40) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import site
>>> site.addsitedir("/Applications/Autodesk/maya2024/Maya.app/Contents/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages")
>>> import maya.standalone
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Applications/Autodesk/maya2024/Maya.app/Contents/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/maya/standalone.so, 0x0002): Library not loaded: '@executable_path/libDeformSlice.dylib'
  Referenced from: '/Applications/Autodesk/maya2024/Maya.app/Contents/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/maya/standalone.so'
  Reason: tried: '/Applications/Autodesk/maya2024/Maya.app/Contents/Frameworks/Python.framework/Versions/3.10/Resources/Python.app/Contents/MacOS/libDeformSlice.dylib' (no such file), '/usr/local/lib/libDeformSlice.dylib' (no such file), '/usr/lib/libDeformSlice.dylib' (no such file)
>>> from PySide2.QtWidgets import QApplication
PySide2/__init__.py: Unable to import shiboken2 from , /Applications/Autodesk/maya2024/Maya.app/Contents/Frameworks/Python.framework/Versions/3.10/lib/python310.zip, /Applications/Autodesk/maya2024/Maya.app/Contents/Frameworks/Python.framework/Versions/3.10/lib/python3.10, /Applications/Autodesk/maya2024/Maya.app/Contents/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload, /Users/x/Library/Caches/pypoetry/virtualenvs/maya-stubs-distrib-znvsCPJZ-py3.10/lib/python3.10/site-packages, /Users/x/Documents/Flix/maya-stubs/docspec-to-jinja, /Users/x/Documents/Flix/maya-stubs/maya-stubgen, /Users/x/Documents/Flix/maya-stubs/maya-stubs, /Applications/Autodesk/maya2024/Maya.app/Contents/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Applications/Autodesk/maya2024/Maya.app/Contents/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/PySide2/__init__.py", line 107, in <module>
    _setupQtDirectories()
  File "/Applications/Autodesk/maya2024/Maya.app/Contents/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/PySide2/__init__.py", line 58, in _setupQtDirectories
    import shiboken2
  File "/Applications/Autodesk/maya2024/Maya.app/Contents/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/shiboken2/__init__.py", line 27, in <module>
    from .shiboken2 import *
ImportError: dlopen(/Applications/Autodesk/maya2024/Maya.app/Contents/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/shiboken2/shiboken2.cpython-310-darwin.so, 0x0002): Library not loaded: '@rpath/libshiboken2.cpython-310-darwin.5.15.dylib'
  Referenced from: '/Applications/Autodesk/maya2024/Maya.app/Contents/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/shiboken2/shiboken2.cpython-310-darwin.so'
  Reason: tried: '/usr/local/lib/libshiboken2.cpython-310-darwin.5.15.dylib' (no such file), '/usr/lib/libshiboken2.cpython-310-darwin.5.15.dylib' (no such file)

DYLD_LIBRARY_PATH has to be set before the Python process starts which means that if we want to be macOS compatible the only two options I can think of is to use a wrapper script which sets the variable before running poetry run, or to require using mayapy. (And really, what is ensure_maya() if not a wrapper script à la the previous mayapy.bat, except written in Python?)

Muream commented 11 months ago

Cool, that works!

I'll merge this now and make a new release and we can rework the developer environment later on if the subprocess stuff becomes problematic

Thanks a lot for the pull request!

arvidfm commented 11 months ago

Awesome, thank you! Feel free to ping me in the future if you have any further questions about the changes here.

If I can I would like to find some time to look at implementing documentation parsers for OpenMaya 1 and 2 too, but it depends entirely on how much time we can manage to allocate to this at work, so I can't guarantee anything unfortunately.

Muream commented 11 months ago

Absolutely! I don't have much plans to work on those at the moment so have at it at your own pace :) I'm also happy for you to open issues to discuss things before hand if you need or to open Draft PRs and update them as you go

Muream / maya-stubs

Stub improvements and Maya 2024 support #18

Development environment

Python command stubs

Python API stubs

Other various changes