Python 3 Migration

[jjburton]

Figure out how to do this.

• [x] Ask Matt what that tool was
• [x] RD people's experiences

RD

• https://matiascodesal.com/blog/how-to-convert-maya-python-2-code-to-python-3/
• https://discourse.techart.online/t/maya-2022-and-python3/13706

[jjburton]

Notes from twitter

Mark

• Compiled modules so the have different compiled versions. We don't do this so should be okay
• Took fresh branch python2to3 converted. Then dealt with issues

JB

• just made brackets around print statements

Ryan

• 2to3 and future

[jjburton]

Process

• [x] Setup clean copy in maya 2023
• [x] New repos
• [x] Get python 2to3
• [x] Run said app
• [x] Test

[jjburton]

General flow

• [x] Make sure you have python
• [x] Verify pip
• [x] pip install 2to3
• [x] Process dir with 2to3

Issues

• path from pip install wasn't letting me run 2to3 >>> add path per link below, then reopen command prompt

Installing pip https://www.liquidweb.com/kb/install-pip-windows/

• I had to double click the python script and run via python as the command line stuff was failing but it worked

Processing directories http://python3porting.com/2to3.html#:~:text=Running%202to3%20is%20very%20simple,the%20changes%20to%20the%20files.

Adding a pip install path to the environment https://superuser.com/questions/1372793/the-script-is-installed-in-directory-which-is-not-path

[jjburton]

Day1

• immediate blocks from cgm import pointing to some old code which really wasn't necessary
• red9 Block -- setup had some bad formatted print statements. Checked repos code base and it had it too. Asked Mark at TDA to see if he had a sep python 3 branch.
• He does - https://github.com/markj3d/Red9_StudioPack_Python3 . Grabbed it and moving forward.

Quirks from conversion -from . import maya doesn't like (Sometimes?) Replaced to just import with find/replace -Ran into issues from red9's _version import. Probably need to run Red9.setup.addPythonPackages() at cgm start to deal with it

• exception handler borked...
• path stuff has changed as well. Looking more and more like dropping or rewriting Hamish's Path class is in the cards. We'll see
• time.clock changed?
• Importing by string changed | __import__('cgm.core.cgm_General', globals(), locals(), ['*'],-1) no bueno. Lose the -1 at the end

[jjburton]

Cull

• core.rigger | this is old pre MRS stuff want to try to remove
• cgmPuppetMeta | I don't think we're using this

[jjburton]

Day 2

• time.clock | is no more. Tried a push with just swapping to time.time. Fingers crossed
• Got unit tests running again and....most are failing. Going to work through those.

Get tests working

• [x] core.attr
• [x] core.valid
• [x] core.path
• [x] core.nodefactory
• [x] cgmMeta.base
• [x] cgmMeta.mClasses
• [x] cgmMeta.PuppetMeta

cgmMeta.base

• [x] ms.ls flag int no longer in use || this ended up being long flag but the 2to3 thought it was a declaration and changed them to int
• [x] cgmPuppet not found in mType registry

cgmMeta.mClasses

• [x] int flag again
• [x] no longer using cgmPuppet, change these tests over to cgmRigPuppet

Class issues

__new__ usage appears to have changed. return super(cls.__class__, cls).__new__(cls,*args,**kws) >> return super(cls.__class__, cls).__new__(cls)

Path issues

• Hamish's Path class setup being sub to string will not work combining. Must use the Path.asString() call to get a string to use.

  import cgm.core.cgmPy.path_Utils as PATH
  importlib.reload(PATH)
  os.path.no
  import cgm
  _dir = PATH.Path(cgm.__path__[0])
  type(_dir.up().asString())
  issubclass(type(_dir.up()),str)
  _file = os.path.normpath(os.path.join(_dir.up().asString(),'cgmToolbox.py'))
  _file

NodeFactory issues

• unicode went away and was throwing errors
• trying to remember how argsToNodes works was...difficult

Red9 Json

• [ ] unit test failing. Guesing due to dictionary ordering...? need to modify test to check stored values/keys rather than assertion of total dict

[jjburton]

2to3 issues

• long flags on ls for example were changed to int=. Did a find and replace of ,int= to ,long=. , protects against other kws ending in 'int'. , int= to , long=good as well.
• reload now importlib.reload. | 2to3 is pretty good at these but found a few issues.

[jjburton]

To remember

• dict.iteritems is gone. /cry. for k,v in list(d.items()):
• return super(cls.__class__, cls).__new__(cls,*args,**kws) >> return super(cls.__class__, cls).__new__(cls)
• time.clock | is no more. Use time.time.
• reload(module >> importlib.reload(module)
• if buffer > 0 >> if len(buffer) > 0 or just if buffer

• Python 3 wants more explicit == rather than is declarations for string variables

  _keys = list(_d.keys())
  _keys.sort()
  
  to...
  _keys = sorted(_d)

• If you used indices as dictionary keys in python 2 and it worked fine. You'll probably need to sort the keys before pulling the data. Hit a few issues here.

_d = {0: 'orbFrontLeft',
     4: 'jawTopLeft',
     2: 'orbLeft',
     3: 'cheek_0_3_left',
     1: 'cheek_0_1_left'}
for ii,handle in list(_d.items()):#...no worky
    print("{} | {}".format(ii,handle))

for ii in sorted(_d):#...works as expected
    print("{} | {}".format(ii,_d[ii]))

[jjburton]

Day 3

• culling on cgm_PuppetMeta caused an issue | cgmMasterNull was there, Moving to rigBlocks. Need to check on cgmInfoNode being used
• When except Exception as err || raise Exception(err) >> raise err -Hamish's old code doesn't work. Need to look for a cleaner solution in time

  #loc = p / filename
  loc = Path(os.path.join(p.asString(), filename))

• Had a silly compilcated image size checker before. This is much simpler

  def get_image_size(file_path):
  import maya.api.OpenMaya as om
  img = om.MImage()
  img.readFromFile(file_path)
  return img.getSize()

• previous core reload pyc check not working because of how python 3 walks the structure

Exception Handling

This has been a bugger. Lots of issues with how MRS is handling errors. Trying to sort it

• Feels like putting exception handling after finally was what was causing real issues
• https://portingguide.readthedocs.io/en/latest/exceptions.html -https://docs.python.org/3/library/traceback.html

Builder - Build Modules

• [x] Master
• [x] Limb
• [x] Segment
• [x] Head

[jjburton]

Day 4

Exception Handling

This has been a bugger. Lots of issues with how MRS is handling errors. Trying to sort it

• Feels like putting exception handling after finally was what was causing real issues
• https://portingguide.readthedocs.io/en/latest/exceptions.html -https://docs.python.org/3/library/traceback.html

Process

• Move traceback handling into timer as well
• Lose most of the exception handling in MRS and let it push back to cgmGen's
• Can't have exception handling in the finally section well. Needs to be before that
• Getting much better error dumping tracing back through piles of nested functions
• Ran into issue with how red9 metaclass was handling dead nodes with python 3. The beahviour changed from previously warning to throwing exceptions. Talked to Mark and made some fixes to make things work as previously or better (got to fail a little earlier in the codebase)

Build issues

Segment ear wasn't building and in the end it turned out that a dict wasn't returning in order in python 3 with how I was doing things. I reworked it and added a new list return in ATTR.

Python dictionary changes

d = {3:'cat',1:'dog',2:'and',10:'!'}
#py2...
keys = d.keys()
keys.sort()

#py3...
sorted(d)#...just a sorted list of keys
d2 = {k:d[k] for k in sorted(d)}#...sorted dict
list(d2.keys())#...list keys

#p2 iteritems
for k,l in d2.iteritems():
    print("{},{}".format(k,l))

#p3 iteritems
for k,l in list(d2.items()):
    print("{},{}".format(k,l))

https://docs.python.org/3/howto/sorting.html

[jjburton]

Day 5

Coming back from Holiday break. Trying to remember where I was. Going to look for the sort error in the code base.

[jjburton]

Day 6

• [x] 2023 Cougar build
• [x] 2022 Drax template builds
• [x] Does the Python3 code work in 2018 | NO

Other stuff

• More code fixes. Cleaned up some == from is changes.

Does the Python3 code work in 2018

• no
• Tried a simple copy over of the python 3 cgm content to see if it would load. Immediately hit issues. Don't think a single repos is going to be doable
• Next question is will certain python 3 modules work - mrs rig blocks are my specific hope to be able to work well

[jjburton]

Playback Cache

• Starting to finally test stuff in Maya 22/23 after getting a Python 3 build of our tools. Noticed that our mirroring stuff is stupid slow (30 sec) when cache is on and near instant when off (.7 sec). Is there a good way to disable it during certain functions when it's on? I do something similar with undos and viewport updates.
• TD mates all recommend just telling people to disable playback caching

Resources

• https://help.autodesk.com/view/MAYAUL/2022/ENU/?guid=__CommandsPython_index_html
• https://download.autodesk.com/us/company/files/MayaCachedPlayback/2019/MayaCachedPlaybackWhitePaper.html

Wrote a new wrapper for cgmGen to handle some of this after testing the playback cache thing

def CleanCall(func):
    '''
    Call to handle the typical pre/post do stuff calls like:
    - undoChunk
    - suspend refresh
    - cached playback (maya 2019+)
    - autokey status
    - closing the maya progress bar if it was stuck open from an exception
    '''
    @wraps(func)
    def CleanCaller(*args, **kws):
        _str_func = 'CleanCaller {}'.format(func)
        res = None
        print(logString_sub(_str_func,'pre'))

        _cachedPlaybackEnable = mc.optionVar(q='cachedPlaybackEnable')
        if _cachedPlaybackEnable:
            mc.optionVar( iv=('cachedPlaybackEnable', 0))
        mc.undoInfo(openChunk=True,chunkName="undo{0}".format(func))
        #mc.undoInfo(openChunk=True)
        _autoKey = mc.autoKeyframe(q=True,state=True)
        mc.refresh(su=1)

        print(logString_sub(_str_func,'do'))
        try:
            res=func(*args,**kws) 
        finally:
            print(logString_sub(_str_func,'post'))

            mc.undoInfo(closeChunk=True)#...close our chunk
            if _autoKey:#turn back on if it was
                mc.autoKeyframe(state=True)

            mc.refresh(su=0) #turn off suspension

            if _cachedPlaybackEnable:
                mc.optionVar( iv=('cachedPlaybackEnable', 1))

            try:
                mayaMainProgressBar = mel.eval('$tmp = $gMainProgressBar')
                mc.progressBar(mayaMainProgressBar, edit=True, endProgress=True)                    
            except:pass
            if res:
                return res
    return CleanCaller

[jjburton]

Day 7

• Regards py2/3 compatibility. One TD friend recommended this: https://github.com/benjaminp/six

On stack track stuff. To get local dat, the best new way is to

def test_exception(self,*args,**kws):
    try:
        localTest = 'hello world'
        raise ValueError("here")
    except:
        cgmGEN.log_tb()
        raise

If you don't need the local data the new wrappers work fine

[jjburton]

Week 01.01.23

• [ ] Production in 2023 and see where we get
• [ ] Can we use rigBlock modules from py3 in py2 build successfully?

Muzzle issues

Was getting bad data sorts from old usage. new usage sorts

_d = {0: 'orbFrontLeft',
     4: 'jawTopLeft',
     2: 'orbLeft',
     3: 'cheek_0_3_left',
     1: 'cheek_0_1_left'}
for ii,handle in list(_d.items()):#...no worky
    print("{} | {}".format(ii,handle))

for ii in sorted(_d):#...works as expected
    print("{} | {}".format(ii,_d[ii]))

Experiment with Moving modules over from Py3 to Py2 builds to see if all break or just certain ones

• [x] cgmGeneral
• [x] Core
• [x] MRS
• [x] core.rig

reload was the biggest thing that caused issues in code between modules so wrote a simple bridge for cgmGEN

_b_py3 = False
if platform.python_version().startswith('3'):
    _b_py3 = True
    _reloadMod = importlib.reload
else:
    _reloadMod = reload    

[jjburton]

Test tools in Maya 2023

• [x] animFilters
• [x] Locinator
• [x] Poser
• [x] ---Load Pose
• [x] ---Create dir
• [x] Animate
• [x] ---spaceswitching
• [x] ---base functions
• [x] Builder
• [x] ---build template
• [x] ---drax work
• [x] Scene
• [x] ---export
• [x] ---anim
• [ ] ---rig

01.04

• [x] Shots
• [x] Raycasting create
• [x] Raycast snap
• [ ] RandomizeAttr | same function as 2018. Needs to be retooled to work with nudge and clean up functionality
• [x] SetTools
• [x] BlockDat,ShapeDat,
• [x] AttrTools
• [ ] TransformTools
• [ ] SnapTools
• [ ] DynParentTool

Blockdat -- added blockparent string validation with new call SEARCH.find_from_string to deal with 'cgmRigBlocks|BlockParent' specifically. Working better