Here is the code that I tried unsuccessfully
wireframeMode = osg.PolygonMode:setMode(osg.PolygonMode.Mode.FRONT_AND_BACK,osg.PolygonMode.Mode.FILL)
mystate:setAttributeAndModes(wireframeMode)
This code runs without errors but draws a filled polygon regardless of the numeric enums put in:
wireframeMode = osg.PolygonMode()
wireframeMode:setMode(0, 1)
mystate:setAttributeAndModes(wireframeMode)
Found by @chiefkikio: