asma does not assemble when used with Python 3.7.1

[srorso]

asma does not assemble source files when used with Python 3.7.1. The issue does not occur when asma is used with Python 3.6.7, the most recent release before 3.7.1.

The following was generated on Windows 10 update 1809 in a non-elevated command prompt using Python 3.7.1 and the most recent version of SATK.

C:\common\Hercules\SATK-master\samples\asma>..\..\tools\asma.py hello.asm
asma.py Copyright (C) 2014-2017 Harold Grovesteen
Phase 'pass0_1' INTERNAL ERROR WHILE PROCESSING THIS STATEMENT
    [47]          SPACE 1
Traceback (most recent call last):
  File "C:\common\Hercules\SATK-master\tools/lang\lexer.py", line 541, in __next__
    raise StopIteration
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\common\Hercules\SATK-master\tools\asma.py", line 419, in <module>
    ASMA(tool,dm).run()
  File "C:\common\Hercules\SATK-master\tools\asma.py", line 149, in run
    result=self.assembler.assemble(filename=self.source)
  File "C:\common\Hercules\SATK-master\asma\assembler.py", line 1525, in assemble
    return self.SP.run(self,filename)
  File "C:\common\Hercules\SATK-master\asma\assembler.py", line 3306, in run
    return self.process()   # Return the result of last phase run
  File "C:\common\Hercules\SATK-master\asma\asmbase.py", line 1656, in process
    self.result=phase(self.asm,fail=self.asm.fail)  # Execute it!
  File "C:\common\Hercules\SATK-master\asma\assembler.py", line 3152, in Pass0_1
    self.getStmts0_1(asm,fail=fail,debug=debug)
  File "C:\common\Hercules\SATK-master\asma\assembler.py", line 3136, in getStmts0_1
    s.Pass0(asm,macro=mb)
  File "C:\common\Hercules\SATK-master\asma\asmstmts.py", line 4860, in Pass0
    super().Pass0(asm,macro=macro,debug=debug,trace=trace)
  File "C:\common\Hercules\SATK-master\asma\asmstmts.py", line 1206, in Pass0
    self.parse_sep(asm,debug=pdebug)     # May raise assembler.AssemblerError
  File "C:\common\Hercules\SATK-master\asma\asmstmts.py", line 795, in parse_sep
    self.P0_operands=asm.PM.parse_sep(self,debug=debug)
  File "C:\common\Hercules\SATK-master\asma\parsers.py", line 1034, in parse_sep
    result=parser.parse_operands(stmt,debug=debug)
  File "C:\common\Hercules\SATK-master\asma\asmopnd.py", line 385, in parse_operands
    opnd=self.parse_operand(stmt,opnd,n,debug=debug)
  File "C:\common\Hercules\SATK-master\asma\asmopnd.py", line 329, in parse_operand
    scope=self.parse(opnd.text,scope=scope,fail=False)
  File "C:\common\Hercules\SATK-master\asma\asmbase.py", line 731, in parse
    return super().parse(string,scope=scope,fail=fail)
  File "C:\common\Hercules\SATK-master\tools/lang\fsmparser.py", line 104, in parse
    for token in self.lex.analyze(string,pos=0,fail=fail,lines=lines,line=line):
  File "C:\common\Hercules\SATK-master\tools/lang\lexer.py", line 627, in analyze
    for t in self.tokenize(string,pos,fail=fail,lines=lines,line=line):
RuntimeError: generator raised StopIteration

C:\common\Hercules\SATK-master\samples\asma>

Assembly using Python 3.6.7 generates the following:

C:\common\Hercules\SATK-master\samples\asma>..\..\tools\asma.py hello.asm
asma.py Copyright (C) 2014-2017 Harold Grovesteen
[3485] MNOTE *,ARCHLVL - ARCHITECTURE LEVEL SET - 3

C:\common\Hercules\SATK-master\samples\asma>

Please let me know if you would like the problem reproduced on Linux, BSD, or macOS platforms.

Many thanks for taking a look at this.

[s390guy]

Python PEP 479 - "Change StopIteration handling inside generators" proposes that the normal termination of a generator by a StopIteration exception should be changed to a RuntimeError exception. Previous generators did not require the StopIteration exception to be caught by the run-time code. They were consumed by the generator. As of the introduction of this Python change in 3.6.0, all later releases require explicit capture of the StopIteration exception by the generator function, otherwise a RuntimeError is raised. This became explicitly the behavior of Python 3.6 and beyond. Hence the exception described in the previous comment.

There is no work around for this issue. ASMA code will require changes. Interestingly, one of the examples in PEP 479 involves parsing, which is the exact situation within ASMA. The example suggests this will be relatively simple. Experience and backward compatibility will determine if this is the case.

The version of Python on which this issue was found is no longer readily available for the version of Linux used for development. The plan forward is to build the latest version of Python for testing of existing code (verification of the problem) and then corrections and ultimately backward compatibility.

[s390guy]

This has been fixed by removal of StopIteration from the code.