VSG crashes when called with "all_phases" to analyse a nested generate followed by another generate.

[JHertz5]

Environment

$ python --version
Python 3.11.5
$ vsg --version
VHDL Style Guide (VSG) version: 3.25.0

Describe the bug This appears to be some kind of edge case that causes VSG to crash.

To Reproduce Steps to reproduce the behavior:

1. Create a file called test.vhd with the following contents:

   
   architecture rtl of regs is

begin

gen_1 : for gv_reg in t_subrange generate begin

gen_2 : if c_reg generate
begin

end generate;

end generate;

gen_3 : for gv_reg in t_reg generate

end generate;

end architecture;

2. run `vsg -f test.vhd --all_phases`
3. Observe the output

$ vsg -f test.vhd --all_phases multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib64/python3.11/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) ^^^^^^^^^^^^^^^^^^^ File "/vsg/lib64/python3.11/site-packages/vsg/apply_rules.py", line 119, in apply_rules oRules.check_rules( File "/vsg/lib64/python3.11/site-packages/vsg/rule_list.py", line 226, in check_rules oRule.analyze(self.oVhdlFile) File "/vsg/lib64/python3.11/site-packages/vsg/rules/align_tokens_in_region_between_tokens.py", line 85, in analyze lToi = self._get_tokens_of_interest(oFile) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/vsg/lib64/python3.11/site-packages/vsg/rules/align_tokens_in_region_between_tokens_when_between_tokens_unless_between_tokens.py", line 38, in _get_tokens_of_interest return oFile.get_tokens_bounded_by_token_when_between_tokens(self.left_token, self.right_token, self.lBetween[0], self.lBetween[1]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/vsg/lib64/python3.11/site-packages/vsg/vhdlFile/vhdlFile.py", line 249, in get_tokens_bounded_by_token_when_between_tokens return extract.get_tokens_bounded_by_token_when_between_tokens( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/vsg/lib64/python3.11/site-packages/vsg/vhdlFile/extract/get_tokens_bounded_by_token_when_between_tokens.py", line 8, in get_tokens_bounded_by_token_when_between_tokens lMyPairs = get_token_pairs([oLeft, oRight], [oStart, oEnd], oTokenMap) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/vsg/lib64/python3.11/site-packages/vsg/vhdlFile/extract/get_tokens_bounded_by_token_when_between_tokens.py", line 28, in get_token_pairs lLeft, lRight = oTokenMap.get_token_pair_indexes(lInnerPair[0], lInnerPair[1]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/vsg/lib64/python3.11/site-packages/vsg/token_map.py", line 61, in get_token_pair_indexes return extract_start_end_indexes(lStartIndexes, lEndIndexes) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/vsg/lib64/python3.11/site-packages/vsg/token_map.py", line 228, in extract_start_end_indexes return extract_indexes_from_pairs(lPairs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/vsg/lib64/python3.11/site-packages/vsg/token_map.py", line 274, in extract_indexes_from_pairs if pair[0] == iStartIndex:

IndexError: list index out of range
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<path>/vsg/bin/vsg", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "<path>/vsg/lib64/python3.11/site-packages/vsg/__main__.py", line 154, in main
    for tResult in pool.imap(f, enumerate(commandLineArguments.filename)):
  File "/usr/lib64/python3.11/multiprocessing/pool.py", line 873, in next
    raise value
IndexError: list index out of range
```

**Expected behavior**
I expect VSG to be able to handle this input.

[JHertz5]

Note - I've noticed that if I add a begin to gen_3 in the snippet above, the problem is no longer present.

[JHertz5]

I've dug deeper and discovered more about the problem. In this instance, the rule generate_400 is running, which calls align_tokens_in_region_between_tokens_when_between_tokens_unless_between_tokens, which in turn calls get_tokens_bounded_by_token_when_between_tokens to get all of the declarative tokens between token.for_generate_statement.generate_keyword (the for generate generate keyword) and token.generate_statement_body.begin_keyword (the for generate begin keyword) when inside of a for generate.

The problem arises when the function get_token_pair_indexes is called lower down to get the pairs of tokens. It uses get_token_indexes to get the indexes of the for generate generate keywords, and to get the indexes of the for generate begin keywords. The problem is that the second for generate doesn't have a matching begin keyword, but this function returns the begin keyword for the if generate as the matching begin for the second generate! I haven't yet dug in to the solution to this problem - perhaps the begin tokens should be divided up depending on which type of generate statement they are part of? The VHDL LRM defines generate_statement_body independently of the type of generate though, so there's probably a simpler solution.

[JHertz5]

I think I've solved it. Deeper within the stack, extract_start_end_indexes calls extract_pairs which returns [[29, 32], []] (note that the second list is empty). It then runs extract_indexes_from_pairs on this, and when it gets to the empty list, the line

            if pair[0] == iStartIndex:

causes the failure, because pair[0] (pair being the empty list) does not exist. I'll see if I can put in a fix for this.

[JHertz5]

I've raised a PR #1222 which fixes this problem.

[jeremiah-c-leary]

Evening @JHertz5 ,

I appreciate the pull request and your efforts to help others with similar issues before I could approve the pull request. I have been slammed at work and have been unable to work on VSG at all.

The pull request looks good and I will be merging it to master.

Again, thanks for your support.

Regards,

--Jeremy

[JHertz5]

Hi @jeremiah-c-leary

No problem at all! It was a fun debugging exercise which did not take long, and I'm glad to contribute a bit when you've given up so much of your free time to make this great tool!