[BUG] [Formatter] IndexError: list index out of range

[palansher]

Thanks for useful tool!

I understand that formatting is beta feature.

Just a bug reporting.

System Info

• OS: e.g. ubuntu 20.04
• Python Version 3.8
• djLint Version (1.7.0)
• template language: e.g. twig

Issue

--check and --reformat follows to Phyton exception in some twig files. But --lint passed ok these files.

djlint view/admin/driver_form.html.twig --check --profile=nunjucks

concurrent.futures.process._RemoteTraceback:                                                                                                                                                          
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/vladp/.local/lib/python3.8/site-packages/djlint/__init__.py", line 259, in process
    output["format_message"] = reformat_file(
[problem_page.html.twig.txt](https://github.com/Riverside-Healthcare/djLint/files/9075293/problem_page.html.twig.txt)
config, this_file)
  File "/home/vladp/.local/lib/python3.8/site-packages/djlint/reformat.py", line 23, in reformat_file
    indented = indent_html(condensed, config)
  File "/home/vladp/.local/lib/python3.8/site-packages/djlint/formatter/indent.py", line 183, in indent_html
    tmp = re.sub(
  File "/home/vladp/.local/lib/python3.8/site-packages/regex/regex.py", line 278, in sub
    return pat.sub(repl, string, count, pos, endpos, concurrent, timeout)
  File "/home/vladp/.local/lib/python3.8/site-packages/djlint/formatter/attributes.py", line 271, in format_attributes
    attributes = format_template_tags(config, attributes)
  File "/home/vladp/.local/lib/python3.8/site-packages/djlint/formatter/attributes.py", line 218, in format_template_tags
    attributes = add_indentation(config, attributes)
  File "/home/vladp/.local/lib/python3.8/site-packages/djlint/formatter/attributes.py", line 44, in add_indentation
    start_test = (
IndexError: list index out of range
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/vladp/.local/bin/djlint", line 8, in <module>
    sys.exit(main())
  File "/home/vladp/.local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/vladp/.local/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/vladp/.local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/vladp/.local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/vladp/.local/lib/python3.8/site-packages/djlint/__init__.py", line 216, in main
    file_errors.append(future.result())
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
IndexError: list index out of range

How To Reproduce

problem file is attached problem_page.html.twig.txt

[christopherpickering]

Hi @palansher thanks for reporting. I wasn't able to get an error here - can you send a screenshot of the terminal? Do you have a pyproject.toml or .djlintrc with settings in it?

Thanks!

[christopherpickering]

I tried out with the attached file on ubuntu as well (looks like you are in ubuntu?)

docker run -i -t ubuntu /bin/bash
apt-get update
apt-get install -y python3 python3-pip wget
python3 -m pip install djlint
wget -O error.txt https://github.com/Riverside-Healthcare/djLint/files/9075282/problem_page.html.twig.txt
djlint error.txt --check

I get "1 file will be updated"

[palansher]

Hello! First, I want to thank you for the idea of ​​such comfortable docker-way testing!

I got miracle and could not repeat this Phyton error on my Ubuntu. Something strange.

Am I happy? Not really.

1) Because --reformat breaks my twig files on comment lines. It is important to hear your opinion for that:

Every time I do --reformat (or do the same from VS Code interface), the twig comments shifts right-side. More and more right-side. Please look at video. I attached two .twig files that I tested on recording. You can see the problem row numbers in VS Code.

https://youtu.be/1vGQyaUeVTg

https://youtu.be/iuV3NP3lncI

top_settings.html.twig.txt driver_form.html.twig.txt

Same result I got in your pure docker environment (djlint driver_form.html.twig.txt --reformat ..).

---

2) Also, another point pls - not problem, but aesthetic issue: The html elements indented correctly during reformat - by chosen tab settings (3, 5 .. spaces) But element's properties indented differently - not divisible by tab setting. And this leads to the editor's highlight, that I need to fix manually:

It would be good to indent everything by tab size.

[christopherpickering]

Thanks, I'll take a look at the new files 👍🏽

[christopherpickering]

On my mac I cannot reproduce it, but I can on docker+ubuntu.

[palansher]

On my mac I cannot reproduce it, but I can on docker+ubuntu. at least you caught it :) I am ready to assist some how .. to diagnose

[palansher]

On my mac I cannot reproduce it, but I can on docker+ubuntu.

at least you caught it :) I am ready to assist some how .. to diagnose

[christopherpickering]

Thanks, here's a simple example that is having a problem on ubuntu:

{% if %}
    {#
      line
    #}
{% endif %}

It seems like it is multi line {# #} inside another {% %} statement.

[palansher]

It seems like it is multi line {# #} inside another {% %} statement.

Is it illegal in twig syntax ?

[christopherpickering]

No, I think it should be good. I think I've got it fixed.

Can you test out on the dev branch? If you are using pip on the test ubuntu:

apt-get install -y git
pip3 install git+https://github.com/Riverside-Healthcare/djlint.git@dev

[palansher]

Can you test out on the dev branch?

Hello! I tried original attached .twig files from this post above.

Notice: After pip3 install git+https://github.com/Riverside-Healthcare/djlint.git@dev I got djlint, version 1.4.0, not dev version. Maybe it's OK.

On top_settings.html.twig.txt I got no problems. It seems formatted well. No comment blocks shifted.

About driver_form.html.twig.txt:

1. After reformat, the hierarchy text structure became flat:

Proof: driver_form.html.twig.reformatted.txt

2. just wish: it might be better to remove unnecessary spaces during reformat:

[palansher]

Good day!

One more on your dev branch:

if in this file (below), the comment exists: {# value="{{ driverId|default(' Новый водитель') }}"/> #} on line 9,

we have false positive:

driver_form.html.twig.falsepos.txt

[christopherpickering]

Thanks! I think I have these addressed in the latest push on dev. If you want to reinstall from dev and test it, I would appreciate it!

[christopherpickering]

:tada: This issue has been resolved in version 1.7.1 :tada:

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot :package::rocket:

[palansher]

Good day!

Sorry, I could not test dev version. Today I installed 1.7.1. Same result on exactly same file:

maybe it is some cache issue ?

[palansher]

Unfortunately, the comment formatting issue also persists: https://youtu.be/Y82egHOCsG4

on the same file: driver_form.html.twig

ver 1.7.1

[christopherpickering]

Thanks, is there a way to verify that the vscode python env is using the same version as your shell in the pic? Im wondering if the updated version is in a different python env.

Im not sure how to check that, maybe @monosans has an idea, he's better than I at vscode 👍

[palansher]

Same from cli (without VS Code) :

vladp@dev:~$ python 
Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

[palansher]

cli check also do strange staff with twig comment:

djlint --check /home/vladp/dev/gtdev/view/admin/test/driver_form.html.twig

[christopherpickering]

@palansher is it possible for you to send your latest files? When I run --lint on the driver_form.html.twig, I get no H025 errors:

Also , reformatting, I get a perfect file.

docker run -i -t ubuntu /bin/bash
apt-get update
apt-get install -y python3 python3-pip wget
python3 -m pip install djlint
wget -O error.txt https://github.com/Riverside-Healthcare/djLint/files/9114850/driver_form.html.twig.txt
djlint error.txt --reformat # this fixes the file
djlint error.txt --check # 0 errors found
djlint error.txt --lint # 0 H025 errors found

Maybe with new files I can reproduce it.

[palansher]

Dear Christopher,

Before I will send you the file again, would you pls help me (give me an idea) to check the probability that my Ubuntu 20.04 Python installation /config is broken. It can save time for our diagnosis. I got this error message. Maybe it is the reason we have different results. Thank you! Unfortunately, I am not experienced in Python.

$ pip list
Package                Version
---------------------- --------------------
ansible                5.8.0
ansible-core           2.12.6
attrs                  19.3.0
Automat                0.8.0
blinker                1.4
certifi                2019.11.28
chardet                3.0.4
click                  8.1.3
colorama               0.4.5
command-not-found      0.3
configobj              5.0.6
constantly             15.1.0
cryptography           2.8
dbus-python            1.2.16
distlib                0.3.4
distro                 1.4.0
distro-info            0.23ubuntu1
djlint                 1.7.1
entrypoints            0.3
fail2ban               0.11.1
filelock               3.7.0
html-tag-names         0.1.2
html-void-elements     0.1.0
httplib2               0.14.0
hyperlink              19.0.0
idna                   2.8
importlib-metadata     4.12.0
incremental            16.10.1
iniparse               0.4
Jinja2                 3.1.2
jsonpatch              1.22
jsonpointer            2.0
jsonschema             3.2.0
keyring                18.0.1
language-selector      0.1
launchpadlib           1.10.13
lazr.restfulclient     0.14.2
lazr.uri               1.0.3
MarkupSafe             2.1.1
more-itertools         4.2.0
netifaces              0.10.4
oauthlib               3.1.0
packaging              21.3
pathspec               0.9.0
pexpect                4.6.0
pip                    22.1.2
platformdirs           2.5.2
pyasn1                 0.4.2
pyasn1-modules         0.2.1
PyGObject              3.36.0
PyHamcrest             1.9.0
pyinotify              0.9.6
PyJWT                  1.7.1
pymacaroons            0.13.0
PyMySQL                0.9.3
PyNaCl                 1.3.0
pyOpenSSL              19.0.0
pyparsing              3.0.9
pyrsistent             0.15.5
pyserial               3.4
python-apt             2.0.0+ubuntu0.20.4.7
python-debian          0.1.36ubuntu1
PyYAML                 6.0
regex                  2022.6.2
requests               2.22.0
requests-unixsocket    0.2.0
resolvelib             0.5.4
SecretStorage          2.3.1
service-identity       18.1.0
setuptools             45.2.0
simplejson             3.16.0
six                    1.14.0
sos                    4.3
ssh-import-id          5.10
systemd-python         234
tomli                  2.0.1
tqdm                   4.64.0
Twisted                18.9.0
ubuntu-advantage-tools 27.9
ufw                    0.36
unattended-upgrades    0.1
urllib3                1.25.8
virtualenv             20.14.1
wadllib                1.3.3
wheel                  0.37.1
zipp                   1.0.0
zope.interface         4.7.1
--- Logging error ---
Traceback (most recent call last):
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_internal/utils/logging.py", line 177, in emit
    self.console.print(renderable, overflow="ignore", crop=False, style=style)
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_vendor/rich/console.py", line 1752, in print
    extend(render(renderable, render_options))
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_vendor/rich/console.py", line 1390, in render
    for render_output in iter_render:
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_internal/utils/logging.py", line 134, in __rich_console__
    for line in lines:
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_vendor/rich/segment.py", line 245, in split_lines
    for segment in segments:
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_vendor/rich/console.py", line 1368, in render
    renderable = rich_cast(renderable)
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_vendor/rich/protocol.py", line 36, in rich_cast
    renderable = cast_method()
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_internal/self_outdated_check.py", line 130, in __rich__
    pip_cmd = get_best_invocation_for_this_pip()
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_internal/utils/entrypoints.py", line 58, in get_best_invocation_for_this_pip
    if found_executable and os.path.samefile(
  File "/usr/lib/python3.8/genericpath.py", line 101, in samefile
    s2 = os.stat(f2)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/pip'
Call stack:
  File "/home/vladp/.local/bin/pip", line 8, in <module>
    sys.exit(main())
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 70, in main
    return command.main(cmd_args)
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 101, in main
    return self._main(args)
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    self.handle_pip_version_check(options)
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 148, in handle_pip_version_check
    pip_self_version_check(session, options)
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_internal/self_outdated_check.py", line 237, in pip_self_version_check
    logger.info("[present-rich] %s", upgrade_prompt)
  File "/usr/lib/python3.8/logging/__init__.py", line 1446, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.8/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 954, in handle
    self.emit(record)
  File "/home/vladp/.local/lib/python3.8/site-packages/pip/_internal/utils/logging.py", line 179, in emit
    self.handleError(record)
Message: '[present-rich] %s'
Arguments: (UpgradePrompt(old='22.1.2', new='22.2.1'),)

[ruanmed]

Hello @christopherpickering ,

I reproduced this error running djlint <my_600_lines_file>.html --reformat --profile jinja with djlint==1.9.4 :

concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "C:\Python37\lib\concurrent\futures\process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "[***]\lib\site-packages\djlint\__init__.py", line 273, in process
    output["format_message"] = reformat_file(config, this_file)
  File "[***]\lib\site-packages\djlint\reformat.py", line 28, in reformat_file
    beautified_code = indent_html(condensed, config)
  File "[***]\lib\site-packages\djlint\formatter\indent.py", line 188, in indent_html
    tmp,
  File "[***]\lib\site-packages\regex\regex.py", line 278, in sub
    return pat.sub(repl, string, count, pos, endpos, concurrent, timeout)
  File "[***]\lib\site-packages\djlint\formatter\attributes.py", line 271, in format_attributes       
    attributes = format_template_tags(config, attributes)
  File "[***]\lib\site-packages\djlint\formatter\attributes.py", line 218, in format_template_tags    
    attributes = add_indentation(config, attributes)
  File "[***]\lib\site-packages\djlint\formatter\attributes.py", line 59, in add_indentation
    )[-1]
IndexError: list index out of range
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "[***]\Scripts\djlint.exe\__main__.py", line 7, in <module>
  File "[***]\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "[***]\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "[***]\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "[***]\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Python37\lib\contextlib.py", line 74, in inner
    return func(*args, **kwds)
  File "[***]\lib\site-packages\djlint\__init__.py", line 230, in main
    file_errors.append(future.result())
  File "C:\Python37\lib\concurrent\futures\_base.py", line 428, in result
    return self.__get_result()
  File "C:\Python37\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
IndexError: list index out of range

And fixed it locally on my machine by altering the djlint\formatter\attributes.py file on line 59.

...
        start_test = (
            list(
                re.finditer(
                    re.compile(
                        r"^.*?(?=" + config.template_indent + r")", re.I | re.X | re.M
                    ),
                    attributes.splitlines()[0].strip(),
                )
            )
            + list(
                re.finditer(
                    re.compile(r"^<\w+\b\s*[^\"']+?[\"']", re.M),
                    attributes.splitlines()[0].strip(),
                )
            )
----        )[-1]
++++        )
++++        if start_test:
++++            start_test = start_test[-1]
...

This fix seems cohesive with line 72:

            indent_adder = len(start_test.group()) - base_indent if start_test else 0

After that the reformat worked as intended. Not sure about side effects though.

I can submit as pull request if needed.

[christopherpickering]

Great, thank you! Any chance you can pinpoint which line of html is causing it? Try removing chunks of the file, 50% at a time until you get to a line throwing the error?

I'd like to be able to add a test to make sure it doesn't happen again, but am happy if you start a pr! Thanks for finding a fix!

[ruanmed]

Sure, found out what was causing the issue on my side:

<select multiple
      class="selectpicker show-tick"
      id="device-select"
      title="">
</select>

The HTML \<select> multiple attribute apparently causes it.

Removing it or relocating it to after the other attributes, the issue does not happen.

From the problem_page.html.twig.txt originally provided by the BUG reporter I think the attributes that caused the issue are:

readonly attribute on line 23:

                                            <input readonly
                                                class="form-control"
                                                type="text"
                                                name="driver_id"
                                                value="{{ id|default(' Новый водитель') }}"/>

So the root cause seems to be any boolean HTML tag attribute that is placed just after the tag start.

[ruanmed]

@christopherpickering , I've submitted the fix with some tests as a pull request on #326 . Please fell free to edit it if needed.

If you have the time, could you please test if this fix resolves the issue on your environment, @palansher ?

[palansher]

Hello @christopherpickering !

If you have the time, could you please test if this fix resolves the issue on your environment, @palansher ?

I fixed my Python 3.8 environment (some system error messages).

Upgraded to v 1.9.4

I did not have time to check "comment formatting issue", but false positive issue still the same:

the file is also the same (I just renamed it to not mix it with working files): test_driver_form.html.twig.txt

[christopherpickering]

Thanks @ruanmed ! This is great! Im on vacation for a week and will check put the pr when im back.

[aarongoldenthal]

I had the same error with the following nunjucks line (in Windows and Linux containers [Debian Bullseye and Alpine]):

<ol reversed class="postlist" style="counter-reset: start-from {{ (postslistCounter or postslist.length) + 1 }}">

If split up, as follows, there is no error:

{% set count %}{{ (postslistCounter or postslist.length) + 1 }}{% endset %}
<ol reversed class="postlist" style="counter-reset: start-from {{ count }}">

I tested with the fix in #326 and it did resolve the problem.

[christopherpickering]

@palansher I am still not able to reproduce the linter output. Can you try reducing the file (split in half) until you find the lines that are triggering the error? Please open it in a separate issue as it doesn't appear to be related to the index error.

Thanks @ruanmed and @aarongoldenthal !

[christopherpickering]

:tada: This issue has been resolved in version 1.9.5 :tada:

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot :package::rocket: