Much more powerful gq replacement function

[JelteF]

This fixes #27 and contains the commit from #28 as well. It contains quite a lot of edge cases, like when there is no splitting possible and when trying to reformat docstrings. I'm not sure how to write tests for this because it would have to test for something different than the other tests.

[hynek]

Playing with it, it does this:

def f():
    f = ("safdj asldfj  öalsdfj sdölj aölfj asdöflj asdöflj asölfj asdölfj asdölfj  dsöalfj ")

to

def f():
    f =
    ("safdj asldfj  öalsdfj sdölj aölfj asdöflj asdöflj asölfj asdölfj asdölfj  dsöalfj ")

Trying to format

def f():
   """
   safdj asldfj  öalsdfj sdölj aölfj asdöflj asdöflj asölfj asdölfj asdölfj  dsöalfj 
   """

in its entirety leads to

def f():
   """ safdj asldfj  öalsdfj sdölj aölfj asdöflj asdöflj asölfj asdölfj asdölfj  dsöalfj """

Could you explicitly spell out what this is supposed to achieve again? :)

Thanks again!

[blueyed]

@JelteF Any update on this?

[blueyed]

I'm not sure how to write tests for this because it would have to test for something different than the other tests.

What do you mean? There are tests for gq already, which could be used as a base.

[JelteF]

I forgot about this a bit. More up to date code, that fixes some bugs, is over at https://github.com/pangloss/vim-javascript I currently don't have a lot of time to do a lot of open source stuff. If you want you could fork from my repo and apply the changes you mentioned plus the fixes done over at the javascript plugin. Some links to specific commits/issues, that could be useful: https://github.com/pangloss/vim-javascript/pull/229 https://github.com/pangloss/vim-javascript/commit/18106a3db0ce01b74c9b65b30870af2f04e20beb

[blueyed]

@JelteF Any update?

[JelteF]

I just updated the code based on the version that was available in vim-javascript until recently. To be clear of the point of this branch is that gq does not break on spaces inside single line strings. The following snippet is what currently works and what doesn't:

# currently working

(222222222222222222222222, 'aaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaa aaayxza  uuu uubce uae aeuoe eaueo')
a = list('ueauea euaeouaeo uea', 'ueaeu eau oeuao ', 'ueaue eauoeu ', 'ueau uaeu aeuaeoua')
# ueau  euaeou  euaeo ua eou aeou eao ueo uaeo u eo u eaoueo ua eou aeo u eaou aeo ue a uoe uaoe u aeou

def a():
    return ('safdj asldfj  öalsdfj ' 'sdölj aölfj asdöflj asdöflj asölfj asdölfj asdölfj  dsöalfj ' 'euaeouaeuaeoua eau eou ae uae ouaeou aeou aeou ' 'ueaue eaueuaeuaeoueo eaueoua euaeouaeouao ')

# currently not working
def b():
    # A check should be added to see if the split position is between brackets
    # otherwise it should cancel
    return ('safdj asldfj  öalsdfj sdölj aölfj asdöflj asdöflj asölfj asdölfj asdölfj  dsöalfj q')

# Here no split position is available, for some reason the cursor moves to the first column after pressing gqq instead of doing nothing.
('u eua eua uae oueao ueo ueoa ueo uaeo aaaaaaaaaaaaaaaaaaaaaaaaa uea e eaueou euauea ouaoe aaaaaaaaa aaayxza  uuu uubc')

def c():
    # docstrings should be done with normal gq like comments

    """ safdj asldfj  öalsdfj sdölj aölfj asdöflj asdöflj asölfj asdölfj asdölfj  dsöalfj """

I'm not very experienced with vimscript, so if you know of a simple way or a pointer to fix one of the three bugs below please let me know.

[blueyed]

It might be worth adding some tests for this.. :)

For the bugs, I recommend calling it manually using :debug call GetPythonPEPFormat(…) and then step through what it is doing (use s, n, echo, etc).

[JelteF]

Tests are definitely needed. I'll add them later when I have some time again. I'll try to fix the bugs then as well.

[JelteF]

I fixed the issues you had with the pull request. Creating tests is not as simple as I would have thought, because mine does not require checking the current indent, but instead the place where the indent took place.

[JelteF]

All bugs that I found myself are fixed now:

These are the tests I'm currently manually doing and all of them are sane.

(222222222222222222222222, 'aaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaa aaayxza  uuu uubce uae aeuoe eaueo')
[222222222222222222222222, 'aaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaa aaayxza  uuu uubce uae aeuoe eaueo']
{222222222222222222222222, 'aaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaa aaayxza  uuu uubce uae aeuoe eaueo'}
a = list('ueauea euaeouaeo uea', 'ueaeu eau oeuao ', 'ueaue eauoeu ', 'ueau uaeu aeuaeoua')
# ueau  euaeou  euaeo ua eou aeou eao ueo uaeo u eo u eaoueo ua eou aeo u eaou aeo ue a uoe uaoe u aeou

def a():
    return ('safdj asldfj  öalsdfj ' 'sdölj aölfj asdöflj asdöflj asölfj asdölfj asdölfj  dsöalfj ' 'euaeouaeuaeoua eau eou ae uae ouaeou aeou aeou ' 'ueaue eaueuaeuaeoueo eaueoua euaeouaeouao ')

def b():
    a = ['safdj asldfj  öalsdfj sdölj aölfj asdöflj asdöflj asölfj asdölfj asdölfj  dsöalfj q']
    b = {'safdj asldfj  öalsdfj sdölj aölfj asdöflj asdöflj asölfj asdölfj asdölfj  dsöalfj q'}
    return ('safdj asldfj  öalsdfj sdölj aölfj asdöflj asdöflj asölfj asdölfj asdölfj  dsöalfj q')

# Here no split position is available
('u eua eua uae oueao ueo ueoa ueo uaeo aaaaaaaaaaaaaaaaaaaaaaaaa uea e eaueou euauea ouaoe aaaaaaaaa aaayxza  uuu uubc')

def c():
    # docstrings should be done normally like regular strings

    """
    safdj asldfj  öalsdfj sdölj aölfj asdöflj asdöflj asölfj asdölfj asdölfj  dsöalfj
    """

[JelteF]

I just updated the pull request again with the new feature that allows breaking in strings to return valid python. I did break some of my tests again though and added some new ones:

(222222222222222222222222, 'aaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaa aaayxza  uuu uubce uae aeuoe eaueo')
[222222222222222222222222, 'aaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaa aaayxza  uuu uubce uae aeuoe eaueo']
{222222222222222222222222, 'aaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaa aaayxza  uuu uubce uae aeuoe eaueo'}
a = list('ueauea euaeouaeo uea', 'ueaeu eau oeuao ', 'ueaue eauoeu ', 'ueau uaeu aeuaeoua')
# ueau  euaeou  euaeo ua eou aeou eao ueo uaeo u eo u eaoueo ua eou aeo u eaou aeo ue a uoe uaoe u aeou

def a():
    return ('safdj asldfj  oalsdfj ' 'sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj ' 'euaeouaeuaeoua eau eou ae uae ouaeou aeou aeou ' 'ueaue eaueuaeuaeoueo eaueoua euaeouaeouao ')

def a():
    return ('safdj asldfj  oalsdfj asdoflj asolfj asdolfj asdolfj  dsoalfj ' 'euaeouaeuaeoua eau eou ae uae ouaeou aeou aeou ' 'ueaue eaueuaeuaeoueo eaueoua euaeouaeouao ')

def b():
    a = ['safdj asldfj  oalsdfj sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj q']
    b = {"safdj asldfj  oalsdfj sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj q"}
    # This one is broken (infinite loop)
    c = {"uuaeuaoeuaeouaeouaeosafdjasldfjoalsdfjsdoljaolfjasdofljasdofljasolfjasdolfjasdolfjdsoalfjq"}

    return ('safdj asldfj  oalsdfj sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj q')

('u eua eua uae oueao ueo ueoa ueo uaeo aaaaaaaaaaaaaaaaaaaaaaaaa uea e aueou uauea ouaoe aaaaaaaaa aaayxza  uuu uubc')
("u eua eua uae oueao ueo ueoa ueo uaeo aaaaaaaaaaaaaaaaaaaaaaaaa uea e eaueou uauea ouaoe aaaaaaaaa aaayxza  uuu uubc")

def c():
    # docstrings should be done normally like regular strings

    """
    safdj asldfj  oalsdfj sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj dsoalfj
    """

# These are broken (test for multiline is broken)
def b():
    a = ['safdj asldfj  oalsdfj sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj q']
    b = {"safdj asldfj  oalsdfj sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj q"}
    # This one is broken (infinite loop)
    c = {"uuaeuaoeuaeouaeouaeosafdjasldfjoalsdfjsdoljaolfjasdofljasdofljasolfjasdolfjasdolfjdsoalfjq"}

    return ('safdj asldfj  oalsdfj sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj q')

[JelteF]

I made gq much more powerful with the latest changes. It now not only always results in valid python code, but it makes it look quite good automatically as well. Please just try the test file below with the version of the library in this pull request to see what I mean.

There is now only one bug left that I know of (and it is not that big of a problem). One of the bugs was fixed by changing python-syntax, so this updated version is needed for everything to work: https://github.com/hdima/python-syntax/pull/55

The current testing method in this repository is hard to change for the tests that would be needed for this feature. It would probably be better to have an input file and do gqq on certain lines and make sure the output file is what is expected after these operations. Here is the updated file I use for manual testing at the moment, with the only bug that's left at the end (it contains a lot of edgecases, but some are also just double):

(222222222222222222222222, 'aaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaa aaayxza  uuu uubce uae aeuoe eaueo')
[222222222222222222222222, 'aaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaa aaayxza  uuu uubce uae aeuoe eaueo']
{222222222222222222222222, 'aaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaa aaayxza  uuu uubce uae aeuoe eaueo'}
a = list('ueauea euaeouaeo uea', 'ueaeu eau oeuao ', 'ueaue eauoeu ', 'ueau uaeu aeuaeoua')
# ueau  euaeou  euaeo ua eou aeou eao ueo uaeo u eo u eaoueo ua eou aeo u eaou aeo ue a uoe uaoe u aeou

def a():
    return ('safdj asldfj  oalsdfj ' 'sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj ' 'euaeouaeuaeoua eau eou ae uae ouaeou aeou aeou ' 'ueaue eaueuaeuaeoueo eaueoua euaeouaeouao ')

def a():
    return ('safdjaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaauaeuaeouaeouaeou' 'euaeouaeuaeoua eau eou ae uae ouaeou aeou aeou ' 'ueaue eaueuaeuaeoueo eaueoua euaeouaeouao ')

def b():
    a = ['safdj asldfj  oalsdfj sdolj aolfj sdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj q']
    b = {"safdj asldfj  oalsdfj sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj q"}

    return ('safdj asldfj  oalsdfj sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj q')

('u eua eua uae oueao ueo ueoa ueo uaeo aaaaaaaaaaaaaaaaaaaaaaaaa uea e aueou uauea ouaoe aaaaaaaaa aaayxza  uuu uubc')
("u eua eua uae oueao ueo ueoa ueo uaeo aaaaaaaaaaaaaaaaaaaaaaaaa uea e eaueou uauea ouaoe aaaaaaaaa aaayxza  uuu uubc")

def c():
    # docstrings should be done normally

    """
    safdj iü asldfj  oalsdfj sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj dsoalfj
    """

def b():
    a = ['safdj asldfj  oalsdfj sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj q', 'ueaueaou']
    b = {"safdj asldfj  oalsdfj sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj q"}
    return ('safdj asldfj  oalsdfj sdolj aolfj asdoflj asdoflj asolfj asdolfj asdolfj  dsoalfj q')

ceeuaeouaeu = {"uuaeuaoeuaeouaeouaeosafdjasldfjoalsdfjsdoljaolfjasdofljasdofljasolfjasdolfjasdolfjdsoalfjq", 'ueaeuoaeou eaueouaeouaeou'}
abc = "ueauoe eauoeau eaueoua eouaeo uaoeuaeou euaeo uaoe uaoeua euaeou eou aoeu aoeu aoeu aeoua oeu aoeu aoeuaeouaeou aeouaoeuaoeua"

def a():
    return ('safdjaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaauaeuaeouaeouaeou' 'euaeouaeuaeoua eau eou ae uae ouaeou aeou aeou ' 'ueaue eaueuaeuaeoueo eaueoua euaeouaeouao ')

(r'u eua eua uae oueao ueo ueoa ueo uaeo aaaaaaaaaaaaaaaaaaaaaaaaa uea e aueou uauea ouaoe aaaaaaaaa aaayxza  uuu uubc')
(r"u eua eua uae oueao ueo ueoa ueo uaeo aaaaaaaaaaaaaaaaaaaaaaaaa uea e eaueou uauea ouaoe aaaaaaaaa aaayxza  uuu uubc")
(b'u eua eua uae oueao ueo ueoa ueo uaeo aaaaaaaaaaaaaaaaaaaaaaaaa uea e aueou uauea ouaoe aaaaaaaaa aaayxza  uuu uubc')
(b"u eua eua uae oueao ueo ueoa ueo uaeo aaaaaaaaaaaaaaaaaaaaaaaaa uea e eaueou uauea ouaoe aaaaaaaaa aaayxza  uuu uubc")

('üüüüüüüüüü üüüüüüüüü üüüüüüüüü üüüüüüüüü üüüüüüüüü üüüüüüüüü ueauaeouaeouue üüüüüüüüüuuuuu')
('üüüüüüüüüü üüüüüüüüü üüüüüüüüü üüüüüüüüü üüüüüüüüü üüüüüüüüü üüüüüüüüüüüüüü üüüüüüüüü üüüüüüüüü üüüüüüüüü')
('uuuuuuuuuu uuuuuuuuu uuuuuuuuu uuuuuuuuu uuuuuuuuu uuuuuuuuu uuuuuuuuuuuuuu uuuuuuuuu uuuuuuuuu uuuuuuuuu')
('uuuuuuuuuu uuuuuuuuu uuuuuuuuu uuuuuuuuu uuuuuuuuu uuuuuuuuu uuuuuuuuuuuuuuu uuuuuuuuu uuuuuuuuu uuuuuuuuu')
('üüüüüüüüüü üüüüüüüüü üüüüüüüüü üüüüüüüüü üüüüüüüüü üüüüüüüüü uüüüüüüüüüüüüüü üüüüüüüüü üüüüüüüüü üüüüüüüüü')

# BUG: eats spaces directly after the breakpoint
("u eua eua uae oueao ueo ueoa ueo uaeo aaaaaaaaaaaaaaaaaaaaaaaaa                               uea e eaueou uauea ouaoe aaaaaaaaa aaayxza  uuu uubc")

[blueyed]

This looks awesome!

What do you think about using https://github.com/junegunn/vader.vim for new tests? I use it in several projects already, and could help with setting it up. We can keep the existing tests as they are, but I also found it always hard to work with them.

[blueyed]

There is now only one bug left that I know of (and it is not that big of a problem). One of the bugs was fixed by changing python-syntax, so this updated version is needed for everything to work: hdima/python-syntax#55

How does it behave with the syntax file from Vim? How are they different? https://github.com/vim/vim/blob/master/runtime/syntax/python.vim

[blueyed]

If it's about docstring strings, those are in the official syntax file: https://github.com/vim/vim/blob/master/runtime/syntax/python.vim#L112-L124 (IIRC the patch comes from or at least was driven by me.. ;))

[JelteF]

The vader testing seems much better suitable indeed. If you could set it up that would be great.

As for the syntax file, I needed to be able to differentiate between regular strings and multiline strings by using synattr, since different gq behaviour is needed. Looking at the regular python syntax file it should be easy to modify the code to work with that as well if you can request the name of the matchgroups instead of the name with synIDattr.

[JelteF]

I just checked, synIDattr can't return matchgroups, only name and a lot of styling stuff. So, the official syntax would also have to be changed to include multiline in the name.

[blueyed]

The vader testing seems much better suitable indeed. If you could set it up that would be great.

Will come back to this eventually, but I am lacking the time and energy for this currently.

pythonTripleQuotes is available on the quotes itself, using :echo reverse(map(synstack(1, 1), 'synIDattr(v:val,"name")')) on a file starting with """. (It's not available in the string however) You could therefore check if it's a string in general and then use searchpairpos('"""', '', '"""', 'b') to find it for the check - although that does not guarantee it to be the same string then - but maybe some of the existing methods can help with that.

[blueyed]

Please rebase this on master, which should give us a coverage report.

[JelteF]

I'm going to close this. I don't use it myself anymore, I've been using black for a while now to sortof get the same result.