WIP: width aware, pretty print formatter

[domluna]

There are some kinks but I'd like you folks to have a look. Contrary to the current approach which makes precision edits to the file, this approach outputs a canonical pretty printed version of the file. The canonical version is tuned by the indent_width (spaces of an indent) and max_width, the maximum width of a line in the file. Current width adjustable types are:

MacroCall [x] 
BinaryOpSyntaxCall/BinaryOpCall [x]
Conditional [x]
WhereOpCall [x]
Call [x]
Parameters [x]
TupleH [x]
Vect [x]
InvisBrackets [x]
Braces [x]
Export [x]
Import [x]
Using [x]

This also aligns comments based on the indentation context, removes unnecessary newlines, i.e. if you have 5 consecutive newlines somewhere it'll cut it down to 1.

• pretty is a recursive function returns an Edit (the pretty printed output).
• merge_edits combines two Edits.
• newline_ranges creates an array of line ranges, this is used to gather the start and end lines of an Edit which are used when merging to determine whether they should be joined on the same line and separate lines, also it's used collect comments which are in between Edits.

Example outputs

I think this is a good example output to compare:

https://github.com/JuliaDiff/ChainRules.jl/blob/master/src/rules.jl output

This one is a dump of possible combinations output

I haven't done paid any attention to performance but here is a sample benchmark formatting https://github.com/MikeInnes/MacroTools.jl/blob/master/src/utils.jl.

output

# master
julia> @btime DocumentFormat.format0($s);
  40.233 ms (940472 allocations: 27.90 MiB)

# this branch
julia> @btime DocumentFormat.format($s);
  5.629 ms (40119 allocations: 2.49 MiB)

# functions that don't have anything to do with formatting
julia> @btime CSTParser.parse($s, true);
  1.662 ms (7269 allocations: 381.94 KiB)

julia> @btime collect(CSTParser.Tokenize.tokenize($s));
  580.955 μs (8752 allocations: 538.81 KiB)

TODO

• [ ] More tests
• [ ] Output triple strings properly. I believe this is an issue with CSTParser not labelling the tokens correctly. This doesn't affect docstrings.
• [ ] Unicode operator lengths might be > 1, currently they are 1. Example: lastindex("→") = 1, while lastindex("→ ") = 4.
• [ ] Refactor pretty function into 2-3 separate functions.

[davidanthoff]

Thanks for putting so much time into this! Here are some quick reactions: 1) whatever we do, I think we should wait with such a large refactoring until we have shipped the VS Code extension v0.11, and then tackle this for v0.12. 2) I think line breaks could be an excellent optional feature that folks can opt-into, but I'm a bit nervous about making that the default, i.e. I think I would prefer if we could keep the default for VS Code to not change anything about line breaks (neither add nor remove any). An option that allows that to change though would be great!

A more general comment: it would be awesome if at some point we could enable the automatic formatting of a line during editing, i.e. when one starts a new line. I think the LS protocol supports this, and mainly we would have to enable that. I don't know how that would interact with a whole doc approach here, but I think for that feature, we would need a story that allows us to only format a single line.

[domluna]

Is there an issue with how VSCode handles line breaks?

[davidanthoff]

Is there an issue with how VSCode handles line breaks?

I'm not clear what you mean by this :) Can you elaborate a bit more?

[ZacLN]

Hi Dom, glad you took this up! I'll have time to review this thoroughly on the weekend

[domluna]

@davidanthoff @ZacLN I've continued working on this https://github.com/domluna/JLFmt. It works much better now.

Gonna close this PR since its outdated at this point.

[davidanthoff]

Is there a chance that we could eventually make this available here as well somehow? I'm sorry that I can't be of more help, this part of the code is all in @ZacLN's hands :)

[domluna]

@davidanthoff Why not just switch to JLFmtonce the tires are kicked in? Wouldn't it just be switching using DocumentFormat to using JLFmt?

[davidanthoff]

I don't really know, @ZacLN did all the work here. If that works, it would be great!

[domluna]

The formatter at JLFmt is completely different than the one here, so merging it back here doesn't make much sense. The actual API is exactly the same though, just call format (which is exported). So it should be as simple as

using JLFmt # instead of DocumentFormat
format(s)

[norru]

I'd love to have an option in VSC to switch formatting backend from DocumentFormat to JLFmt/JuliaFormatter!