Formatting is very slow

[jmcantrell]

I've just started trying out this package, and it's extremely slow. I just wanted to be sure I wasn't doing something wrong.

For a simple file, print("hello, world"), the following command takes over 20 seconds:

julia -e 'using JuliaFormatter; format_file("test.jl")'

I'm using the version in the arch linux repos (julia version 1.7.3).

[domluna]

on my machine, macbook air M1 (2020)

λ ~: time julia -e 'using JuliaFormatter; format_file("test.jl")'

julia -e 'using JuliaFormatter; format_file("test.jl")'  6.68s user 1.45s system 111% cpu 7.299 total

This is on 1.8 rc

time to first format is slow at the moment. It's something I'd like to tackle eventually. I think with the new tooling introduced in 1.8+ we could make some improvements

https://github.com/domluna/JuliaFormatter.jl/pull/599 is another option or if you use VSCode then this is already done for you.

[jmcantrell]

I'm wondering if something similar could be done to make a command line app that could be used in build processes or https://github.com/dense-analysis/ale. Is it noticeably faster when compiled?

[MilesCranmer]

You can use a sysimage for faster formatting. Here's a script to do this.

First, build the sys image by formatting an example project and saving the Julia state:

EXAMPLE_PROJECT=/Users/mcranmer/Documents/SymbolicRegression.jl
OUTPUT_SYSIMAGE=/Users/mcranmer/julia_formatter.so
FORMATTER="BlueStyle"

julia -O3 --threads=auto -e 'using Pkg; Pkg.activate(;temp=true); Pkg.add(["JuliaFormatter", "PackageCompiler"]); open("precompile_file.jl", "w") do io; write(io, "using JuliaFormatter; format(\"'$EXAMPLE_PROJECT'\", '$FORMATTER'())"); end; using PackageCompiler; create_sysimage(["JuliaFormatter"]; sysimage_path="'$OUTPUT_SYSIMAGE'", precompile_execution_file="precompile_file.jl")'

(Note that this will set up a temp environment for you to build it; no need to worry about installing PackageCompiler yourself)

This will saves a sysimage to /Users/mcranmer/julia_formatter.so. Then, I can startup much quicker with this:

julia --threads=auto -J /Users/mcranmer/julia_formatter.so  -e 'using JuliaFormatter; format(".", BlueStyle())'

On my system, this takes me from 13.04 s down to 1.56 s!

[MilesCranmer]

I come back to this issue all the time when I forget how to do this. So, here are some bash functions that I put into my .bashrc. Hopefully this is helpful to others:

export JULIA_FORMATTER_SO=$HOME/julia_formatter.so

function jformat {
    project=${1:-$PWD}
    OLD=$PWD
    if [ -e $JULIA_FORMATTER_SO ]; then
    else
        echo "Could not find $JULIA_FORMATTER_SO, so will build it..."
        build_jformat
    fi

    cd $project
    julia --startup-file=no --threads=auto -J $JULIA_FORMATTER_SO -O0 --compile=min -e 'using JuliaFormatter; format("."; verbose=true)'
    cd $OLD
}
function build_jformat {
    # Build a formatting image using an example project
    OLD=$PWD
    WORKDIR=$(mktemp -d)
    cd $WORKDIR
    git clone --depth 1 --quiet https://github.com/MilesCranmer/SymbolicRegression.jl  # Not used; just an example project with a lot of code to format. Change if you want.
    cd SymbolicRegression.jl
    { 
        julia --startup-file=no --compile=yes -O3 --threads=auto -e 'using Pkg; Pkg.activate(; temp=true); Pkg.add(["JuliaFormatter", "PackageCompiler"]); open("precompile_file.jl", "w") do io; write(io, "using JuliaFormatter; format(\".\")"); end; using PackageCompiler; create_sysimage(["JuliaFormatter"]; sysimage_path="'$JULIA_FORMATTER_SO'", precompile_execution_file="precompile_file.jl")'
    } || {
        echo "Building format file failed. Exiting."
    }
    cd $OLD
}

This gives you a very fast jformat command, which will format the current directory (default), or whatever directory you pass (e.g., jformat ~/my/julia/project).

The first time you run jformat, it will build the julia_formatter.so file:

Could not find /mnt/home/mcranmer/julia_formatter.so, so will build it...
/tmp/tmp.wvryCttsgd ~/SymbolicRegression.jl
/tmp/tmp.wvryCttsgd/SymbolicRegression.jl /tmp/tmp.wvryCttsgd ~/SymbolicRegression.jl
  Activating new project at `/tmp/jl_pivnKl`
   Resolving package versions...
    Updating `/tmp/jl_pivnKl/Project.toml`
  [98e50ef6] + JuliaFormatter v1.0.26
  [9b87118b] + PackageCompiler v2.1.5
    Updating `/tmp/jl_pivnKl/Manifest.toml`
...
[ Info: PackageCompiler: Executing /tmp/tmp.wvryCttsgd/SymbolicRegression.jl/precompile_file.jl => /tmp/jl_packagecompiler_YeNbB9/jl_HOzZT7
[ Info: PackageCompiler: Done
 [02m:42s] PackageCompiler: compiling incremental system image

this creates $HOME/julia_formatter.so (default, or just change that env variable), which is a system image for fast startup of JuliaFormatter. The command jformat will use this when running:

jformat

which automatically runs JuliaFormatter.format(".") with a compiled system image, and --threads=auto.

[xgdgsc]

I see unstable formatting times in vscode remote (sometimes instant, sometimes 10 seconds). Does compiled package also have unstable times? Would it be possible a prebuilt package (large size is fine) is provided and a vscode setting to set using this cli app is provided for those notice the slowness?

[complyue]

I noticed it too, each time a new VSCode window opened, the first time formatting .jl file will delay a few seconds, later formatting happens instantly. Maybe https://github.com/julia-vscode/LanguageServer.jl can provide an option to do the tricks here to ease the users?

[complyue]

May this trick work for JuliaFormatter.jl?

https://github.com/JuliaLang/JuliaSyntax.jl/blob/main/src/precompile.jl

# Just parse some file as a precompile workload
let filename = joinpath(@__DIR__, "literal_parsing.jl")
    text = read(filename, String)
    parseall(Expr, text)
    parseall(SyntaxNode, text)
    if _has_v1_6_hooks
        enable_in_core!()
        Meta.parse("1 + 2")
        enable_in_core!(false)
    end
end