Compiler very slow + intermediate files in source directory

[Bertie2011]

I believe this issue is closely related to #30. However, it doesn't seem to be fixed (or broken again) currently. The issue occurs when the exact same config as OP is used:

"Output": {
    "Preserve": true,
    "Directory":  "wwwroot/css"
  }

These are the issues I observed that might be related:

• Intermediate .min.css and .css files are created in the source directory.
• As long as those files exist the process is extremely slow (normally 2s, now 50s)
• I read somewhere that "Preserve": false has a significant performance impact, but since the files existing in the source directory slows the process down, it actually ends up being better to not preserve intermediate files.

If I had to guess, I think that the entire compilation process finishes first and that the output is copied to the output directory as the last step, but that this output path should be taken into account much earlier.

[stefanloerwald]

Hi @Bertie2011,

Just to verify; Do you want to have the files preserved, or do you want to only see the final output file?

BR Stefan

[Bertie2011]

I want the process to be as fast as possible. I currently have Preserve: false, since Preserve: true makes the process slower, even though it's said that it should be the other way around. (I will edit the original message to be more clear)

[stefanloerwald]

Thanks for clarifying, @Bertie2011! So far, I've never looked at performance characteristics, so I'll perform a quick test to see how it behaves on my machine.

[stefanloerwald]

Hi @Bertie2011,

on my machine I can't observe the same effect, but it might be down to a few different factors:

1. file size: how large are the files that are compiled? can you provide an example file of similar size?
2. disk access speed: how fast ist your device at reading/writing files of that size from/to disk?

Are you observing these extreme compilation times when files didn't change or when files did change?

[Bertie2011]

1. We're using Telerik UI for Blazor. But compiling any decently sized scss framework should be a good test.
2. It's an SSD, I don't know exact speeds, but it should be high enough. Also, I can see the relative difference (2s without intermediate files, 50s with), regardless of absolute speeds.
3. Both.

Maybe the full config helps, here it is:

{
  "Minifiers": {
    "GZip": false,
    "Enabled": true,
    "Css": {
      "CommentMode": "Important",
      "ColorNames": "Hex",
      "TermSemicolons": true,
      "OutputMode": "SingleLine",
      "IndentSize": 2
    },
    "Javascript": {
      "RenameLocals": true,
      "PreserveImportantComments": true,
      "EvalTreatment": "Ignore",
      "TermSemicolons": true,
      "OutputMode": "SingleLine",
      "IndentSize": 2
    }
  },
  "Autoprefix": {
    "Enabled": true,
    "ProcessingOptions": {
      "Browsers": [
        "last 4 versions"
      ],
      "Cascade": true,
      "Add": true,
      "Remove": true,
      "Supports": true,
      "Flexbox": "All",
      "Grid": "None",
      "IgnoreUnknownVersions": false,
      "Stats": "",
      "SourceMap": true,
      "InlineSourceMap": false,
      "SourceMapIncludeContents": false,
      "OmitSourceMapUrl": false
    }
  },
  "CompilerSettings": {
    "Sass": {
      "IndentType": "Space",
      "IndentWidth": 2,
      "OutputStyle": "Nested",
      "Precision": 5,
      "RelativeUrls": true,
      "LineFeed": "Lf",
      "SourceMap": false
    }
  },
  "Output": {
    "Preserve": false,
    "Directory":  "wwwroot/css"
  }
}

With this command: dotnet tool run webcompiler -- Styles/styles.scss -c webcompilerconfiguration.json

[stefanloerwald]

Thanks for providing the config file. However, I cannot reproduce the problem at all, even with a larger file. I guess I would need the exact file you're using to investigate further. Can you provide that?

[Bertie2011]

The NPM package of "Telerik UI for Blazor" that we use, should contain .scss as well. Try the material package with the given config and command.

[stefanloerwald]

The package contains a very large structure of scss files which I cannot easily replicate on my machine. I do not have the time to invest in setting all this up, so I'm afraid that without a self-contained example file, I cannot investigate this issue any further.

[Bertie2011]

The following package contains everything you need to replicate the issue:

• Execute install.bat to install the webcompiler (it simply contains a dotnet tool restore command)
• Execute run.ps1 (a shell script) to delete existing .min.css and .css files, measure the time it takes for a fresh compilation and measure the time it takes for a compilation with existing .min.css and .css files.

You can open the scripts in a text editor if you want to inspect them first.

You should be able to observe:

• The first run takes a short time (for me it was about 3 seconds, see TotalSeconds in the timings output)
• The second run takes >50 seconds.
• There are leftover .min.css and .css files in the Styles directory.

Again, the goal is:

• Get the duration of the command down when preserving files.
• This might require moving the intermediate files to the output directory.

WebCompilerDemo.zip

[stefanloerwald]

Thanks for providing this zip file.

The issue here was an extreme dependency tree (150230 dependencies). To determine whether a file needs recompilation, all dependencies need to be considered. If any of them has a "last modified" timestamp which is more recent than the output file, a recompilation is necessary, otherwise isn't. In most cases, it is very fast to gather all dependencies and then determining "last modified" timestamps.

A large number of dependencies here point to the same files, so the compilation actually only depends on 1063 files, not 150k. I was able to prevent this duplication of dependency discovery, but the underlying perf characteristic will remain: to determine necessity of recompilation, dependencies must be evaluated, which involve reading files and matching against regex. This costs time and can exceed the time of simply recompiling.

An improved version is published now (v2.7.14). I hope it helps your specific use case to keep compilations quick, regardless of the preserve option.

[Bertie2011]

So to summarize: If you have seen an import once already, you're not going to analyze and open the file again, which basically kills a ton of recursive branches.

regardless of the preserve option

The preserve option is said to improve speeds, should Preserve: true be enough? (Need certain source maps or other settings to be enabled?) And just to confirm, it wasn't necessary to move the intermediate files to the output directory?

I'll give the new version a try soon.

[stefanloerwald]

So to summarize: If you have seen an import once already, you're not going to analyze and open the file again, which basically kills a ton of recursive branches.

Yes, that's the improvement in 37de7bc.

regardless of the preserve option

The preserve option is said to improve speeds, should Preserve: true be enough? (Need certain source maps or other settings to be enabled?)

It can, because in many cases, comparing a few file timestamps is quicker than compiling a scss file. But of course things can be different. In this particular case, I now get a consistent faster second compilation with preserve: true.

And just to confirm, it wasn't necessary to move the intermediate files to the output directory?

The intermediate files do not belong in the output directory, as they are not considered to be output files.

[Bertie2011]

Oh my god, it went from 3s (fresh) and 50s (preserved) to 3s (fresh) and 0.8s (preserved). 🎉

Thank you for the quick responses and brilliant resolution!

[stefanloerwald]

Thanks for the feedback! I'm glad it improved performance for you.