Production Build with Vaadin 24 is very slow on a large project

[simasch]

Description of the bug

The production build with Vaadin 24 takes around 40 minutes compared to 5 minutes with Vaadin 23.

The project has around 400 Routes.

Expected behavior

It shouldn't take longer than with previous versions.

Minimal reproducible example

Unfortunately, I cannot share the project as it's not opensource.open source

Versions

Vaadin 24.1.3

[mcollovati]

First findings:

• During production build, class scanning is performed twice in Vaadin 24
• Byte code scanning performs the same both on V23 and V24, but V24 has an additional recursive step to collect reachable classes for every entry point. This seems to be slow, especially if the entry point can reach hundreds of classes.

[mcollovati]

A first improvement consists in pre-compiling the regular expression used to detect visitable classes. A simple test on a dummy project with 400 routes shows a significant build time reduction

8:17:37.387 [INFO] --- vaadin-maven-plugin:24.1.3:build-frontend (default) @ spring-skeleton ---
08:17:37.723 [INFO] Reflections took 292 ms to scan 127 urls, producing 4866 keys and 25879 values
08:17:37.743 [INFO] Scanning classes to find frontend configurations and dependencies...
08:17:46.729 [INFO] Visited 1377 classes. Took 8985 ms.

08:20:59.760 [INFO] --- flow-maven-plugin:24.2-SNAPSHOT:build-frontend (default) @ spring-skeleton ---
08:20:59.963 [INFO] Reflections took 196 ms to scan 127 urls, producing 4866 keys and 25880 values
08:20:59.969 [INFO] Scanning classes to find frontend configurations and dependencies...
08:21:02.292 [INFO] Visited 1370 classes. Took 2322 ms.

[mcollovati]

Looking at log files for a very slow build, it seems that also AbstractUpdateImports.process() may take a long time during particular circumstances

[gpsfl]

Is it possible to also manually exclude classes / package names? In our project it scans about 11000 classes most of which have nothing to do with vaadin. It takes 80s

[simasch]

That's a very good question @gpsfl

In my project it is [INFO] Visited 16572 classes. Took 94647 ms

[mshabarov]

Vaadin 24 and 23 use a slightly different algorithm to validate the license during production build. Vaadin 23 checks node_modules for installed components, whereas Vaadin 24 uses class scanning, because node_modules may not exist due to using pre-compiled bundle (new feature in V24). This leads to class scanning is performed twice in Vaadin 24. The possible solution might be either to cache the class scanning results and use it during license validation, or bring back old approach for license validation by scanning node_modules if Flow sees that a new build is triggered (i.e. node modules are installed anyway).

[mshabarov]

However, this, most likely, has much less impact than the second observation about byte code scanning.

[mshabarov]

This issue can be reproduced with a spring starter project having thousands clones of MainView, 2000 views in my test. Build time growth is visible starting from ~1000 views on my machine. The slowest place is AbstractUpdateImports::mergeJavascript and AbstractUpdateImports::process methods, which tend to growth significantly, when number of the views growths over 1000 views. These are main culprits I think. Will investigate further how it can be optimised.

UPD: no, I was wrong, just having 2000 clones of MainView doesn't seem to impact AbstractUpdateImports much. Just retested with 10000 view and here are the most of the contributors to the build time:

• FrontendDependencies - 2 x 75 secs
• UpdateImports - 5 secs
• Running Vite - 180 secs

Total time - 344 secs.

Here is the timeline in profiler:

(empty stripe in the middle is the Vite process running)

So this doesn't reproduce 34 minutes spent by AbstractUpdateImports between writing Vite config file and writing chunks JS into chunk files. Apparently there are other factors apart from Flow routes/view count that impact the update imports time.

[mshabarov]

Additional notes and observations:

• ChunkInfo misses two fields in hashCode and equals, which might be a problem for large projects, where update imports algorithm deals a lot with a maps where this class is used as a key.
• Flow generates chunk files even if they have identical content. This is a wrong behaviour and improved to have a chunks with only a unique contents.

Still don't fully understand why update imports takes 34 minutes. The number of classes is ~400, this is nothing in a sense of productivity even if it takes inner loops. For better understanding, I've added a bit more debug logging into AbstractUpateImports.

[mshabarov]

Optimisation for lazy chunks is done in https://github.com/vaadin/flow/releases/tag/24.1.9. @simasch could you please try to run your build with this version, if possible? You need to override a Flow version (not released in Vaadin platform yet):

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.vaadin</groupId>
                <artifactId>flow-bom</artifactId>
                <version>24.1.9</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>com.vaadin</groupId>
                <artifactId>vaadin-bom</artifactId>
                <version>${vaadin.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

and Flow maven plugin:

<plugin>
                        <groupId>com.vaadin</groupId>
                        <artifactId>flow-maven-plugin</artifactId>
                        <version>24.1.9</version>
                        <executions>
                            <execution>
                                <goals>
                                    <goal>build-frontend</goal>
                                </goals>
                                <phase>compile</phase>
                            </execution>
                        </executions>
                    </plugin>

Don't forget to remove optimizeBundle parameter, so that the default value will be used to reproduce the problematic configuration.

[vaadin-bot]

This ticket/PR has been released with Vaadin 24.2.0.alpha10 and is also targeting the upcoming stable 24.2.0 version.

[mshabarov]

Now available in Vaadin 24.1.8.

[simasch]

@mshabarov I tried 24.1.8. It's around 5 minutes faster but still way too slow.

[mshabarov]

To proceed with investigation, we need logs from prod build with latest V24.1 version and debug level enabled, as discussed with @simasch in the Discord channel.

[simasch]

This results from a local build on my MacBook Pro with an M2 Pro processor. It takes around 16 minutes.

https://www.dropbox.com/scl/fi/xbf8gotw4757ioum68jkx/build.log?rlkey=2ekqnqubsmwzvms9vciftsimx&dl=0

This looks like a huge step forward with Vaadin 24.1.10

[knoobie]

It's probably more related to your Mac vs Jenkins 😅

[simasch]

@knoobie Indeed I'm still waiting for the build to finish on Jenkins. So far 25 minutes elapsed :-(

[simasch]

Unfortunately, it was indeed my Mac. On the Jenkins server still the same

[mcollovati]

@simasch we just released Flow 24.1.13 which has an improvement for production build.

If you mind getting it a try, you can follow Mikhail's instruction from this comment https://github.com/vaadin/flow/issues/17234#issuecomment-1701078148

[simasch]

@mcollovati Unfortunately no change. The one with over 1 hour is with Flow 24.1.13

[mcollovati]

That is somehow surprising, compared with local results :thinking: Do you mind providing the logs of the build with debug info (even via DM if you prefer)?

[simasch]

https://www.dropbox.com/scl/fi/wf1io4gpaz0zwi3iz6lu9/build.log.zip?rlkey=fdniyas7uga8ci634khfswh5j&dl=0

[mcollovati]

@simasch just a confirmation: given the times in the logs, I suppose the build is from your local machine, not Jenkins. Is it so?

[simasch]

Yes, that's right.

[mcollovati]

But it looks like it is not running the latest Flow plugin

[INFO] --- vaadin:24.1.10:build-frontend (default) @ tosca-develop ---
[DEBUG] Loading mojo com.vaadin:vaadin-maven-plugin:24.1.10:build-frontend from plugin realm ClassRealm[plugin>com.vaadin:vaadin-maven-plugin:24.1.10, parent: jdk.internal.loader.ClassLoaders$AppClassLoader@251a69d7]
[DEBUG] Configuring mojo execution 'com.vaadin:vaadin-maven-plugin:24.1.10:build-frontend:default' with basic configurator -->

[mcollovati]

Or better, it is running both the plugins

[INFO] --- flow:24.1.13:build-frontend (default) @ tosca-develop ---
...
[DEBUG] Start updating imports file and chunk files.
[DEBUG] Found 437 modules, 437 scripts and 437 css.
[DEBUG] Start collecting scanned JS modules and scripts.
[DEBUG] JS modules and scripts collected in 681 ms.
[DEBUG] Start sorting imports to lazy and eager.
[DEBUG] Imports sorting took 62 ms.
[DEBUG] Start generating lazy loaded chunks.
[DEBUG] Lazy chunks generation took 3707 ms.
...
[DEBUG] Imports and chunks update took 4463 ms.
....
[INFO] --- vaadin:24.1.10:build-frontend (default) @ tosca-develop ---
...
[DEBUG] Start updating imports file and chunk files.
[DEBUG] Found 437 modules, 437 scripts and 437 css.
[DEBUG] Start collecting scanned JS modules and scripts.
[DEBUG] JS modules and scripts collected in 730 ms.
[DEBUG] Start sorting imports to lazy and eager.
[DEBUG] Imports sorting took 41 ms.
[DEBUG] Start generating lazy loaded chunks.
[DEBUG] Lazy chunks generation took 701675 ms.

[mcollovati]

@simasch it looks like that flow:24.1.13:build-frontend performs much better for lazy chunk generation.

[simasch]

After configuring the plugin correctly the build time is down to 6 minutes! Great job!

[mcollovati]

Great to hear! Thank you so much for helping to test the fix and providing the logs :bow:

[mshabarov]

If no objections, I'd propose to close this issue. The tested Flow version will be available in the next maintenance release for V24.1.

[simasch]

Thank you very much for the effort to fix this.