Workspace Agent is slow to boot on macOs

[benoitf]

Running the workspace agent is quite slow, especially when I/O is not so fast. Which is the case on my environment : macOs / docker

Reproduction Steps: Start Eclipse Che workspace, and look at the time used by the workspace agent to boot.

Running profiler shows that it can run faster.

• Some libraries of Tomcat could be removed
• All jars (from WEB-INF/lib) included in the war file are scanned
• All Guice Singletons are initialized at startup but some of them have very costly constructor time. We could use lazy initialization.

PRs:

• [x] Get rid of JSP in assemblies so we can remove JSP stuff from Tomcat https://github.com/eclipse/che/pull/6174
• [x] Lazy init of maven local repository plugin maven server https://github.com/eclipse/che/pull/6178
• [x] Lazy init of tika instance in project service https://github.com/eclipse/che/pull/6179
• [x] workspace agent : avoid scan of WEB-INF/lib jars for classes #6186

[garagatyi]

Can you elaborate on ways to make it faster because it is not clear from the description to me

[benoitf]

@garagatyi I'm updating the issue. Some PR will be linked

[skabashnyuk]

Can we see numbers of perfomance report before we start applaing some changes?

[benoitf]

@skabashnyuk On my computer, on docker or minishift the start time of the agent is reduced by 5 seconds (15 seconds before to less than 10 seconds after)

[skabashnyuk]

Perfect. What is the timings for container start, individual components? What are the results on different OS OSX vs Linux?

[benoitf]

@skabashnyuk I suppose your comment is a little bit odd. In current design, code is eating memory, cpu and I/O and what my PR are doing is that it reduces memory, cpu and I/O without changing the logic. So there is no macOS vs Linux vs Windows. Also reducing I/O is good when you've several of workspaces starting on the same physical machine. (and memory is good as well)

[skabashnyuk]

@benoitf I just want to understand what is the price and what is the profit.

[benoitf]

@skabashnyuk I see only profit ?

[skabashnyuk]

I think that topic you've touch about performance is very important and at the same time it very sensitive to environment and opinions of persons who are doing measurements. In this case, to avoid any speculation I'm asking to base this work on some scientifical data, profiler data, logs report. It can help to better understand the way how to solve existed problems and do not make problems in future.

Closer to business:

• Get rid of JSP in assemblies so we can remove JSP stuff from Tomcat #6174
  • Lazy init of maven local repository plugin maven server #6178
  • Lazy init of tika instance in project service #6179

I think pure profit, except some subjective codestyle questions. They are quite easy money, interesting to see how much time from your 5 sec it takes.

• workspace agent : avoid scan of WEB-INF/lib jars for classes #6186

This is not so easy because it changes some fundamental approaches that we use for a while. Would be nice to see how much time we won in this time, to understand if it's worth of time discussing circumstances.

[benoitf]

@skabashnyuk work has been done on profiler data. It's not speculation. Any person running profiler will see that. How do you think I was able to detect some singletons taking up time to start ???

Also I start to think you never read any document on how to improve applications startup time on Tomcat

[skabashnyuk]

@skabashnyuk work has been done on profiler data. It's not speculation. Any person running profiler will see that. How do you think I was able to detect some singletons taking up time to start

ok. Can you share this data? About what level of numbers we are talking about, nano seconds, milliseconds, ten's of seconds.

Also I start to think you never read any document on how to improve applications startup time on Tomcat

It was rude and I do not think it is related to the topic. Every improvement has its price and in case of https://github.com/eclipse/che/pull/6186 I doubt that it's worth it.

[benoitf]

@skabashnyuk let say the opposite: why do you think app server have options to avoid the scan of classes if it was nano seconds ?

[benoitf]

Also guess what is the first point of https://wiki.apache.org/tomcat/HowTo/FasterStartUp

It is Jar Scanning

Among the scans the annotation scanning is the slowest. That is because each class file (except ones in ignored JARs) has to be read and parsed looking for annotations in it.

[skabashnyuk]

@benoitf before doing some conclusions, I expect some sort of performance report like this https://github.com/codenvy/codenvy/issues/1466#issuecomment-270922181 on which base we can elaborate. And yes I do know what Jar Scanning is. No need to convince me what it takes time.

[slemeur]

@skabashnyuk : Why don't we have automated performance tests and profiling suites like this https://github.com/codenvy/codenvy/issues/1466#issuecomment-270922181 ?

We should track and monitor that in an automated fashion. As we don't want to introduce functional regression, and we have functional tests for that, we don't want to introduce performances regressions either.

[skabashnyuk]

@slemeur good question. I don't know. May be because it's not what easy to automate.

[benoitf]

Here is a screencast https://gifyu.com/image/MSKH

Environment: macOS 16.7.0 / docker

docker:

$ docker version
Client:
 Version:      17.07.0-ce
 API version:  1.31
 Go version:   go1.8.3
 Git commit:   8784753
 Built:        Tue Aug 29 17:41:08 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.07.0-ce
 API version:  1.31 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   8784753
 Built:        Tue Aug 29 17:46:50 2017
 OS/Arch:      linux/amd64
 Experimental: false

First, I start che 5.17.0 then my "optimized che nightly" including the 4 PRs of this issue then again 5.17.0

result are : around 10s for 5.17.0 around 6s for optimized che

[garagatyi]

@benoitf can you clarify whether you got these results by applying all 4 improvements? And what was the target OS?

[benoitf]

@garagatyi I've updated my previous comment adding macOs/docker version and adding the fact that I had applied the 4 improvements

[garagatyi]

thanks

[benoitf]

@garagatyi if it's not clear enough, let me know I can add details

[garagatyi]

It's clear for me. From what I experienced on Mac (not with the best disk Apple provides), it is usually much slower than Linux in disk operations (because of Docker4Mac virtualization stuff). So it would be interesting to see results on Linux and especially on openshift instance (not virtualized). And I'm not forcing you to do those tests, just saying my thoughts out loud. BTW in case of development on Mac it was very slow because of heavy usage of mounted binaries. Do you know whether it works in the same way now?

[benoitf]

also I have not yet opened issues but during the profiling I've found other objects that takes time to initialize

org.eclipse.jdt.internal.ui.JavaPlugin.start
     org.eclipse.jdt.internal.core.JavaCorePreferenceInitializer.initializeDefaultPreferences() 

org.eclipse.che.api.languageserver.service.TextDocumentService.configureMethods() 

SimpleGeneratorStrategy : init of vfs

org.eclipse.che.api.project.server.WorkspaceHolder$$FastClassByGuice$$d17e5c7e.newInstance(int, Object[]) 

ResourcesPlugin:
  new workspace
      new WorkManager

[benoitf]

@garagatyi

• I assume that the gain on Linux are if you start many workspaces at the same time on the same docker node. (as there will be a lot of I/O at the same time)
• only /projects is mounted so for the start of the agent it's not taken into account.
• if you're speaking of the -v :/repo with mounting the whole che assembly, I'm not using that as it's too slow (almost unusable) due to default volume mount. But we could optimize this by adding extra flags on the mounted volume. https://docs.docker.com/docker-for-mac/osxfs-caching/