Closed efdknittlfrank closed 6 years ago
This sounds quite a lot like https://github.com/spring-projects/spring-boot/issues/11261. A couple of things to try:
org.springframework.core.io.support.PathMatchingResourcePatternResolver
and share the output,Not sure if the full debug log output would help, but here's the top sources of log messages level DEBUG and up:
1290 o.s.c.a.ClassPathScanningCandidateComponentProvider
1333 o.s.c.a.ClassPathBeanDefinitionScanner
...
3002 o.s.w.s.m.m.a.RequestMappingHandlerMapping
3844 o.s.o.j.JpaTransactionManager
4304 o.s.b.f.a.AutowiredAnnotationBeanPostProcessor
4415 o.s.c.a.AnnotationUtils
5240 o.h.c.a.SimpleValueBinder
5770 o.h.c.a.PropertyBinder
5796 o.h.c.Ejb3Column
6147 o.h.e.i.Collections
7894 o.h.e.i.TwoPhaseLoad
8847 o.s.d.r.c.RedisConnectionUtils
9623 o.s.b.f.a.InjectionMetadata
13299 o.s.a.f.CglibAopProxy
21790 o.h.l.p.b.i.s.QuerySpacesImpl
50837 o.s.b.f.s.DefaultListableBeanFactory
51659 o.h.p.w.s.MetamodelGraphWalker
59422 o.s.c.i.s.PathMatchingResourcePatternResolver
I will try the component candidate index thingy as soon as I find time.
Thanks. Looks like plenty of resource pattern resolution and Hibernate "stuff" is going on. The time between pairs of log entries produced by PathMatchingResourcePatternResolver
that I noted in this comment on #11261 would be interesting too. Happy to look at the whole log if you can anonymise it fairly easily.
Here's a zipfile with only the PathMatchingResourcePatternResolver
messages from the log (~56k). Zipped, because plain text is 20 MB. I've shortened/anonymized most/all class names. Hope that helps
Thanks. It does look like resource resolution is slower than we'd typically expect. For example:
[2018-02-07 14:28:26.617] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Looking for matching resources in directory tree [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service]
[2018-02-07 14:28:26.627] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.630] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.631] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contact] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.632] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.650] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\validity] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.656] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.671] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contact] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.673] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.675] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\validity] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.679] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\exception] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.683] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.683] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\communication] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.684] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\communication\tab] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.685] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contact] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.686] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contactmapping] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.687] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Searching directory [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contactmapping\validity] for files matching pattern [C:/dev/project/modules/backend/service/out/production/classes/com/company/project/services/communication/service/**/*.class]
[2018-02-07 14:28:26.693] DEBUG [main] --- o.s.c.i.s.PathMatchingResourcePatternResolver: Resolved location pattern [classpath*:com/company/project/services/communication/service/**/*.class] to resources [file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Absice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Absice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Areice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Areice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Audice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Audice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Aution.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Comice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Comice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Comice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Comice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Comice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Comice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Comice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Comice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Comice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Conice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Conice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Conice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Conice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Conice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Conice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Conice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Conice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\Comter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\Comter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contact\Conter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contact\Conter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contact\Conter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contact\Conter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contact\Conter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contact\Conter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contact\Conter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contact\Conter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contact\Conter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contact\Conter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Abster.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Areter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Areext.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Areter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Objter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Objext.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Objter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Perter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Perext.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Perter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Perter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Perext.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Perter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\validity\Abster.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\validity\Areter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\validity\Objter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\validity\Perter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\validity\Perter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\validity\Vehter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Vehter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Vehext.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\contactmapping\Vehter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\Gatter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\Gatter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\comter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\comter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\Talter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\Talter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\Talter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\converter\Talter.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\devion.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Dgnion.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Dgnice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Gatice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Gatice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Gatice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\comion.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\comice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Audent.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Audche.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Aude$1.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Audice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Audrer.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Audmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Audage.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\AutDTO.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Autmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Bastor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Calice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Calate.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Calype.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Calent.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Calche.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Calute.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Calice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Comice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Commpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Commpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Commpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Comrer.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Commpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Come$1.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\ComKey.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Coment.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Comype.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Comche.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Comype.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\ComDTO.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Commpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Commpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Comder.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Commpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Commpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Comate.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Coml$1.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Comack.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Commpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\ComDTO.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Coms$1.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Comils.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Comice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contact\Absmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contact\Conmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contact\Conmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contact\Conmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contact\Conmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contact\Conmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contact\Conmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contact\Conmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contact\Conmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\Absmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\Arempl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\Objmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\Permpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\Pertil.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\Permpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\validity\Absmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\validity\Arempl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\validity\Objmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\validity\Permpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\validity\Permpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\validity\Vehmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\contactmapping\Vehmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\deve$1.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\devnfo.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\device.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Devtor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Dgnnfo.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Dgnnfo.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Dgnmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Dumice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Endice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Ente$1.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Entche.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\exception\Comion.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\exception\Comion.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\exception\NoAion.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\exception\NoCion.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\exception\Useion.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\exception\wwsion.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\exception\wwsion.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Gatmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Gatmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Gatmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Gloice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\comDTO.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\commpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Mesged.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Mesged.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Mesged.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Mesged.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Mesged.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Notent.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Notice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Plaype.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Plampl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Pttice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\commpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Repmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Tall$1.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Talmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Talype.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\TalDTO.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Talmpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Teae$1.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Teaice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Unimpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Visice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\impl\Vismpl.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Mesged.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Objice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Objice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Perice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Perice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Perice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Perice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\comion.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\comice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Repice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Talice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Talice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\TalTea.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Uniice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\communication\tab\Comtor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contact\Contor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contact\Contor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contact\Contor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contact\Contor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contact\Contor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contact\Contor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contact\Contor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contact\Contor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contactmapping\Aretor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contactmapping\Contor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contactmapping\Objtor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contactmapping\Pertor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contactmapping\Pertor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contactmapping\validity\Aretor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contactmapping\validity\Objtor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contactmapping\validity\Pertor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contactmapping\validity\Pertor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contactmapping\validity\Vehtor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\validator\contactmapping\Vehtor.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Vehice.class], file [C:\project\modules\backend\service\out\production\classes\com\company\project\services\communication\service\Vehice.class]]
76 ms to search 16 directories. How many files are in each of those directories?
The services/communication/service
directory contains 221 class files. This is on a windows machine, maybe antivirus is negatively impacting performance?
AV doesn't improve performance - turn it off and retry...
The services/communication/service directory contains 221 class files
Is that 221 files just in that top-level directory, or in the entire tree? The entire tree is scanned so the total numbers of files across all 16 directories is of interest.
maybe antivirus is negatively impacting performance?
It's been known to have a severe impact as it can scan every file before it's accessed. As @JanStureNielsen suggests, please turn it off and retry.
The services/communication/service directory contains 221 class files
Is that 221 files just in that top-level directory, or in the entire tree? The entire tree is scanned so the total numbers of files across all 16 directories is of interest.
Total number of files (find project\services\communication\service -name '*.class' | wc -l
).
Here's the distribution across directories:
$ for d in . converter converter/{contact,contactmapping} converter/contactmapping/validity impl impl/{contact,contactmapping,exception} impl/contactmapping/validity validator validator/{communication,contact,contactmapping} validator/communication/tab validator/contactmapping validator/contactmapping/validity; do
echo -n $d:\ ;
find project/services/communication/service/$d -maxdepth 1 -name '*.class' | wc -l;
done
.: 48
converter: 10
converter/contact: 10
converter/contactmapping: 16
converter/contactmapping/validity: 6
impl: 82
impl/contact: 9
impl/contactmapping: 7
impl/exception: 7
impl/contactmapping/validity: 6
validator: 0
validator/communication: 0
validator/contact: 8
validator/contactmapping: 6
validator/communication/tab: 1
validator/contactmapping: 6
validator/contactmapping/validity: 5
[…], please turn it off and retry.
Having AV turned off in a corporate environment is not as easy as it sounds. I'll see what I can do.
Thanks. Rather than turning AV off completely, perhaps you could whitelist the directories with your app's classes in them?
Rather than turning AV off completely, perhaps you could whitelist the directories with your app's classes in them?
That's already done for this project's directory. But AV behavior can be quite erratic sometimes, so to rule it out as source of the slowdown, it has to be disabled completely. I've seen really strange behavior in the past 🙂
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.
We see exactly this behaviour on Spring Boots app launching on EC2, I can guarantee no antivirus is running there.
From profiling, we see most of the time is spent in class loading and loading files.
It takes 2 minutes to boot up the app and the moment the Classes loaded metrics top off, CPU drops to zero and the apps starts up
The profiling above is on a otherwise idle Macbook Pro with no antivirus/malware running. Loaded from a Gradle generate fat jar with java -jar ...
and no additional options apart from the profiler
It looks like this is/was caused in part by the @EnableJpaRepositories
annotation. We tried to be as specific as possible and had a list of basePackages
:
basePackages = {
"com.company.product.a",
"com.company.product.b",
"com.company.product.c",
"com.company.product.d",
"com.company.product.e",
"…" // 35 base packages in total
}
Reducing this to only a single root base package ("com.company.product"
) and adding two excludeFilters
of type REGEX
improved startup times a lot (from 20 minutes down to 4 minutes):
basePackages = "com.company.product",
excludeFilters = {
@ComponentScan.Filter(type = REGEX, pattern = "^com.company.product.x.template",
@ComponentScan.Filter(type = REGEX, pattern = "Dummy"
}
This is somewhat unintuitive, I would think that scanning fewer directories (only the leaf directories of the tree) would be faster than scanning the full directory tree. But it seems the spring component scanner/loader thinks differently.
So loading the asm metadata of the class is what makes it slow then. Are you using Spring Boot 2.0.0.RELEASE
now?
Also, have you tried using the context indexer on this project? It won't fix the actual issue but if classpath scanning is slow for your particular project, it should have quite an impact.
I'm not sure what asm metadata is exactly, but it looks like spring is unpacking the fat jar for each separate basePackages
entry, so with 35 base packages, the jar is extracted 35 times (which can take some time with a 100+ MB jar file). Using only a single entry, it is extracted only once.
Thanks that helps.
have you tried using the context indexer on this project?
If you haven't, could you please give it a try?
No, I have not, and cannot do so at the moment. I will comment once I have tried the context indexer
That's a very interesting data point, @efdknittlfrank. Thank you.
The way that package scanning behaves depends on whether or not the classes that are to be found are in a jar file or in a directory structure on the file system. In the case of the latter, the more base packages you specify, the quicker I would expect scanning to be as scanning can quickly focus on only those files that are in the back packages. In the case of the former, more base packages can, as you have seen, slow things down as scanning cannot focus any more closely than on a jar file.
In the jar file case, for each base package that's specified, the first step is to find all of the jars that contain that package. Each "jar" can be BOOT-INF/classes
inside your fat jar or a jar that's nested inside BOOT-INF/lib
in your fat jar. The entries for each jar that contains the package are then iterated over to find all of the classes that are in the base package or one of its sub-packages.
If you have 35 base packages that are all in the same jar, that jar's entries will be scanned 35 times. If you specify a single base package that jar's entries will be scanned once.
I'll look forward to hearing about the difference that the context indexer makes to your application.
Speaking to @efdknittlfrank at JAX conference, I've created a ticket in Spring Data to investigate performance of our fragment resolution, which we suspect to significantly add onto the startup time.
Thank you, @olivergierke. You may also be interested in SPR-16649.
@efdknittlfrank The inefficient behaviour you described above will be addressed by SPR-16649. In the meantime, I think the context indexer should alleviate the problem. Did you get a change to try it?
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.
We've significantly improved the repository bootstrap in Lovelace RC2. I'd love to see what numbers you get when you move to that version, @efdknittlfrank. Would you mind giving it a try?
Great! I hope I can find some time to upgrade to 2.1 temporarily and give it a spin.
A spring boot application takes very long to start (usually between 5 and 30 minutes, sometimes even 60 minutes).
Starting the application from within IntelliJ IDEA on Windows usually gives startup times of 3 minutes; when started from a fat jar on GNU/Linux the "normal" start time is around 20 minutes. This slows down development and makes continuous deployment very cumbersome.
The application runs Spring Boot v2.0.0.M2 and Spring v5.0.0.RC2. It uses additional dependencies from spring boot, including mvc, redis, elasticsearch, rabbitmq, security, jpa, data, etc. (so yes, quite a lot)
I will attach a log file of the local server startup (from intellij) with level DEBUG. There seem to be long pauses between some log statements, sometimes almost one minute long. Example:
I'm grateful for any pointers you can provide on how to further investigate the issue and narrow down the cause. I consider application startup of 5 minutes on the server still too slow, but could live with it. But 20 minutes simply does not work. For local development I would expect a spring boot application to be ready in less than a minute.
Spring boot AutoConfiguration is enabled, as is component scanning of the classpath (which is too large for IntelliJ to pass directly to the application – maybe that's a problem?).
Here's the log file (anonymized to protect the guilty). If there is information lacking, let me know and I'll happily provide them: