FISH-9222 bugfix: disable PayaraExecutorService events before server shuts down

[lprimak]

Description

Disable PayaraExecutorService events before server shuts down. Prevents exceptions on shutdown as described in linked issue fixes #6879

[lprimak]

@Pandrex247 Another one for you :)

[lprimak]

The fix is incomplete. Got another exception: Fixed now:

[2024-08-17T22:19:50.693-0500] [Payara 6.2024.9] [WARNING] [] [fish.payara.appserver.micro.services.PayaraInstanceImpl] [tid: _ThreadID=137 _ThreadName=payara-executor-service-scheduled-task] [timeMillis: 1723951190693] [levelValue: 900] [[
  Failed to get running Grizzly listener.
MultiException stack 1 of 1
java.lang.IllegalStateException: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 1 but it has a run level of 10.  The full descriptor is SystemDescriptor(
    implementation=com.sun.enterprise.v3.services.impl.GrizzlyService
    contracts={com.sun.enterprise.v3.services.impl.GrizzlyService,org.glassfish.api.container.RequestDispatcher,fish.payara.monitoring.collect.MonitoringDataSource}
    scope=org.glassfish.hk2.runlevel.RunLevel
    qualifiers={}
    descriptorType=CLASS
    descriptorVisibility=NORMAL
    metadata=runLevelValue={10},Bundle-SymbolicName={fish.payara.server.core.nucleus.kernel},Bundle-Version={6.19.0.SNAPSHOT}
    rank=50
    loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [fish.payara.server.core.nucleus.kernel [322]], State = [READY],1281793904)
    proxiable=null
    proxyForSameScope=null
    analysisName=null
    id=1160
    locatorId=0
    identityHashCode=449083851
    reified=true)
    at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.validate(AsyncRunLevelContext.java:422)
    at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:275)
    at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2103)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:769)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:732)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:702)
    at fish.payara.appserver.micro.services.PayaraInstanceImpl.initialiseInstanceDescriptor(PayaraInstanceImpl.java:407)
    at org.glassfish.grizzly.utils.Holder$3.evaluate(Holder.java:50)
    at org.glassfish.grizzly.utils.Holder$LazyHolder.get(Holder.java:85)
    at fish.payara.appserver.micro.services.PayaraInstanceImpl.lambda$event$0(PayaraInstanceImpl.java:253)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1570)
]]

[breakponchito]

@lprimak I tried to replicate but no success until now, as you said Do I need to shutdown the server with the monitor console enable for around 30 times? If you have clear steps to replicate please share to me

[lprimak]

Sorry, this happens only on CI server. Only rarely. I have no reliable reproducer for you unfortunately, but it's pretty clear what happens here though.

[Pandrex247]

@breakponchito

[2024-08-17T22:19:50.693-0500] [Payara 6.2024.9] [WARNING] [] [fish.payara.appserver.micro.services.PayaraInstanceImpl] [tid: _ThreadID=137 _ThreadName=payara-executor-service-scheduled-task] [timeMillis: 1723951190693] [levelValue: 900] [[
  Failed to get running Grizzly listener.
MultiException stack 1 of 1
java.lang.IllegalStateException: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 1 but it has a run level of 10.  The full descriptor is SystemDescriptor(

I've managed to see that error on Windows.

• Start a DAS
• Create and start an instance
• Create a deployment group and add the instance to the deployment group
• From the admin console - click stop deployment group
• Check the logs of the instance

[breakponchito]

@lprimak @Pandrex247 I already replicated the issue on windows following the steps you shared @Pandrex247 thank you: [2024-08-23T21:15:26.828-0600] [Payara 6.2024.9] [ADVERTENCIA] [] [fish.payara.appserver.micro.services.PayaraInstanceImpl] [tid: _ThreadID=166 _ThreadName=payara-executor-service-scheduled-task] [timeMillis: 1724469326828] [levelValue: 900] [[ Failed to get running Grizzly listener. MultiException stack 1 of 1 java.lang.IllegalStateException: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 1 but it has a run level of 10. The full descriptor is SystemDescriptor( implementation=com.sun.enterprise.v3.services.impl.GrizzlyService contracts={com.sun.enterprise.v3.services.impl.GrizzlyService,org.glassfish.api.container.RequestDispatcher,fish.payara.monitoring.collect.MonitoringDataSource} scope=org.glassfish.hk2.runlevel.RunLevel qualifiers={} descriptorType=CLASS descriptorVisibility=NORMAL

[breakponchito]

yes with this fix now the threads are finishing correctly and logs are clean, without any issue after shutdown deployment groups

[breakponchito]

@lprimak thank you for your contribution, this was already merged