search

dCache / dcache

dCache - a system for storing and retrieving huge amounts of data, distributed among a large number of heterogenous server nodes, under a single virtual filesystem tree with a variety of standard access methods
https://dcache.org
290 stars 136 forks source link

dCache 8.2.4 cleaner errors #6879

Closed cfgamboa closed 1 year ago

cfgamboa commented 1 year ago

Hello

Following error is reported at the cleaner log This instance is recently upgraded to 8.2.4

Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: java.lang.NullPointerException: null
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at org.dcache.chimera.namespace.PoolInformationBase.remove(PoolInformationBase.java:78)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at org.dcache.chimera.namespace.PoolInformationBase.messageArrived(PoolInformationBase.java:95)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at org.dcache.cells.CellMessageDispatcher$ShortReceiver.deliver(CellMessageDispatcher.java:272)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at org.dcache.cells.CellMessageDispatcher.call(CellMessageDispatcher.java:188)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at org.dcache.cells.AbstractCell.messageArrived(AbstractCell.java:302)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at dmg.cells.nucleus.CellAdapter.messageArrived(CellAdapter.java:856)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at dmg.cells.nucleus.CellNucleus$DeliverMessageTask.run(CellNucleus.java:1274)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at org.dcache.util.BoundedExecutor$Worker.run(BoundedExecutor.java:247)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at dmg.cells.nucleus.CellNucleus.lambda$wrapLoggingContext$2(CellNucleus.java:727)
Nov 23 11:38:09 dcqos001 dcache@cleanerDomain: at java.base/java.lang.Thread.run(Thread.java:829)

Could you please advise

All the best, Carlos

lemora commented 1 year ago

Hi Carlos. Thanks for reporting, I will look into it. Has this error appeared more than once in the cleaner log, and has it affected the cleaner behaviour at all?

cfgamboa commented 1 year ago

@lemora thank you and sorry I missed your question. Thought is already marked as bug. I will basically confirm that is a appeared more than once on the logs.

lemora commented 1 year ago

Sure, every NPE is a bug, of course. But I still need to understand how and what is happening. How often do you see this message per day, and does that correlate with other messages -- do files not get cleaned as a consequence?

cfgamboa commented 1 year ago

Hello

See below:

[dcqos001] (cleaner@cleanerDomain) admin > show pinboard 10000
01 Dec 2022 14:45:21 [cleaner-19] [dcqos002_20 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:21 [cleaner-19] [dcqos002_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:21 [cleaner-19] [dcqos002_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:21 [cleaner-19] [dcqos002_23 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:21 [cleaner-19] [dcqos002_17 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_24 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_15 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_21 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_19 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_18 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_22 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_16 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:26 [cleaner-19] [dcqos005_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:29 [cleaner-19] [dcqos005_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_3 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_1 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_4 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:50 [cleaner-19] [dcqos001_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:50 [cleaner-19] [dcqos001_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:50 [cleaner-19] [dcqos001_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:50 [cleaner-19] [dcqos001_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:50 [cleaner-19] [dcqos001_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:51 [cleaner-19] [dcqos002_20 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:51 [cleaner-19] [dcqos002_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:51 [cleaner-19] [dcqos002_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:51 [cleaner-19] [dcqos002_23 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:51 [cleaner-19] [dcqos002_17 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_24 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_15 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_21 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_19 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_18 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_22 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_16 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:53 [pool-6-thread-31] [Disk-Cleaner] New run...
01 Dec 2022 14:45:56 [cleaner-19] [dcqos005_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:45:59 [cleaner-19] [dcqos005_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_3 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_1 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_4 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:20 [cleaner-19] [dcqos001_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:20 [cleaner-19] [dcqos001_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:20 [cleaner-19] [dcqos001_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:20 [cleaner-19] [dcqos001_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:20 [cleaner-19] [dcqos001_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:21 [cleaner-19] [dcqos002_20 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:21 [cleaner-19] [dcqos002_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:21 [cleaner-19] [dcqos002_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:21 [cleaner-19] [dcqos002_23 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:21 [cleaner-19] [dcqos002_17 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_24 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_15 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_21 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_19 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_18 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_22 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_16 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:26 [cleaner-19] [dcqos005_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:29 [cleaner-19] [dcqos005_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_3 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_1 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_4 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:50 [cleaner-19] [dcqos001_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:50 [cleaner-19] [dcqos001_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:50 [cleaner-19] [dcqos001_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:50 [cleaner-19] [dcqos001_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:50 [cleaner-19] [dcqos001_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:51 [cleaner-19] [dcqos002_20 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:51 [cleaner-19] [dcqos002_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:51 [cleaner-19] [dcqos002_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:51 [cleaner-19] [dcqos002_23 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:51 [cleaner-19] [dcqos002_17 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_24 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_15 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_21 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_19 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_18 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_22 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_16 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:56 [cleaner-19] [dcqos005_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:46:59 [cleaner-19] [dcqos005_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_3 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_1 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_4 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:20 [cleaner-19] [dcqos001_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:20 [cleaner-19] [dcqos001_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:20 [cleaner-19] [dcqos001_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:20 [cleaner-19] [dcqos001_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:20 [cleaner-19] [dcqos001_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:21 [cleaner-19] [dcqos002_20 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:21 [cleaner-19] [dcqos002_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:21 [cleaner-19] [dcqos002_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:21 [cleaner-19] [dcqos002_23 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:21 [cleaner-19] [dcqos002_17 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_24 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_15 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_21 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_19 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_18 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_22 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_16 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:26 [cleaner-19] [dcqos005_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:29 [cleaner-19] [dcqos005_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_3 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_1 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_4 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:50 [cleaner-19] [dcqos001_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:50 [cleaner-19] [dcqos001_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:50 [cleaner-19] [dcqos001_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:50 [cleaner-19] [dcqos001_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:50 [cleaner-19] [dcqos001_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:51 [cleaner-19] [dcqos002_20 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:51 [cleaner-19] [dcqos002_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:51 [cleaner-19] [dcqos002_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:51 [cleaner-19] [dcqos002_23 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:51 [cleaner-19] [dcqos002_17 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_24 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_15 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_21 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_19 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_18 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_22 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_16 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:53 [pool-6-thread-31] [Disk-Cleaner] New run...
01 Dec 2022 14:47:56 [cleaner-19] [dcqos005_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:47:59 [cleaner-19] [dcqos005_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null

01 Dec 2022 14:48:12 [pool-6-thread-98] [] Finished deleting from pool dcqos001_2
lemora commented 1 year ago

Thanks. As a side question -- do you run both the disk and hsm components of the cleaner here?

I have an idea what is going on. Could you please check if the listed pools for which the NPE occurs are not attached to an hsm backend or if they are flagged lfs=precious?

cfgamboa commented 1 year ago

Only DISK components not HSM

Carlos

On Dec 2, 2022, at 7:51 AM, Lea @.***> wrote:

Thanks. Do you run both the disk and hsm components of the cleaner here?

— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1335183590, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMOYIKD4B2Q32L2EIBGLWLHWF5ANCNFSM6AAAAAASJFGKB4. You are receiving this because you authored the thread.

lemora commented 1 year ago

Hey Carlos.

Okay, so I expect the pools are marked as such. Even if you only run the disk cleaner this is relevant here. The fix will be included in the next bugfix release series. Could you then please confirm if it works for you?

Thanks. Lea

cfgamboa commented 1 year ago

Hi Lea,

The pools is configured as 

[${host.name}thirteenDomain]
[${host.name}thirteenDomain/pool]
pool.name=${host.name}_13
pool.path=/data11/dcache_pool_13/pool
pool.wait-for-files=${pool.path}/data
pool.lfs=precious

Carlos

cfgamboa commented 1 year ago

Hi Lea,

This might be related to https://github.com/dCache/dcache/issues/6915 All the best, Carlos

lemora commented 1 year ago

Hi Carlos,

thanks for the configuration. I don't think the linked issue is related; at least the problem you saw here is explanable by a recent change, where pools marked as pool.lfs=precious no longer announce to the cleaner that they are connected to an hsm and thus available to the hsm component. This was not correctly handled on the cleaner side but is fixed with the posted patch. Since disk and hsm cleaner are still merged in dCache 8.2, this even affected cleaner instances in which the hsm component was turned off like in your case.

Regards Lea

cfgamboa commented 1 year ago

Hey Lea,

I think I was able to reproduce this 

07 Dec 2022 16:07:09 [cleaner-2] [dcqos001_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-2java.lang.NullPointerException: null

If I disable the pool.lfs=precious and reload the dcqos001_12 pool the message is not longer show in the pinboard.

My understanding is that for DISK only pools this flag might not be necessary pool.lfs=precious @kofemann or you could please confirm?

All the best, Carlos

lemora commented 1 year ago

If I disable the pool.lfs=precious and reload the dcqos001_12 pool the message is not longer show in the pinboard.

Okay, good to know. Before, the cleaner did not care either way: pool.lfs=precious or not. You can remove the flag if files on these pools are pinned or otherwise ensured to not be cache-removed, but to be clear here: the NullPointerException is in either case a bug on our side that needed fixing.

You will not need to deploy the fix on the pools, just the cleaner instance in order for everything to hopefully be NPE-free again. The newest bugfix versions containing the fix were released today.

cfgamboa commented 1 year ago

OK I am using 8.2.7 on these nodes/pools. The results/tests reported today were done using that release.

lemora commented 1 year ago

Hey, oh, so do I understand correctly that you still see the NPEs in 8.2.7 when there are pools marked as pool.lfs=precious? It is interesting that the PoolManager pinboard messages disappear when pools are not lfs=precious anymore... This needs some more digging.

cfgamboa commented 1 year ago

Hey

Your understanding is correct. For example I have enabled the pool.lfs=precious Pool conf

[${host.name}fourteenDomain]
[${host.name}fourteenDomain/pool]
pool.name=${host.name}_14
pool.path=/data12/dcache_pool_14/pool
pool.wait-for-files=${pool.path}/data
pool.lfs=precious

Then after rebooting the pool The cleaner pinboard

08 Dec 2022 10:20:45 [cleaner-2] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2java.lang.NullPointerException: null

08 Dec 2022 10:20:45 [cleaner-2] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2java.lang.NullPointerException: null

08 Dec 2022 10:20:45 [cleaner-2] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2java.lang.NullPointerException: null

08 Dec 2022 10:20:45 [cleaner-2] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2java.lang.NullPointerException: null

And the PoolManager pinboard 

08 Dec 2022 10:20:34 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Pool dcqos001_14 changed from mode enabled  to disabled(fetch,store,stage,p2p-client,p2p-server,dead).
08 Dec 2022 10:20:34 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Restore Manager : got 'poolRestarted' for dcqos001_14
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Pool dcqos001_14 changed from mode disabled(fetch,store,stage,p2p-client,p2p-server,dead)  to disabled(fetch,store,stage,p2p-client,p2p-server).
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Restore Manager : got 'poolRestarted' for dcqos001_14
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Restore Manager : got 'poolRestarted' for dcqos001_14
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Pool dcqos001_14 changed from mode disabled(fetch,store,stage,p2p-client,p2p-server)  to disabled(store,stage,p2p-client,loading).
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Restore Manager : got 'poolRestarted' for dcqos001_14
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Pool dcqos001_14 changed from mode disabled(store,stage,p2p-client,loading)  to enabled.
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Restore Manager : got 'poolRestarted' for dcqos001_14
08 Dec 2022 10:21:15 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Restore Manager : got 'poolRestarted' for dcqos001_14
lemora commented 1 year ago

OK I am using 8.2.7 on these nodes/pools. The results/tests reported today were done using that release.

Just to clarify -- did you also upgrade the cleaner to 8.2.7, not just the pools (the latter was not needed in any case)?

Otherwise, could you please try again to find a full stack trace for the NPE after the upgrade?

lemora commented 1 year ago

Okay, I think I understand why the PoolManager pinboard is flooded with these 'poolRestarted' messages when there are pool.lfs=precious-pools -- it again relates to the original "fix" which lets the cleaner know not to send hsm delete requests to pools marked as lfs-precious. I'll post another fix that should address the issue -- including GitHub issue #6915 , if that is okay with you, @alrossi, since you already started looking into it. But these two indeed seem related.

But first, Carlos, I need confirmation from you that you indeed upgraded the cleaner as well, since that could mean that I don't understand the problem as well as I think.

cfgamboa commented 1 year ago

Hi Lea, Indeed the cleaner is running on 8.2.7. All the best, Carlos

lemora commented 1 year ago

Hi Carlos. Has the full stack trace changed at all? Regards, Lea

cfgamboa commented 1 year ago

Hola Lea,

Yes, the stack trace changed, before showed more messages an example is included at the beginning of the ticket. This is the current one.

Dec  8 15:34:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:34:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2
Dec  8 15:34:45 dcqos001 dcache@cleanerDomain: java.lang.NullPointerException: null

Carlos

lemora commented 1 year ago

Hi Carlos.

Our logging system suppresses the whole stack trace if it is logged too often, then one just sees what you sent in your last message, for example: java.lang.NullPointerException: null Could you try to find the first occurrence of the NPE in order to get the full stack trace? The truncated one unfortunately does not help much.

Thanks. Lea

cfgamboa commented 1 year ago

Hi Lea, I have enabled the cleaner in debug all.

Dec  8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] Executing SQL query [SELECT DISTINCT ilocation FROM t_locationinfo_trash WHERE itype=1]
Dec  8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] Fetching JDBC Connection from DataSource
Dec  8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [] Executing SQL query [SELECT ipnfsid FROM t_locationinfo_trash t1 WHERE itype=2 AND NOT EXISTS (SELECT 1 FROM t_locationinfo_trash t2 WHERE t2.ipnfsid=t1.ipnfsid AND t2.itype <> 2)]
Dec  8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [] Fetching JDBC Connection from DataSource
Dec  8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [] Finished deleting from pool dcqos001_2
Dec  8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] sendMessage : <7379323899589971090:-5208671900136200888> send to [>admin@adminDoorDomain]
Dec  8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] sendMessage : next hop at 16: admin@adminDoorDomain
Dec  8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] sendMessage : using route : CellRoute{cell=*, domain=*, gateway=c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain, type=Default, zone=Undefined}
Dec  8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] sendMessage : next hop at 15: c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain
Dec  8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] addToEventQueue : message arrived : <CM: S=[>cleaner@cleanerDomain:*@cleanerDomain];D=[>admin@adminDoorDomain];C=Stream;O=<7379323899589971090:-5208671900136200888>;LO=<321142552260658636:-7201269758871426662>;SID=caE:2o;TTL=300000>
Dec  8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [] HikariPool-1 - Closing connection org.postgresql.jdbc.PgConnection@7a8e717a: (connection has passed maxLifetime)
Dec  8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] New run...
Dec  8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] INFO: Refresh Interval : 120 SECONDS
Dec  8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] INFO: Number of files processed at once: 10000
Dec  8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] Executing SQL query [SELECT DISTINCT ilocation FROM t_locationinfo_trash WHERE itype=1]
Dec  8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] Fetching JDBC Connection from DataSource
Dec  8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] List of Pools from the trash-table : [dcqos001_2]
Dec  8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] 1 pools are currently blacklisted.
Dec  8 15:50:15 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:15 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : <-7571265469618568800:-8456256691570891820> send to [>dcqos001_14@dcqos001fourteenDomain]
Dec  8 15:50:15 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:15 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : next hop at 16: dcqos001_14@dcqos001fourteenDomain
Dec  8 15:50:15 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:15 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : using route : CellRoute{cell=*, domain=*, gateway=c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain, type=Default, zone=Undefined}
Dec  8 15:50:15 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:15 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : next hop at 15: c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain
Dec  8 15:50:15 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:15 (cleaner) [dcqos001_14 PoolManagerPoolUp] addToEventQueue : message arrived : <CM: S=[>cleaner@cleanerDomain:*@cleanerDomain];D=[>dcqos001_14@dcqos001fourteenDomain];C=Stream;O=<-7571265469618568800:-8456256691570891820>;LO=<8341864328583135877:-7157650930398023863>>
Dec  8 15:50:15 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:15 (cleaner) [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2
Dec  8 15:50:15 dcqos001 dcache@cleanerDomain: java.lang.NullPointerException: null
Dec  8 15:50:18 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:18 (cleaner) [] HikariPool-1 - Before cleanup stats (total=2, active=0, idle=2, waiting=0)
Dec  8 15:50:18 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:18 (cleaner) [] HikariPool-1 - After cleanup  stats (total=1, active=0, idle=1, waiting=0)
Dec  8 15:50:18 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:18 (cleaner) [] HikariPool-1 - Closing connection org.postgresql.jdbc.PgConnection@2ca205d8: (connection has passed idleTimeout)
Dec  8 15:50:42 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:42 (cleaner) [Frontend-dcqos001] sendMessage : <-8298066119329755423:-8704052522908946887> send to [>Frontend-dcqos001@srr-dcqos001Domain]
Dec  8 15:50:42 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:42 (cleaner) [Frontend-dcqos001] sendMessage : next hop at 16: Frontend-dcqos001@srr-dcqos001Domain
Dec  8 15:50:42 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:42 (cleaner) [Frontend-dcqos001] sendMessage : using route : CellRoute{cell=*, domain=*, gateway=c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain, type=Default, zone=Undefined}
Dec  8 15:50:42 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:42 (cleaner) [Frontend-dcqos001] sendMessage : next hop at 15: c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain
Dec  8 15:50:42 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:42 (cleaner) [Frontend-dcqos001] addToEventQueue : message arrived : <CM: S=[>cleaner@cleanerDomain:*@cleanerDomain];D=[>Frontend-dcqos001@srr-dcqos001Domain];C=Stream;O=<-8298066119329755423:-8704052522908946887>;LO=<-1501833629673042479:-7302381553561486839>;TTL=60000>
Dec  8 15:50:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : <-271099534625585754:-6215812878134086060> send to [>dcqos001_14@dcqos001fourteenDomain]
Dec  8 15:50:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : next hop at 16: dcqos001_14@dcqos001fourteenDomain
Dec  8 15:50:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : using route : CellRoute{cell=*, domain=*, gateway=c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain, type=Default, zone=Undefined}
Dec  8 15:50:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : next hop at 15: c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain
Dec  8 15:50:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] addToEventQueue : message arrived : <CM: S=[>cleaner@cleanerDomain:*@cleanerDomain];D=[>dcqos001_14@dcqos001fourteenDomain];C=Stream;O=<-271099534625585754:-6215812878134086060>;LO=<9036826481595072635:-7979064345865280826>>
Dec  8 15:50:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2
Dec  8 15:50:45 dcqos001 dcache@cleanerDomain: java.lang.NullPointerException: null
Dec  8 15:50:48 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:48 (cleaner) [] HikariPool-1 - Before cleanup stats (total=1, active=0, idle=1, waiting=0)
Dec  8 15:50:48 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:48 (cleaner) [] HikariPool-1 - After cleanup  stats (total=1, active=0, idle=1, waiting=0)
lemora commented 1 year ago

Debug doesn't help to procure the stack trace, unfortunately. One has to search for the first NPE occurrence after the upgrade and restart of the cleaner, I fear. The place is simliar, as expected, though. But anyway, I'll try to understand what is going on tomorrow. I expected the patch to fix this.

cfgamboa commented 1 year ago

Hello Lea,

OK. In any case, If you would like to discuss further or troubleshoot more we could schedule a zoom call for tomorrow, just let me know.

All the best, Carlos

On Dec 8, 2022, at 3:57 PM, Lea @.***> wrote:

Debug doesn't help to procure the stack trace, unfortunately. One has to search for the first NPE occurrence after the upgrade and restart of the cleaner, I fear. But anyway, I'll try to understand what is going on tomorrow. I expected the patch to fix this.

— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1343343202, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO6Y6CPES2C4K3OGUXTWMJDVPANCNFSM6AAAAAASJFGKB4. You are receiving this because you authored the thread.

lemora commented 1 year ago

Hey.

If you find the time to look, I would really like to see that stack trace. :) Other than that I don't think a zoom meeting would benefit us here, though.

Cheers Lea

cfgamboa commented 1 year ago

Hi

Where I should look for the trace files? Carlos

On Dec 8, 2022, at 4:34 PM, Lea @.***> wrote:

Hey.

If you find the time to look, I would really like to see that stack trace. :) Other than that I don't think a zoom meeting would benefit us here, though.

Cheers Lea

— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1343394086, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO5H2IQWBHRQ2PEUBQDWMJH4ZANCNFSM6AAAAAASJFGKB4. You are receiving this because you authored the thread.

lemora commented 1 year ago

You will find it in the regular logs, it should look like the one you initially posted here. The first NullPointerException after you restarted the cleaner for upgrading it to 8.2.7 yesterday should probably be the full one.

kofemann commented 1 year ago

Hi @cfgamboa ,

You can use java flight recorder to collect hidden stacktraces as described in

https://github.com/dCache/dcache/blob/master/docs/TheBook/src/main/markdown/cookbook-debugging.md

If you provide the resulting jfr file, then we can see the stacktrace, even if it's not in the logs.

cfgamboa commented 1 year ago

Hi I will give a try tomorrow, thank you for the reference. Carlos

On Dec 8, 2022, at 5:17 PM, Tiramisu Mokka @.***> wrote:

Hi @cfgamboa https://github.com/cfgamboa ,

You can use java flight recorder to collect hidden stacktraces as described in

https://github.com/dCache/dcache/blob/master/docs/TheBook/src/main/markdown/cookbook-debugging.md

If you provide the resulting jfr file, then we can see the stacktrace, even if it's not in the logs.

— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1343447156, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMOZLL7QNNKNLNKFGDHDWMJNAJANCNFSM6AAAAAASJFGKB4. You are receiving this because you were mentioned.

cfgamboa commented 1 year ago

@kofemann

I am following the instructions I am getting Could not initialize class java.time.zone.ZoneRulesProvider, any ideas?

[root@dcqos001 jvm]# jcmd 2062 JFR.start filename=/tmp/dcache.jfr settings=/usr/lib/jvm/java-11-openjdk-11.0.17.0.8-2.el7_9.x86_64/lib/jfr/default.jfc
2062:
Could not initialize class java.time.zone.ZoneRulesProvider

All the best, Carlos

cfgamboa commented 1 year ago

Hello,

I managed to generate the trace requested,(digging into Java distribution intrinsics ;)). I will upload this file via RT.

All the best, Carlos

lemora commented 1 year ago

If the previous fix did not help, this might not be effective here either. But it might address the PoolManager logging.

cfgamboa commented 1 year ago

For the record, trace file was uploaded to RT ticket www.dcache.org #10406

lemora commented 1 year ago

Then we were waiting on each other, it seems. On what date did you upload the file; can you please check if the upload succeeded? I don't see any new messages from you since December 8th.

Regards Lea

cfgamboa commented 1 year ago

I have uploaded the file on Monday to the ticket you have specified 

 [www.dcache.org #10406] dCache 8.2.7 Poolmanager cell pinboard flooded with messages  Restore Manager : got 'poolRestarted' for  (Issue #6915)
cfgamboa commented 1 year ago

The message sent to support@dcache.org did not bounced back or was rejected.

lemora commented 1 year ago

Thanks, I'll try to find out where it went.

cfgamboa commented 1 year ago

Please let me know otherwise I can sent it directly to you.

lemora commented 1 year ago

Please do send it to me directly, then! While I keep investigating RT, as this should not happen again.

cfgamboa commented 1 year ago

Hi Lea,

Ok I will please note that I sent one on Saturday 10th and then resent it on Monday 12 as you suggested to use a specific ticket,

All the best, Carlos

On Dec 15, 2022, at 10:01 AM, Lea @.***> wrote:

Please do send it to me directly, then! While I keep investigating RT, as this should not happen again.

— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1353228998, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO6Y4M7L2XIPPMB5ME3WNMXGJANCNFSM6AAAAAASJFGKB4. You are receiving this because you were mentioned.

cfgamboa commented 1 year ago

No worries. Thank you for following this.

On Dec 15, 2022, at 11:03 AM, Lea @.***> wrote:

You already sent it to me on Saturday..?! Neither was received. Sorry for the issues

— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1353321558, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO2M67T2RG4Y2MOKBOTWNM6NPANCNFSM6AAAAAASJFGKB4. You are receiving this because you were mentioned.

cfgamboa commented 1 year ago

Now I understand the reason of the upload request on Monday. You were following up this. Thank you, Carlos

On Dec 15, 2022, at 11:18 AM, Carlos Fernando Gamboa @.***> wrote:

No worries. Thank you for following this.

On Dec 15, 2022, at 11:03 AM, Lea @.***> wrote:

You already sent it to me on Saturday..?! Neither was received. Sorry for the issues

— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1353321558, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO2M67T2RG4Y2MOKBOTWNM6NPANCNFSM6AAAAAASJFGKB4. You are receiving this because you were mentioned.

cfgamboa commented 1 year ago

Hello @lemora

Thank you. The error reported here in the cleaner cell and pool manager are not longer observed after installing the dCache 8.2.8 release and performing different tests. This observation was included in the dCache support RT#10406

We can close this ticket for now.

All the best and thank you for your work.

Carlos