Closed cfgamboa closed 1 year ago
Hi Carlos. Thanks for reporting, I will look into it. Has this error appeared more than once in the cleaner log, and has it affected the cleaner behaviour at all?
@lemora thank you and sorry I missed your question. Thought is already marked as bug. I will basically confirm that is a appeared more than once on the logs.
Sure, every NPE is a bug, of course. But I still need to understand how and what is happening. How often do you see this message per day, and does that correlate with other messages -- do files not get cleaned as a consequence?
Hello
See below:
[dcqos001] (cleaner@cleanerDomain) admin > show pinboard 10000
01 Dec 2022 14:45:21 [cleaner-19] [dcqos002_20 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:21 [cleaner-19] [dcqos002_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:21 [cleaner-19] [dcqos002_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:21 [cleaner-19] [dcqos002_23 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:21 [cleaner-19] [dcqos002_17 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_24 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_15 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_21 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_19 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_18 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_22 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_16 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:22 [cleaner-19] [dcqos002_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:26 [cleaner-19] [dcqos005_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:29 [cleaner-19] [dcqos005_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_3 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_1 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_4 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:49 [cleaner-19] [dcqos001_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:50 [cleaner-19] [dcqos001_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:50 [cleaner-19] [dcqos001_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:50 [cleaner-19] [dcqos001_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:50 [cleaner-19] [dcqos001_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:50 [cleaner-19] [dcqos001_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:51 [cleaner-19] [dcqos002_20 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:51 [cleaner-19] [dcqos002_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:51 [cleaner-19] [dcqos002_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:51 [cleaner-19] [dcqos002_23 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:51 [cleaner-19] [dcqos002_17 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_24 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_15 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_21 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_19 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_18 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_22 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_16 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:52 [cleaner-19] [dcqos002_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:53 [pool-6-thread-31] [Disk-Cleaner] New run...
01 Dec 2022 14:45:56 [cleaner-19] [dcqos005_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:45:59 [cleaner-19] [dcqos005_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_3 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_1 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_4 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:19 [cleaner-19] [dcqos001_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:20 [cleaner-19] [dcqos001_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:20 [cleaner-19] [dcqos001_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:20 [cleaner-19] [dcqos001_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:20 [cleaner-19] [dcqos001_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:20 [cleaner-19] [dcqos001_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:21 [cleaner-19] [dcqos002_20 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:21 [cleaner-19] [dcqos002_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:21 [cleaner-19] [dcqos002_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:21 [cleaner-19] [dcqos002_23 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:21 [cleaner-19] [dcqos002_17 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_24 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_15 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_21 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_19 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_18 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_22 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_16 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:22 [cleaner-19] [dcqos002_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:26 [cleaner-19] [dcqos005_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:29 [cleaner-19] [dcqos005_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_3 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_1 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_4 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:49 [cleaner-19] [dcqos001_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:50 [cleaner-19] [dcqos001_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:50 [cleaner-19] [dcqos001_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:50 [cleaner-19] [dcqos001_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:50 [cleaner-19] [dcqos001_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:50 [cleaner-19] [dcqos001_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:51 [cleaner-19] [dcqos002_20 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:51 [cleaner-19] [dcqos002_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:51 [cleaner-19] [dcqos002_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:51 [cleaner-19] [dcqos002_23 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:51 [cleaner-19] [dcqos002_17 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_24 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_15 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_21 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_19 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_18 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_22 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_16 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:52 [cleaner-19] [dcqos002_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:56 [cleaner-19] [dcqos005_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:46:59 [cleaner-19] [dcqos005_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_3 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_1 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_4 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:19 [cleaner-19] [dcqos001_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:20 [cleaner-19] [dcqos001_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:20 [cleaner-19] [dcqos001_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:20 [cleaner-19] [dcqos001_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:20 [cleaner-19] [dcqos001_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:20 [cleaner-19] [dcqos001_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:21 [cleaner-19] [dcqos002_20 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:21 [cleaner-19] [dcqos002_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:21 [cleaner-19] [dcqos002_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:21 [cleaner-19] [dcqos002_23 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:21 [cleaner-19] [dcqos002_17 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_24 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_15 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_21 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_19 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_18 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_22 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_16 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:22 [cleaner-19] [dcqos002_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:26 [cleaner-19] [dcqos005_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:29 [cleaner-19] [dcqos005_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_3 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_1 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_4 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:49 [cleaner-19] [dcqos001_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:50 [cleaner-19] [dcqos001_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:50 [cleaner-19] [dcqos001_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:50 [cleaner-19] [dcqos001_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:50 [cleaner-19] [dcqos001_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:50 [cleaner-19] [dcqos001_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:51 [cleaner-19] [dcqos002_20 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:51 [cleaner-19] [dcqos002_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:51 [cleaner-19] [dcqos002_10 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:51 [cleaner-19] [dcqos002_23 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:51 [cleaner-19] [dcqos002_17 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_24 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_5 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_11 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_15 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_13 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_21 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_19 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_18 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_22 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_7 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_16 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:52 [cleaner-19] [dcqos002_6 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:53 [pool-6-thread-31] [Disk-Cleaner] New run...
01 Dec 2022 14:47:56 [cleaner-19] [dcqos005_8 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:47:59 [cleaner-19] [dcqos005_9 PoolManagerPoolUp] Uncaught exception in thread cleaner-19java.lang.NullPointerException: null
01 Dec 2022 14:48:12 [pool-6-thread-98] [] Finished deleting from pool dcqos001_2
Thanks. As a side question -- do you run both the disk and hsm components of the cleaner here?
I have an idea what is going on. Could you please check if the listed pools for which the NPE occurs are not attached to an hsm backend or if they are flagged lfs=precious
?
Only DISK components not HSM
Carlos
On Dec 2, 2022, at 7:51 AM, Lea @.***> wrote:
Thanks. Do you run both the disk and hsm components of the cleaner here?
— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1335183590, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMOYIKD4B2Q32L2EIBGLWLHWF5ANCNFSM6AAAAAASJFGKB4. You are receiving this because you authored the thread.
Hey Carlos.
Okay, so I expect the pools are marked as such. Even if you only run the disk cleaner this is relevant here. The fix will be included in the next bugfix release series. Could you then please confirm if it works for you?
Thanks. Lea
Hi Lea,
The pools is configured as
[${host.name}thirteenDomain]
[${host.name}thirteenDomain/pool]
pool.name=${host.name}_13
pool.path=/data11/dcache_pool_13/pool
pool.wait-for-files=${pool.path}/data
pool.lfs=precious
Carlos
Hi Lea,
This might be related to https://github.com/dCache/dcache/issues/6915 All the best, Carlos
Hi Carlos,
thanks for the configuration.
I don't think the linked issue is related; at least the problem you saw here is explanable by a recent change, where pools marked as pool.lfs=precious
no longer announce to the cleaner that they are connected to an hsm and thus available to the hsm component. This was not correctly handled on the cleaner side but is fixed with the posted patch.
Since disk and hsm cleaner are still merged in dCache 8.2, this even affected cleaner instances in which the hsm component was turned off like in your case.
Regards Lea
Hey Lea,
I think I was able to reproduce this
07 Dec 2022 16:07:09 [cleaner-2] [dcqos001_12 PoolManagerPoolUp] Uncaught exception in thread cleaner-2java.lang.NullPointerException: null
If I disable the pool.lfs=precious
and reload the dcqos001_12 pool the message is not longer show in the pinboard.
My understanding is that for DISK only pools this flag might not be necessary pool.lfs=precious
@kofemann or you could please confirm?
All the best, Carlos
If I disable the pool.lfs=precious and reload the dcqos001_12 pool the message is not longer show in the pinboard.
Okay, good to know. Before, the cleaner did not care either way: pool.lfs=precious
or not. You can remove the flag if files on these pools are pinned or otherwise ensured to not be cache-removed, but to be clear here: the NullPointerException
is in either case a bug on our side that needed fixing.
You will not need to deploy the fix on the pools, just the cleaner instance in order for everything to hopefully be NPE-free again. The newest bugfix versions containing the fix were released today.
OK I am using 8.2.7 on these nodes/pools. The results/tests reported today were done using that release.
Hey,
oh, so do I understand correctly that you still see the NPEs in 8.2.7 when there are pools marked as pool.lfs=precious
?
It is interesting that the PoolManager pinboard messages disappear when pools are not lfs=precious anymore... This needs some more digging.
Hey
Your understanding is correct. For example I have enabled the pool.lfs=precious
Pool conf
[${host.name}fourteenDomain]
[${host.name}fourteenDomain/pool]
pool.name=${host.name}_14
pool.path=/data12/dcache_pool_14/pool
pool.wait-for-files=${pool.path}/data
pool.lfs=precious
Then after rebooting the pool The cleaner pinboard
08 Dec 2022 10:20:45 [cleaner-2] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2java.lang.NullPointerException: null
08 Dec 2022 10:20:45 [cleaner-2] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2java.lang.NullPointerException: null
08 Dec 2022 10:20:45 [cleaner-2] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2java.lang.NullPointerException: null
08 Dec 2022 10:20:45 [cleaner-2] [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2java.lang.NullPointerException: null
And the PoolManager pinboard
08 Dec 2022 10:20:34 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Pool dcqos001_14 changed from mode enabled to disabled(fetch,store,stage,p2p-client,p2p-server,dead).
08 Dec 2022 10:20:34 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Restore Manager : got 'poolRestarted' for dcqos001_14
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Pool dcqos001_14 changed from mode disabled(fetch,store,stage,p2p-client,p2p-server,dead) to disabled(fetch,store,stage,p2p-client,p2p-server).
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Restore Manager : got 'poolRestarted' for dcqos001_14
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Restore Manager : got 'poolRestarted' for dcqos001_14
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Pool dcqos001_14 changed from mode disabled(fetch,store,stage,p2p-client,p2p-server) to disabled(store,stage,p2p-client,loading).
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Restore Manager : got 'poolRestarted' for dcqos001_14
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Pool dcqos001_14 changed from mode disabled(store,stage,p2p-client,loading) to enabled.
08 Dec 2022 10:20:45 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Restore Manager : got 'poolRestarted' for dcqos001_14
08 Dec 2022 10:21:15 [PoolManager-46] [dcqos001_14 PoolManagerPoolUp] Restore Manager : got 'poolRestarted' for dcqos001_14
OK I am using 8.2.7 on these nodes/pools. The results/tests reported today were done using that release.
Just to clarify -- did you also upgrade the cleaner to 8.2.7, not just the pools (the latter was not needed in any case)?
Otherwise, could you please try again to find a full stack trace for the NPE after the upgrade?
Okay, I think I understand why the PoolManager pinboard is flooded with these 'poolRestarted' messages when there are
pool.lfs=precious
-pools -- it again relates to the original "fix" which lets the cleaner know not to send hsm delete requests to pools marked as lfs-precious
. I'll post another fix that should address the issue -- including GitHub issue #6915 , if that is okay with you, @alrossi, since you already started looking into it. But these two indeed seem related.
But first, Carlos, I need confirmation from you that you indeed upgraded the cleaner as well, since that could mean that I don't understand the problem as well as I think.
Hi Lea, Indeed the cleaner is running on 8.2.7. All the best, Carlos
Hi Carlos. Has the full stack trace changed at all? Regards, Lea
Hola Lea,
Yes, the stack trace changed, before showed more messages an example is included at the beginning of the ticket. This is the current one.
Dec 8 15:34:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:34:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2
Dec 8 15:34:45 dcqos001 dcache@cleanerDomain: java.lang.NullPointerException: null
Carlos
Hi Carlos.
Our logging system suppresses the whole stack trace if it is logged too often, then one just sees what you sent in your last message, for example: java.lang.NullPointerException: null
Could you try to find the first occurrence of the NPE in order to get the full stack trace? The truncated one unfortunately does not help much.
Thanks. Lea
Hi Lea, I have enabled the cleaner in debug all.
Dec 8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] Executing SQL query [SELECT DISTINCT ilocation FROM t_locationinfo_trash WHERE itype=1]
Dec 8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] Fetching JDBC Connection from DataSource
Dec 8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [] Executing SQL query [SELECT ipnfsid FROM t_locationinfo_trash t1 WHERE itype=2 AND NOT EXISTS (SELECT 1 FROM t_locationinfo_trash t2 WHERE t2.ipnfsid=t1.ipnfsid AND t2.itype <> 2)]
Dec 8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [] Fetching JDBC Connection from DataSource
Dec 8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [] Finished deleting from pool dcqos001_2
Dec 8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] sendMessage : <7379323899589971090:-5208671900136200888> send to [>admin@adminDoorDomain]
Dec 8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] sendMessage : next hop at 16: admin@adminDoorDomain
Dec 8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] sendMessage : using route : CellRoute{cell=*, domain=*, gateway=c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain, type=Default, zone=Undefined}
Dec 8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] sendMessage : next hop at 15: c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain
Dec 8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [caE:2o admin] addToEventQueue : message arrived : <CM: S=[>cleaner@cleanerDomain:*@cleanerDomain];D=[>admin@adminDoorDomain];C=Stream;O=<7379323899589971090:-5208671900136200888>;LO=<321142552260658636:-7201269758871426662>;SID=caE:2o;TTL=300000>
Dec 8 15:50:00 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:00 (cleaner) [] HikariPool-1 - Closing connection org.postgresql.jdbc.PgConnection@7a8e717a: (connection has passed maxLifetime)
Dec 8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] New run...
Dec 8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] INFO: Refresh Interval : 120 SECONDS
Dec 8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] INFO: Number of files processed at once: 10000
Dec 8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] Executing SQL query [SELECT DISTINCT ilocation FROM t_locationinfo_trash WHERE itype=1]
Dec 8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] Fetching JDBC Connection from DataSource
Dec 8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] List of Pools from the trash-table : [dcqos001_2]
Dec 8 15:50:07 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:07 (cleaner) [Disk-Cleaner] 1 pools are currently blacklisted.
Dec 8 15:50:15 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:15 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : <-7571265469618568800:-8456256691570891820> send to [>dcqos001_14@dcqos001fourteenDomain]
Dec 8 15:50:15 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:15 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : next hop at 16: dcqos001_14@dcqos001fourteenDomain
Dec 8 15:50:15 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:15 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : using route : CellRoute{cell=*, domain=*, gateway=c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain, type=Default, zone=Undefined}
Dec 8 15:50:15 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:15 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : next hop at 15: c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain
Dec 8 15:50:15 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:15 (cleaner) [dcqos001_14 PoolManagerPoolUp] addToEventQueue : message arrived : <CM: S=[>cleaner@cleanerDomain:*@cleanerDomain];D=[>dcqos001_14@dcqos001fourteenDomain];C=Stream;O=<-7571265469618568800:-8456256691570891820>;LO=<8341864328583135877:-7157650930398023863>>
Dec 8 15:50:15 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:15 (cleaner) [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2
Dec 8 15:50:15 dcqos001 dcache@cleanerDomain: java.lang.NullPointerException: null
Dec 8 15:50:18 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:18 (cleaner) [] HikariPool-1 - Before cleanup stats (total=2, active=0, idle=2, waiting=0)
Dec 8 15:50:18 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:18 (cleaner) [] HikariPool-1 - After cleanup stats (total=1, active=0, idle=1, waiting=0)
Dec 8 15:50:18 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:18 (cleaner) [] HikariPool-1 - Closing connection org.postgresql.jdbc.PgConnection@2ca205d8: (connection has passed idleTimeout)
Dec 8 15:50:42 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:42 (cleaner) [Frontend-dcqos001] sendMessage : <-8298066119329755423:-8704052522908946887> send to [>Frontend-dcqos001@srr-dcqos001Domain]
Dec 8 15:50:42 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:42 (cleaner) [Frontend-dcqos001] sendMessage : next hop at 16: Frontend-dcqos001@srr-dcqos001Domain
Dec 8 15:50:42 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:42 (cleaner) [Frontend-dcqos001] sendMessage : using route : CellRoute{cell=*, domain=*, gateway=c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain, type=Default, zone=Undefined}
Dec 8 15:50:42 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:42 (cleaner) [Frontend-dcqos001] sendMessage : next hop at 15: c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain
Dec 8 15:50:42 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:42 (cleaner) [Frontend-dcqos001] addToEventQueue : message arrived : <CM: S=[>cleaner@cleanerDomain:*@cleanerDomain];D=[>Frontend-dcqos001@srr-dcqos001Domain];C=Stream;O=<-8298066119329755423:-8704052522908946887>;LO=<-1501833629673042479:-7302381553561486839>;TTL=60000>
Dec 8 15:50:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : <-271099534625585754:-6215812878134086060> send to [>dcqos001_14@dcqos001fourteenDomain]
Dec 8 15:50:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : next hop at 16: dcqos001_14@dcqos001fourteenDomain
Dec 8 15:50:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : using route : CellRoute{cell=*, domain=*, gateway=c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain, type=Default, zone=Undefined}
Dec 8 15:50:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] sendMessage : next hop at 15: c-dCacheDomain-AAXvPj-mAUA-AAXvPj-nl4A@cleanerDomain
Dec 8 15:50:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] addToEventQueue : message arrived : <CM: S=[>cleaner@cleanerDomain:*@cleanerDomain];D=[>dcqos001_14@dcqos001fourteenDomain];C=Stream;O=<-271099534625585754:-6215812878134086060>;LO=<9036826481595072635:-7979064345865280826>>
Dec 8 15:50:45 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:45 (cleaner) [dcqos001_14 PoolManagerPoolUp] Uncaught exception in thread cleaner-2
Dec 8 15:50:45 dcqos001 dcache@cleanerDomain: java.lang.NullPointerException: null
Dec 8 15:50:48 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:48 (cleaner) [] HikariPool-1 - Before cleanup stats (total=1, active=0, idle=1, waiting=0)
Dec 8 15:50:48 dcqos001 dcache@cleanerDomain: 08 Dec 2022 15:50:48 (cleaner) [] HikariPool-1 - After cleanup stats (total=1, active=0, idle=1, waiting=0)
Debug doesn't help to procure the stack trace, unfortunately. One has to search for the first NPE occurrence after the upgrade and restart of the cleaner, I fear. The place is simliar, as expected, though. But anyway, I'll try to understand what is going on tomorrow. I expected the patch to fix this.
Hello Lea,
OK. In any case, If you would like to discuss further or troubleshoot more we could schedule a zoom call for tomorrow, just let me know.
All the best, Carlos
On Dec 8, 2022, at 3:57 PM, Lea @.***> wrote:
Debug doesn't help to procure the stack trace, unfortunately. One has to search for the first NPE occurrence after the upgrade and restart of the cleaner, I fear. But anyway, I'll try to understand what is going on tomorrow. I expected the patch to fix this.
— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1343343202, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO6Y6CPES2C4K3OGUXTWMJDVPANCNFSM6AAAAAASJFGKB4. You are receiving this because you authored the thread.
Hey.
If you find the time to look, I would really like to see that stack trace. :) Other than that I don't think a zoom meeting would benefit us here, though.
Cheers Lea
Hi
Where I should look for the trace files? Carlos
On Dec 8, 2022, at 4:34 PM, Lea @.***> wrote:
Hey.
If you find the time to look, I would really like to see that stack trace. :) Other than that I don't think a zoom meeting would benefit us here, though.
Cheers Lea
— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1343394086, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO5H2IQWBHRQ2PEUBQDWMJH4ZANCNFSM6AAAAAASJFGKB4. You are receiving this because you authored the thread.
You will find it in the regular logs, it should look like the one you initially posted here. The first NullPointerException
after you restarted the cleaner for upgrading it to 8.2.7 yesterday should probably be the full one.
Hi @cfgamboa ,
You can use java flight recorder to collect hidden stacktraces as described in
https://github.com/dCache/dcache/blob/master/docs/TheBook/src/main/markdown/cookbook-debugging.md
If you provide the resulting jfr file, then we can see the stacktrace, even if it's not in the logs.
Hi I will give a try tomorrow, thank you for the reference. Carlos
On Dec 8, 2022, at 5:17 PM, Tiramisu Mokka @.***> wrote:
Hi @cfgamboa https://github.com/cfgamboa ,
You can use java flight recorder to collect hidden stacktraces as described in
https://github.com/dCache/dcache/blob/master/docs/TheBook/src/main/markdown/cookbook-debugging.md
If you provide the resulting jfr file, then we can see the stacktrace, even if it's not in the logs.
— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1343447156, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMOZLL7QNNKNLNKFGDHDWMJNAJANCNFSM6AAAAAASJFGKB4. You are receiving this because you were mentioned.
@kofemann
I am following the instructions I am getting Could not initialize class java.time.zone.ZoneRulesProvider
, any ideas?
[root@dcqos001 jvm]# jcmd 2062 JFR.start filename=/tmp/dcache.jfr settings=/usr/lib/jvm/java-11-openjdk-11.0.17.0.8-2.el7_9.x86_64/lib/jfr/default.jfc
2062:
Could not initialize class java.time.zone.ZoneRulesProvider
All the best, Carlos
Hello,
I managed to generate the trace requested,(digging into Java distribution intrinsics ;)). I will upload this file via RT.
All the best, Carlos
If the previous fix did not help, this might not be effective here either. But it might address the PoolManager
logging.
For the record, trace file was uploaded to RT ticket www.dcache.org #10406
Then we were waiting on each other, it seems. On what date did you upload the file; can you please check if the upload succeeded? I don't see any new messages from you since December 8th.
Regards Lea
I have uploaded the file on Monday to the ticket you have specified
[www.dcache.org #10406] dCache 8.2.7 Poolmanager cell pinboard flooded with messages Restore Manager : got 'poolRestarted' for (Issue #6915)
The message sent to support@dcache.org did not bounced back or was rejected.
Thanks, I'll try to find out where it went.
Please let me know otherwise I can sent it directly to you.
Please do send it to me directly, then! While I keep investigating RT, as this should not happen again.
Hi Lea,
Ok I will please note that I sent one on Saturday 10th and then resent it on Monday 12 as you suggested to use a specific ticket,
All the best, Carlos
On Dec 15, 2022, at 10:01 AM, Lea @.***> wrote:
Please do send it to me directly, then! While I keep investigating RT, as this should not happen again.
— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1353228998, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO6Y4M7L2XIPPMB5ME3WNMXGJANCNFSM6AAAAAASJFGKB4. You are receiving this because you were mentioned.
No worries. Thank you for following this.
On Dec 15, 2022, at 11:03 AM, Lea @.***> wrote:
You already sent it to me on Saturday..?! Neither was received. Sorry for the issues
— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1353321558, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO2M67T2RG4Y2MOKBOTWNM6NPANCNFSM6AAAAAASJFGKB4. You are receiving this because you were mentioned.
Now I understand the reason of the upload request on Monday. You were following up this. Thank you, Carlos
On Dec 15, 2022, at 11:18 AM, Carlos Fernando Gamboa @.***> wrote:
No worries. Thank you for following this.
On Dec 15, 2022, at 11:03 AM, Lea @.***> wrote:
You already sent it to me on Saturday..?! Neither was received. Sorry for the issues
— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/6879#issuecomment-1353321558, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHIHMO2M67T2RG4Y2MOKBOTWNM6NPANCNFSM6AAAAAASJFGKB4. You are receiving this because you were mentioned.
Hello @lemora
Thank you. The error reported here in the cleaner cell and pool manager are not longer observed after installing the dCache 8.2.8 release and performing different tests. This observation was included in the dCache support RT#10406
We can close this ticket for now.
All the best and thank you for your work.
Carlos
Hello
Following error is reported at the cleaner log This instance is recently upgraded to 8.2.4
Could you please advise
All the best, Carlos