search

dask / dask-gateway

A multi-tenant server for securely deploying and managing Dask clusters.
https://gateway.dask.org/
BSD 3-Clause "New" or "Revised" License
136 stars 88 forks source link

Reduce log level of k8s controller's reconciling loop's messages #771

Closed consideRatio closed 10 months ago

consideRatio commented 10 months ago
Current logs examplified ``` [I 2023-10-25 11:13:40.135 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:13:50.178 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:13:50.178 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 15 - creating 15 workers, deleting 0 stopped workers [I 2023-10-25 11:13:50.328 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.490 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.491 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.870 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.870 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.879 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.879 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.880 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.880 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.888 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.888 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.889 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.889 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.892 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.892 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.894 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.895 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.907 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.907 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.600 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.601 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.657 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.657 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.671 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.672 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.747 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.747 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.748 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.748 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:58.001 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:58.002 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:58.497 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:58.497 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:11.540 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:11.541 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:12.843 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:12.844 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:13.215 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:13.215 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:14.000 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:14.000 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:15.771 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:15.772 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:15.903 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:15.903 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:16.059 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:16.059 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:16.362 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:16.362 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:19:52.978 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:19:52.978 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:26:33.145 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:26:33.145 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:33:13.468 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:33:13.468 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:03.417 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:03.418 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:05.897 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:05.897 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 1 - deleting 0 pending and 1 stopped workers [I 2023-10-25 11:35:05.920 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:07.916 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:07.916 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 1 - deleting 0 pending and 13 stopped workers [I 2023-10-25 11:35:08.026 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:08.899 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:08.899 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:38:33.606 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:38:33.606 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:39:33.708 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:39:33.708 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:43:39.620 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:43:39.621 KubeController] Shutting down prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:43:39.667 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:43:39.668 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:43:39.668 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade ```
After lowering the log-level, the same logs examplified ``` [I 2023-10-25 11:13:50.178 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 15 - creating 15 workers, deleting 0 stopped workers [I 2023-10-25 11:35:05.897 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 1 - deleting 0 pending and 1 stopped workers [I 2023-10-25 11:35:07.916 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 1 - deleting 0 pending and 13 stopped workers [I 2023-10-25 11:43:39.621 KubeController] Shutting down prod.b3a990d302d84720aae27404f6153ade ```
consideRatio commented 10 months ago

Thank you @TomAugspurger for reviewing!