dask / dask-gateway

A multi-tenant server for securely deploying and managing Dask clusters.
https://gateway.dask.org/
BSD 3-Clause "New" or "Revised" License
136 stars 88 forks source link

Change of controller's log level of "Reconciling cluster" #758

Closed consideRatio closed 10 months ago

consideRatio commented 11 months ago

Here are logs from the controller. Among these logs are things I'd like to read, but its swamped by the log messages about "reconciling". I think it would be good if the log level of these logs make them not seen by the default log level, allowing the rest to surface.

Current logs examplified ``` [I 2023-10-25 11:13:40.135 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:13:50.178 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:13:50.178 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 15 - creating 15 workers, deleting 0 stopped workers [I 2023-10-25 11:13:50.328 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.490 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.491 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.870 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.870 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.879 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.879 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.880 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.880 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.888 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.888 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.889 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.889 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.892 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.892 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.894 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.895 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.907 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:14:58.907 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.600 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.601 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.657 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.657 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.671 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.672 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.747 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.747 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.748 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:57.748 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:58.001 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:58.002 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:58.497 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:15:58.497 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:11.540 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:11.541 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:12.843 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:12.844 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:13.215 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:13.215 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:14.000 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:14.000 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:15.771 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:15.772 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:15.903 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:15.903 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:16.059 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:16.059 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:16.362 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:16:16.362 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:19:52.978 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:19:52.978 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:26:33.145 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:26:33.145 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:33:13.468 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:33:13.468 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:03.417 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:03.418 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:05.897 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:05.897 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 1 - deleting 0 pending and 1 stopped workers [I 2023-10-25 11:35:05.920 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:07.916 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:07.916 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 1 - deleting 0 pending and 13 stopped workers [I 2023-10-25 11:35:08.026 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:08.899 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:35:08.899 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:38:33.606 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:38:33.606 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:39:33.708 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:39:33.708 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:43:39.620 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:43:39.621 KubeController] Shutting down prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:43:39.667 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:43:39.668 KubeController] Reconciling cluster prod.b3a990d302d84720aae27404f6153ade [I 2023-10-25 11:43:39.668 KubeController] Finished reconciling cluster prod.b3a990d302d84720aae27404f6153ade ```
After lowering the log-level, the same logs examplified ``` [I 2023-10-25 11:13:50.178 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 15 - creating 15 workers, deleting 0 stopped workers [I 2023-10-25 11:35:05.897 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 1 - deleting 0 pending and 1 stopped workers [I 2023-10-25 11:35:07.916 KubeController] Cluster prod.b3a990d302d84720aae27404f6153ade scaled to 1 - deleting 0 pending and 13 stopped workers [I 2023-10-25 11:43:39.621 KubeController] Shutting down prod.b3a990d302d84720aae27404f6153ade ```