Closed trivialfis closed 1 day ago
What happens here is that you submit worker_fn()
that runs on each of the Worker
s, in there you run get_client()
which gets a client, that is a Client
from the worker process and is NOT the user-facing Client
. I suspect what you want is to move client.forward_logging(get_logger().name)
to your main()
before submitting the task, e.g.:
...
def worker_fn():
client = get_client()
client.submit(nested_fn).result()
def main():
with Client(scheduler_file="sched.json") as client:
client.forward_logging(get_logger().name)
client.submit(worker_fn).result()
Thank you for pointing it out. The use of nested function is by design, to simulate a user calling the XGBoost dask interface. The user might have arbitrary number of nested tasks before reaching the forward logging call.
Client.forward_logging
is designed such that the loggers are only forwarded to the clients that requested them. In order to forward the logging to your user's client, you'd need some API call to happen user-side and initialize forwarding.
Ah, thank you for sharing! This is really helpful.
Describe the issue:
This might be a bug in my code, but the behavior is dependent on dask, hence opening an issue here to confirm.
There are two issues here:
output:
Currently, XGBoost logs the evaluation results on the scheduler process using its own communication channel. I'm trying to redirect the logs to the client process using Dask facilities.
Minimal Complete Verifiable Example:
Start a fresh cluster, this is required to reproduce the error.
Anything else we need to know?:
Environment: