When using a GridWorker, only the output of the logger of the current Worker (e.g. OfflineTrainer or Tester) is captured. The loggers of the Model & Problem are not able to write to the console or to the .log file.
This is fairly annoying as these loggers have some important information (e.g. whether the Sampler was properly instantiated, when is saved the Model etc).
Not sure yet how to fix this. I think this is related to the logging.config and also how the output of a child process is captured in the GridWorker:
When using a
GridWorker, only the output of the logger of the currentWorker(e.g.OfflineTrainerorTester) is captured. The loggers of theModel&Problemare not able to write to the console or to the.logfile. This is fairly annoying as these loggers have some important information (e.g. whether theSamplerwas properly instantiated, when is saved theModeletc).Not sure yet how to fix this. I think this is related to the
logging.configand also how the output of a child process is captured in theGridWorker:https://github.com/IBM/mi-prometheus/blob/3d4f868320c8914f37543298c634ab6ba6a1425d/miprometheus/grid_workers/grid_trainer_cpu.py#L316-L317
We should probably change
stdout=os.devnulland do something instead likewith open(worker.log, 'w') as devnull:result = subprocess.run(command_str.split(" "), stdout=devnull)