search

keras-team / keras-tuner

A Hyperparameter Tuning Library for Keras
https://keras.io/keras_tuner/
Apache License 2.0
2.86k stars 396 forks source link

Access logs from chief in distributed mode #238

Open ricastell opened 4 years ago

ricastell commented 4 years ago

I have been trying to use keras-tuner to run distributed hyperparameter tuning, but I found out that the only logger implemented now is based on sending messages from the clients to an http endpoint. I believe it would be more easy to have those messages printed to the standard output of the chief worker and then collected through some other mean.

I'm attaching to this issue a PR with a proof of concept.

omalleyt12 commented 4 years ago

@ricastell Thanks for the issue and PR!

Could you comment more on what you're trying to achieve?

IIUC, I think it might make more sense to override the Oracle class to handle something like this, since the Oracle runs on the chief worker in distributed mode:

class LoggingOracle(kt.engine.Oracle):
  def __init__(self, oracle):
    self.oracle = oracle

  def update_trial(self, ...):
    logging.info(...)
    self.oracle.update_trial(...)
ricastell commented 4 years ago

I am trying to make it easier to collect the logs from the chief node and printing to stdout seemed to me the simplest way to do that. The problem with the current system is that each worker logs independently and writes to its file system. By logging the results on the stdout i can just parse it after the tuning is completed and get the best result.

I see your point about subclassing the kt.engine.Oracle, I wasn't sure where to put the logging component. I am wondering how it would then be used in the applications, how would it be possible to specify to use an Oracle inheriting from the LoggingOracle?