search

tensorlab / tensorfx

TensorFlow framework for training and serving machine learning models
Apache License 2.0
196 stars 41 forks source link

framework does not print logs outside its launcher #9

Closed brandondutra closed 7 years ago

brandondutra commented 7 years ago

Running "python -m iris.trainer.main --data-train ..." does not print anything because some environment variable is not set. This also means "gcloud ml-engine local train --module-name trainer.main --package-path trainer/ -- --data-train" does not print anything (but it does do training)

Likewise, gcloud ml-engine local train --distributed does do training, but prints nothing out.

nikhilk commented 7 years ago

Good catch the other day about missing logs. I thought a bit more about this -- we could probably do two things.

  1. Use job: { local: True } when there is no TF_CONFIG env var defined. I think this is what you were also suggesting.
  2. Check if logging has a handler set. This helps take care of the situations where the framework is invoked in the context of some other app.

When local = True, and there is no logging handler, assign a logging handler as we are doing right now.