Unexpected error during training

[AntonioVispi]

The last time I started a training session with my dataset was a few days ago, but today, without changing anything, an unexpected error appeared that I had never seen at the beginning of the training. I emphasize that everything is the same as when it worked perfectly. If any further details are needed, please don't hesitate to ask. I enclose the error text, please help me. Thanks in advance.

Setting up augmentation... Distributing across 1 GPUs... Setting up training phases... Exporting sample images... Initializing logs... Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorboard/compat/init.py", line 42, in tf from tensorboard.compat import notf # noqa: F401 ImportError: cannot import name 'notf' from 'tensorboard.compat' (/usr/local/lib/python3.7/dist-packages/tensorboard/compat/init.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train.py", line 286, in main() # pylint: disable=no-value-for-parameter File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke return callback(args, kwargs) File "train.py", line 281, in main launch_training(c=c, desc=desc, outdir=opts.outdir, dry_run=opts.dry_run) File "train.py", line 96, in launch_training subprocess_fn(rank=0, c=c, temp_dir=temp_dir) File "train.py", line 47, in subprocess_fn training_loop.training_loop(rank=rank, c) File "/content/stylegan3/training/training_loop.py", line 238, in training_loop stats_tfevents = tensorboard.SummaryWriter(run_dir) File "/usr/local/lib/python3.7/dist-packages/torch/utils/tensorboard/writer.py", line 246, in init self._get_file_writer() File "/usr/local/lib/python3.7/dist-packages/torch/utils/tensorboard/writer.py", line 277, in _get_file_writer self.log_dir, self.max_queue, self.flush_secs, self.filename_suffix File "/usr/local/lib/python3.7/dist-packages/torch/utils/tensorboard/writer.py", line 76, in init log_dir, max_queue, flush_secs, filename_suffix File "/usr/local/lib/python3.7/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 72, in init tf.io.gfile.makedirs(logdir) File "/usr/local/lib/python3.7/dist-packages/tensorboard/lazy.py", line 65, in getattr return getattr(load_once(self), attr_name) File "/usr/local/lib/python3.7/dist-packages/tensorboard/lazy.py", line 97, in wrapper cache[arg] = f(arg) File "/usr/local/lib/python3.7/dist-packages/tensorboard/lazy.py", line 50, in load_once module = load_fn() File "/usr/local/lib/python3.7/dist-packages/tensorboard/compat/init.py", line 45, in tf import tensorflow File "/usr/local/lib/python3.7/dist-packages/tensorflow/init.py", line 51, in from ._api.v2 import compat File "/usr/local/lib/python3.7/dist-packages/tensorflow/_api/v2/compat/init.py", line 37, in from . import v1 File "/usr/local/lib/python3.7/dist-packages/tensorflow/_api/v2/compat/v1/init.py", line 30, in from . import compat File "/usr/local/lib/python3.7/dist-packages/tensorflow/_api/v2/compat/v1/compat/init.py", line 37, in from . import v1 File "/usr/local/lib/python3.7/dist-packages/tensorflow/_api/v2/compat/v1/compat/v1/init.py", line 47, in from tensorflow._api.v2.compat.v1 import lite File "/usr/local/lib/python3.7/dist-packages/tensorflow/_api/v2/compat/v1/lite/init.py", line 9, in from . import experimental File "/usr/local/lib/python3.7/dist-packages/tensorflow/_api/v2/compat/v1/lite/experimental/init.py", line 8, in from . import authoring File "/usr/local/lib/python3.7/dist-packages/tensorflow/_api/v2/compat/v1/lite/experimental/authoring/init.py", line 8, in from tensorflow.lite.python.authoring.authoring import compatible File "/usr/local/lib/python3.7/dist-packages/tensorflow/lite/python/authoring/authoring.py", line 43, in from tensorflow.lite.python import convert File "/usr/local/lib/python3.7/dist-packages/tensorflow/lite/python/convert.py", line 29, in from tensorflow.lite.python import util File "/usr/local/lib/python3.7/dist-packages/tensorflow/lite/python/util.py", line 51, in from jax import xla_computation as _xla_computation File "/usr/local/lib/python3.7/dist-packages/jax/init.py", line 59, in from .core import eval_context as ensure_compile_time_eval File "/usr/local/lib/python3.7/dist-packages/jax/core.py", line 47, in import jax._src.pretty_printer as pp File "/usr/local/lib/python3.7/dist-packages/jax/_src/pretty_printer.py", line 56, in CAN_USE_COLOR = _can_use_color() File "/usr/local/lib/python3.7/dist-packages/jax/_src/pretty_printer.py", line 54, in _can_use_color return sys.stdout.isatty() AttributeError: 'Logger' object has no attribute 'isatty'

[AntonioVispi]

I specify that I am working on Colab

[nuclearsugar]

Solution is here - https://github.com/NVlabs/stylegan3/issues/181

[AntonioVispi]

Thank you so much, it worked!

On Fri, 22 Jul 2022, 03:49 nuclearsugar, @.***> wrote:

Solution is here -

181 https://github.com/NVlabs/stylegan3/issues/181

— Reply to this email directly, view it on GitHub https://github.com/NVlabs/stylegan3/issues/183#issuecomment-1192097654, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYOE7GUWJRX2NYCC6B7D3Q3VVH43LANCNFSM532PWEIQ . You are receiving this because you authored the thread.Message ID: @.***>