wandb crashes - Githubissues

Version 0.10.2 Env: Colab Notebook

During the first run I claimed the run in wandb. After first run completes this error is shown. Any more runs cannot be started without this error coming up when generation begins.

Problem at: /usr/local/lib/python3.7/dist-packages/discoart/runner.py 406 do_run

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/wandb/sdk/wandb_init.py", line 1043, in init
    run = wi.init()
  File "/usr/local/lib/python3.7/dist-packages/wandb/sdk/wandb_init.py", line 689, in init
    backend.cleanup()
  File "/usr/local/lib/python3.7/dist-packages/wandb/sdk/backend/backend.py", line 246, in cleanup
    self.interface.join()
  File "/usr/local/lib/python3.7/dist-packages/wandb/sdk/interface/interface_shared.py", line 475, in join
    super().join()
  File "/usr/local/lib/python3.7/dist-packages/wandb/sdk/interface/interface.py", line 666, in join
    _ = self._communicate_shutdown()
  File "/usr/local/lib/python3.7/dist-packages/wandb/sdk/interface/interface_shared.py", line 472, in _communicate_shutdown
    _ = self._communicate(record)
  File "/usr/local/lib/python3.7/dist-packages/wandb/sdk/interface/interface_shared.py", line 226, in _communicate
    return self._communicate_async(rec, local=local).get(timeout=timeout)
  File "/usr/local/lib/python3.7/dist-packages/wandb/sdk/interface/interface_shared.py", line 231, in _communicate_async
    raise Exception("The wandb backend process has shutdown")
Exception: The wandb backend process has shutdown
wandb: ERROR Abnormal program exit

---------------------------------------------------------------------------

Exception                                 Traceback (most recent call last)

[/usr/local/lib/python3.7/dist-packages/wandb/sdk/wandb_init.py](https://localhost:8080/#) in init(job_type, dir, config, project, entity, reinit, tags, group, name, notes, magic, config_exclude_keys, config_include_keys, anonymous, mode, allow_val_change, resume, force, tensorboard, sync_tensorboard, monitor_gym, save_code, id, settings)
   1042         try:
-> 1043             run = wi.init()
   1044             except_exit = wi.settings._except_exit

10 frames

Exception: The wandb backend process has shutdown

The above exception was the direct cause of the following exception:

Exception                                 Traceback (most recent call last)

[/usr/local/lib/python3.7/dist-packages/wandb/sdk/wandb_init.py](https://localhost:8080/#) in init(job_type, dir, config, project, entity, reinit, tags, group, name, notes, magic, config_exclude_keys, config_include_keys, anonymous, mode, allow_val_change, resume, force, tensorboard, sync_tensorboard, monitor_gym, save_code, id, settings)
   1079             if except_exit:
   1080                 os._exit(-1)
-> 1081             raise Exception("problem") from error_seen
   1082     return run

Exception: problem

rm ~/.netrc

In case the following error, delete wandb credentials:

andb: ERROR Error while calling W&B API: user is not logged in (<Response [401]>)
Thread SenderThread:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/lib/retry.py", line 108, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/internal/internal_api.py", line 158, in execute
    return self.client.execute(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute
    result = self._get_result(document, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/wandb/vendor/gql-0.2.0/wandb_gql/transport/requests.py", line 39, in execute
    request.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://api.wandb.ai/graphql

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/wandb/apis/normalize.py", line 25, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/internal/internal_api.py", line 1341, in upsert_run
    response = self.gql(mutation, variable_values=variable_values, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/lib/retry.py", line 124, in __call__
    if not check_retry_fn(e):
  File "/usr/local/lib/python3.8/dist-packages/wandb/util.py", line 883, in no_retry_auth
    raise CommError("Invalid or missing api_key. Run `wandb login`")
wandb.errors.CommError: Invalid or missing api_key. Run `wandb login`

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/internal/internal_util.py", line 51, in run
    self._run()
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/internal/internal_util.py", line 102, in _run
    self._process(record)
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/internal/internal.py", line 310, in _process
    self._sm.send(record)
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/internal/sender.py", line 238, in send
    send_handler(record)
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/internal/sender.py", line 696, in send_run
    self._init_run(run, config_value_dict)
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/internal/sender.py", line 720, in _init_run
    server_run, inserted = self._api.upsert_run(
  File "/usr/local/lib/python3.8/dist-packages/wandb/apis/normalize.py", line 61, in wrapper
    raise CommError(message, err).with_traceback(sys.exc_info()[2])
  File "/usr/local/lib/python3.8/dist-packages/wandb/apis/normalize.py", line 25, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/internal/internal_api.py", line 1341, in upsert_run
    response = self.gql(mutation, variable_values=variable_values, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/lib/retry.py", line 124, in __call__
    if not check_retry_fn(e):
  File "/usr/local/lib/python3.8/dist-packages/wandb/util.py", line 883, in no_retry_auth
    raise CommError("Invalid or missing api_key. Run `wandb login`")
wandb.errors.CommError: Invalid or missing api_key. Run `wandb login`
wandb: ERROR Internal wandb error: file data was not synced
Problem at: /usr/local/lib/python3.8/dist-packages/discoart/runner.py 402 do_run
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/wandb_init.py", line 1043, in init
    run = wi.init()
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/wandb_init.py", line 689, in init
    backend.cleanup()
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/backend/backend.py", line 246, in cleanup
    self.interface.join()
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/interface/interface_shared.py", line 475, in join
    super().join()
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/interface/interface.py", line 666, in join
    _ = self._communicate_shutdown()
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/interface/interface_shared.py", line 472, in _communicate_shutdown
    _ = self._communicate(record)
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/interface/interface_shared.py", line 226, in _communicate
    return self._communicate_async(rec, local=local).get(timeout=timeout)
  File "/usr/local/lib/python3.8/dist-packages/wandb/sdk/interface/interface_shared.py", line 231, in _communicate_async
    raise Exception("The wandb backend process has shutdown")
Exception: The wandb backend process has shutdown
wandb: ERROR Abnormal program exit

jina-ai / discoart

wandb crashes #111