Shifts-Project / shifts

This repository contains data readers and examples for the three tracks of the Shifts Dataset and the Shifts Challenge.
https://shifts.ai
Apache License 2.0
225 stars 49 forks source link

WANDB error: wandb: ERROR Error while calling W&B API: permission denied (<Response [403]>) #35

Closed owoshch closed 2 years ago

owoshch commented 2 years ago

Hi! Thank you for your wonderful work!

I'm trying to reproduce the training of the baselines models you provided.

I created the wandb account owoshch. Within this account, I create the project test. Then I successfully logged in to the account through the command line.

I'm running the command python run.py --dir_data /datasets/YandexShifts/data/ --wandb_project test --verbose True and getting the following error:

wandb: Currently logged in as: owoshch (usewandb login --relogin` to force relogin) wandb: ERROR Error while calling W&B API: permission denied (<Response [403]>) Thread SenderThread: Traceback (most recent call last): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/lib/retry.py", line 102, in call result = self._call_fn(*args, kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal_api.py", line 147, in execute six.reraise(sys.exc_info()) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/six.py", line 719, in reraise raise value File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal_api.py", line 141, in execute return self.client.execute(args, kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute result = self._get_result(document, *args, *kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in _get_result return self.transport.execute(document, args, **kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 39, in execute request.raise_for_status() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/requests/models.py", line 960, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.wandb.ai/graphql

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/apis/normalize.py", line 24, in wrapper return func(*args, kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal_api.py", line 1298, in upsert_run response = self.gql(mutation, variable_values=variable_values, kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/lib/retry.py", line 118, in call if not check_retry_fn(e): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/util.py", line 821, in no_retry_auth raise CommError("Permission denied, ask the project owner to grant you access") wandb.errors.CommError: Permission denied, ask the project owner to grant you access

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal_util.py", line 52, in run self._run() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal_util.py", line 102, in _run self._process(record) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal.py", line 291, in _process self._sm.send(record) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/sender.py", line 236, in send send_handler(record) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/sender.py", line 687, in send_run self._init_run(run, config_value_dict) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/sender.py", line 725, in _init_run commit=run.git.last_commit or None, File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/apis/normalize.py", line 62, in wrapper six.reraise(CommError, CommError(message, err), sys.exc_info()[2]) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/six.py", line 718, in reraise raise value.with_traceback(tb) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/apis/normalize.py", line 24, in wrapper return func(*args, kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal_api.py", line 1298, in upsert_run response = self.gql(mutation, variable_values=variable_values, kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/lib/retry.py", line 118, in call if not check_retry_fn(e): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/util.py", line 821, in no_retry_auth raise CommError("Permission denied, ask the project owner to grant you access") wandb.errors.CommError: Permission denied, ask the project owner to grant you access wandb: ERROR Internal wandb error: file data was not synced Problem at: run.py 53 main Traceback (most recent call last): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 931, in init run = wi.init() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 596, in init backend.cleanup() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/backend/backend.py", line 245, in cleanup self.interface.join() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 458, in join super().join() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface.py", line 599, in join = self._communicate_shutdown() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 455, in _communicateshutdown = self._communicate(record) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 213, in _communicate return self._communicate_async(rec, local=local).get(timeout=timeout) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 218, in _communicate_async raise Exception("The wandb backend process has shutdown") Exception: The wandb backend process has shutdown wandb: ERROR Abnormal program exit Traceback (most recent call last): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 931, in init run = wi.init() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 596, in init backend.cleanup() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/backend/backend.py", line 245, in cleanup self.interface.join() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 458, in join super().join() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface.py", line 599, in join = self._communicate_shutdown() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 455, in _communicateshutdown = self._communicate(record) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 213, in _communicate return self._communicate_async(rec, local=local).get(timeout=timeout) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 218, in _communicate_async raise Exception("The wandb backend process has shutdown") Exception: The wandb backend process has shutdown

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "run.py", line 70, in main() File "run.py", line 53, in main wandb_run = wandb.init(**wandb_args) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 969, in init six.raise_from(Exception("problem"), error_seen) File "", line 3, in raise_from Exception: problem`

Could you please guide me how to resolve this issue with 403 error or how to disable wandb logging?

Thank you.

gchesnokov commented 2 years ago

Hi! I guess you can just omit any of the wandb args and then choose Don't visualize my results in the interactive menu.

(.pytorch) ➜  sdc git:(main) ✗ python run.py
/Users/gerrok/uncertainty-challenge/sdc/ysdc_dataset_api/utils/transform.py:90: NumbaPerformanceWarning: '@' is faster on contiguous arrays, called on (array(float32, 2d, A), array(float32, 2d, C))
  res = transform @ ph
/Users/gerrok/.pytorch/lib/python3.9/site-packages/numba/core/typing/npydecl.py:937: NumbaPerformanceWarning: '@' is faster on contiguous arrays, called on (array(float32, 2d, A), array(float32, 2d, C))
  warnings.warn(NumbaPerformanceWarning(msg))
Running model on CPU.
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: 3
wandb: You chose 'Don't visualize my results'
wandb: W&B syncing is set to `offline` in this directory.  Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
Using 4 workers in PyTorch dataloading.
Training Behavioral Cloning
Model kwargs:
{'bc_deterministic': False,
 'device': 'cpu',
 'dim_hidden': 512,
 'generation_mode': 'sampling',
 'in_channels': 17,
 'output_shape': [25, 2]}
Downloading: "https://github.com/pytorch/vision/archive/v0.9.0.zip" to data/torch_hub/v0.9.0.zip
BC Model: using Gaussian likelihood.
BC Model: using generation mode sampling.

Loading train dataset(s).
Loading dataset moscow__train.
owoshch commented 2 years ago

Thank you!