Closed owoshch closed 2 years ago
Hi! I guess you can just omit any of the wandb args and then choose Don't visualize my results
in the interactive menu.
(.pytorch) ➜ sdc git:(main) ✗ python run.py
/Users/gerrok/uncertainty-challenge/sdc/ysdc_dataset_api/utils/transform.py:90: NumbaPerformanceWarning: '@' is faster on contiguous arrays, called on (array(float32, 2d, A), array(float32, 2d, C))
res = transform @ ph
/Users/gerrok/.pytorch/lib/python3.9/site-packages/numba/core/typing/npydecl.py:937: NumbaPerformanceWarning: '@' is faster on contiguous arrays, called on (array(float32, 2d, A), array(float32, 2d, C))
warnings.warn(NumbaPerformanceWarning(msg))
Running model on CPU.
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: 3
wandb: You chose 'Don't visualize my results'
wandb: W&B syncing is set to `offline` in this directory. Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
Using 4 workers in PyTorch dataloading.
Training Behavioral Cloning
Model kwargs:
{'bc_deterministic': False,
'device': 'cpu',
'dim_hidden': 512,
'generation_mode': 'sampling',
'in_channels': 17,
'output_shape': [25, 2]}
Downloading: "https://github.com/pytorch/vision/archive/v0.9.0.zip" to data/torch_hub/v0.9.0.zip
BC Model: using Gaussian likelihood.
BC Model: using generation mode sampling.
Loading train dataset(s).
Loading dataset moscow__train.
Thank you!
Hi! Thank you for your wonderful work!
I'm trying to reproduce the training of the baselines models you provided.
I created the
wandb
accountowoshch
. Within this account, I create the projecttest
. Then I successfully logged in to the account through the command line.I'm running the command
python run.py --dir_data /datasets/YandexShifts/data/ --wandb_project test --verbose True
and getting the following error:wandb: Currently logged in as: owoshch (use
wandb login --relogin` to force relogin) wandb: ERROR Error while calling W&B API: permission denied (<Response [403]>) Thread SenderThread: Traceback (most recent call last): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/lib/retry.py", line 102, in call result = self._call_fn(*args, kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal_api.py", line 147, in execute six.reraise(sys.exc_info()) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/six.py", line 719, in reraise raise value File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal_api.py", line 141, in execute return self.client.execute(args, kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute result = self._get_result(document, *args, *kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in _get_result return self.transport.execute(document, args, **kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 39, in execute request.raise_for_status() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/requests/models.py", line 960, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.wandb.ai/graphqlDuring handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/apis/normalize.py", line 24, in wrapper return func(*args, kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal_api.py", line 1298, in upsert_run response = self.gql(mutation, variable_values=variable_values, kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/lib/retry.py", line 118, in call if not check_retry_fn(e): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/util.py", line 821, in no_retry_auth raise CommError("Permission denied, ask the project owner to grant you access") wandb.errors.CommError: Permission denied, ask the project owner to grant you access
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal_util.py", line 52, in run self._run() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal_util.py", line 102, in _run self._process(record) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal.py", line 291, in _process self._sm.send(record) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/sender.py", line 236, in send send_handler(record) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/sender.py", line 687, in send_run self._init_run(run, config_value_dict) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/sender.py", line 725, in _init_run commit=run.git.last_commit or None, File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/apis/normalize.py", line 62, in wrapper six.reraise(CommError, CommError(message, err), sys.exc_info()[2]) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/six.py", line 718, in reraise raise value.with_traceback(tb) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/apis/normalize.py", line 24, in wrapper return func(*args, kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/internal/internal_api.py", line 1298, in upsert_run response = self.gql(mutation, variable_values=variable_values, kwargs) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/lib/retry.py", line 118, in call if not check_retry_fn(e): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/util.py", line 821, in no_retry_auth raise CommError("Permission denied, ask the project owner to grant you access") wandb.errors.CommError: Permission denied, ask the project owner to grant you access wandb: ERROR Internal wandb error: file data was not synced Problem at: run.py 53 main Traceback (most recent call last): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 931, in init run = wi.init() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 596, in init backend.cleanup() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/backend/backend.py", line 245, in cleanup self.interface.join() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 458, in join super().join() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface.py", line 599, in join = self._communicate_shutdown() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 455, in _communicateshutdown = self._communicate(record) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 213, in _communicate return self._communicate_async(rec, local=local).get(timeout=timeout) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 218, in _communicate_async raise Exception("The wandb backend process has shutdown") Exception: The wandb backend process has shutdown wandb: ERROR Abnormal program exit Traceback (most recent call last): File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 931, in init run = wi.init() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 596, in init backend.cleanup() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/backend/backend.py", line 245, in cleanup self.interface.join() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 458, in join super().join() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface.py", line 599, in join = self._communicate_shutdown() File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 455, in _communicateshutdown = self._communicate(record) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 213, in _communicate return self._communicate_async(rec, local=local).get(timeout=timeout) File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/interface/interface_shared.py", line 218, in _communicate_async raise Exception("The wandb backend process has shutdown") Exception: The wandb backend process has shutdown
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "run.py", line 70, in
main()
File "run.py", line 53, in main
wandb_run = wandb.init(**wandb_args)
File "/home/fkitashov/anaconda3/envs/densetnt_env_py37_torch16/lib/python3.7/site-packages/wandb/sdk/wandb_init.py", line 969, in init
six.raise_from(Exception("problem"), error_seen)
File "", line 3, in raise_from
Exception: problem`
Could you please guide me how to resolve this issue with
403 error
or how to disable wandb logging?Thank you.