time-series-foundation-models / lag-llama

Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting
Apache License 2.0
1.08k stars 121 forks source link

help with wandb permission access #74

Closed kenadianu closed 2 weeks ago

kenadianu commented 2 weeks ago

Hello,

Trying to run the project on Windows

Launching ./scripts/pretrain.sh in the GitBash console leads to 'W&B API: permission denied'.

The modified instruction in line 59 (of pretrain.sh) is

    C:/Users/user/AppData/Local/Programs/Python/Python311/python.exe G://noi5/LagLlama-timeSeries.github/lag-llama-main/run.py \
 -e $EXP_NAME -d "datasets" --seed $SEED \
    -r "experiments/results" \
    --batch_size 512 -m 1000 -n 128 \
    --wandb_entity "drad1907" --wandb_project "lag-llama-test"  \
--all_datasets  "cpu_limit_minute" "cpu_usage_minute" "function_delay_minute" "instances_minute" "memory_limit_minute" "memory_usage_minute" "platform_delay_minute" "requests_minute" "ett_h1" "ett_h2" "ett_m1" "ett_m2" "beijing_pm25" "AirQualityUCI" "beijing_multisite"  \
    --test_datasets "ett_m2" "platform_delay_minute" "requests_minute" "beijing_pm25" \
    --num_workers 2 --args_from_dict_path $CONFIGPATH --search_batch_size \
    --lr 0.0001

Excerpt from console:

$ ./scripts/pretrain.sh
[some output on console]
Seed set to 11605
experiments/results\pretraining_lag_llama\11605 already exists.
No checkpoints found. Training from scratch.
Training datasets: ['cpu_limit_minute', 'cpu_usage_minute', 'function_delay_minute', 'instances_minute', 'memory_limit_minute', 'memory_usage_minute', 'ett_h1'
, 'ett_h2', 'ett_m1', 'AirQualityUCI', 'beijing_multisite']
Test datasets: ['ett_m2', 'platform_delay_minute', 'requests_minute', 'beijing_pm25']

wandb: **Currently logged in as**: drad1907. Use `wandb login --relogin` to force relogin
wandb: ERROR Error while calling W&B API: permission denied (<Response [403]>)
wandb: ERROR It appears that you do not have permission to access the requested resource. 
Please reach out to the project owner to grant you access. If you have the correct permissions, 
verify that there are no issues with your networking setup.(Error 403: Forbidden)

Tried this fix without success https://stackoverflow.com/questions/64093979/weights-and-biases-login-and-network-errors

Below is an excerpt from the lag_llama log file G:\noi5\lag-llama-main\experiments\results\pretraining_lag_llama\11605\wandb\run-20240605_163036-c87122a8\logs**debug.log**

Loading settings from C:\Users\user\.config\wandb\settings
Loading settings from G:\noi5\LagLlama-timeSeries.github\lag-llama-main\wandb\settings

Unknown environment variable: WANDB_CACHE_DIR
Unknown environment variable: WANDB_CONFIG_DIR

Loading settings from environment variables: {'api_key': '***REDACTED***', 'root_dir': 'G:\\noi5\\LagLlama-timeSeries.github\\lag-llama-main'}

[wandb_init.py:init():567] wandb.init called with sweep_config: {}
[wandb_init.py:init():610] starting backend
[wandb_init.py:init():622] backend started and connected
[wandb_init.py:init():744] communicating run to backend with 90.0 second timeout
[wandb_init.py:init():770] encountered error: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)

What is the 'requested resource' the message refers to? The project "lag-llama-test" exists on wandb and I was already logged into my wandb account, as we can see from the log.

Any suggestions on how to get past this permission access? Thank you.

------- On a non-related note ----------

ashok-arjun commented 2 weeks ago

Hi! Can you try also using --wandb_entity noi5 in the script please? It seems from the following line that W&B automatically uses your default account: wandb: **Currently logged in as**: drad1907. Usewandb login --reloginto force relogin

Also, have you downloaded the datasets?

And thank you for summarising the updates regarding the python version and package

kenadianu commented 2 weeks ago

Thank you for your solution!

Permission access granted when using

--wandb_entity "noi5" --wandb_project "lag-llama-test"

instead of

--wandb_entity "drad1907" --wandb_project "lag-llama-test"

in pretrain.sh

Console output:

wandb: Currently logged in as: drad1907 (noi5). Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.17.0
wandb: Run data is saved locally in experiments/results\pretraining_lag_llama\11605\wandb\run-20240606_164839-c87122a8
wandb: Run `wandb offline` to turn off syncing.
wandb: Resuming run pretraining_lag_llama-seed-11605
wandb:  View project at https://wandb.ai/noi5/lag-llama-test
wandb:  View run at https://wandb.ai/noi5/lag-llama-test/runs/c87122a8