search

OpenLLMAI / OpenRLHF

An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & Mixtral)
https://openrlhf.readthedocs.io/
Apache License 2.0
1.71k stars 160 forks source link

HTTPError when running train_ppo_llama_ray.sh #290

Closed Zeyuan-Liu closed 1 day ago

Zeyuan-Liu commented 1 month ago

What happened + What you expected to happen:

Operation process: ray start --head --node-ip-address 0.0.0.0 --num-gpus 8

Success start head：

Usage stats collection is enabled. To disable this, add --disable-usage-stats to the command that starts the cluster, or run the following command: ray disable-usage-stats before starting the cluster. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details.

Local node IP: 0.0.0.0

Ray runtime started.

Next steps To add another node to this Ray cluster, run ray start --address='0.0.0.0:6379'

To connect to this Ray cluster: import ray ray.init(_node_ip_address='0.0.0.0')

To submit a Ray job using the Ray Jobs CLI: RAY_ADDRESS='http://127.0.0.1:8265' ray job submit --working-dir . -- python my_script.py

See https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html for more information on submitting Ray jobs to the Ray cluster.

To terminate the Ray runtime, run ray stop

To view the status of the cluster, use ray status

To monitor and debug Ray, view the dashboard at 127.0.0.1:8265

If connection to the dashboard fails, check your firewall settings and network configuration.

My Configuration

set -x export PATH=$HOME/.local/bin/:$PATH

ray job submit --address="http://127.0.0.1:8265" \ --runtime-env-json='{"working_dir": "/openrlhf", "pip": "/openrlhf/requirements.txt"}' \ -- python3 examples/train_ppo_ray.py \ --ref_num_nodes 1 \ --ref_num_gpus_per_node 1 \ --reward_num_nodes 1 \ --reward_num_gpus_per_node 1 \ --critic_num_nodes 1 \ --critic_num_gpus_per_node 2 \ --actor_num_nodes 1 \ --actor_num_gpus_per_node 4 \ --pretrain OpenLLMAI/Llama-2-7b-sft-model-ocra-500k \ --reward_pretrain OpenLLMAI/Llama-2-7b-rm-anthropic_hh-lmsys-oasst-webgpt \ --save_path /openrlhf/examples/test_scripts/ckpt/7b_llama \ --micro_train_batch_size 8 \ --train_batch_size 128 \ --micro_rollout_batch_size 16 \ --rollout_batch_size 1024 \ --max_epochs 1 \ --prompt_max_len 1024 \ --generate_max_len 1024 \ --zero_stage 2 \ --bf16 \ --actor_learning_rate 5e-7 \ --critic_learning_rate 9e-6 \ --init_kl_coef 0.01 \ --prompt_data Open-Orca/OpenOrca \ --prompt_data_probs 1 \ --max_samples 80000 \ --normalize_reward \ --actor_init_on_gpu \ --adam_offload \ --flash_attn \ --gradient_checkpointing \ --use_wandb {wandb_token}

Error Information

Traceback (most recent call last): File "/root/miniconda3/envs/lzy/bin/ray", line 33, in sys.exit(load_entry_point('ray==2.12.0', 'console_scripts', 'ray')()) File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/ray/scripts/scripts.py", line 2612, in main return cli() File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(args, kwargs) File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/ray/dashboard/modules/job/cli_utils.py", line 54, in wrapper return func(*args, *kwargs) File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/ray/autoscaler/_private/cli_logger.py", line 856, in wrapper return f(args, kwargs) File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/ray/dashboard/modules/job/cli.py", line 264, in submit client = _get_sdk_client( File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/ray/dashboard/modules/job/cli.py", line 29, in _get_sdk_client client = JobSubmissionClient( File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/ray/dashboard/modules/job/sdk.py", line 109, in init self._check_connection_and_version( File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/ray/dashboard/modules/dashboard_sdk.py", line 248, in _check_connection_and_version self._check_connection_and_version_with_url(min_version, version_error_message) File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/ray/dashboard/modules/dashboard_sdk.py", line 267, in _check_connection_and_version_with_url r.raise_for_status() File "/root/miniconda3/envs/lzy/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: http://127.0.0.1:8265/api/version

Zeyuan-Liu commented 1 month ago

Environment:

Python 3.10.14

Package Version

accelerate 0.29.3 aiohttp 3.9.5 aiohttp-cors 0.7.0 aiosignal 1.3.1 annotated-types 0.6.0 anyio 4.2.0 appdirs 1.4.4 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0 asttokens 2.0.5 async-lru 2.0.4 async-timeout 4.0.3 attrs 23.1.0 Babel 2.11.0 beautifulsoup4 4.12.2 bitsandbytes 0.43.1 bleach 4.1.0 Brotli 1.0.9 cachetools 5.3.3 certifi 2024.2.2 cffi 1.16.0 charset-normalizer 2.0.4 click 8.1.7 coloredlogs 15.0.1 colorful 0.5.6 comm 0.2.1 datasets 2.19.0 debugpy 1.6.7 decorator 5.1.1 deepspeed 0.13.2 defusedxml 0.7.1 dill 0.3.8 distlib 0.3.8 docker-pycreds 0.4.0 einops 0.7.0 exceptiongroup 1.2.0 executing 0.8.3 fastjsonschema 2.16.2 filelock 3.13.1 flash-attn 2.4.2 frozenlist 1.4.1 fsspec 2024.3.1 gitdb 4.0.11 GitPython 3.1.43 gmpy2 2.1.2 google-api-core 2.18.0 google-auth 2.29.0 googleapis-common-protos 1.63.0 grpcio 1.62.2 hjson 3.1.0 huggingface-hub 0.22.2 humanfriendly 10.0 idna 3.4 ipykernel 6.28.0 ipython 8.20.0 ipywidgets 8.1.2 isort 5.13.2 jedi 0.18.1 Jinja2 3.1.3 json5 0.9.6 jsonlines 4.0.0 jsonschema 4.19.2 jsonschema-specifications 2023.12.1 jupyter 1.0.0 jupyter_client 8.6.0 jupyter-console 6.6.3 jupyter_core 5.5.0 jupyter-events 0.8.0 jupyter-lsp 2.2.0 jupyter_server 2.10.0 jupyter_server_terminals 0.4.4 jupyterlab 4.0.11 jupyterlab-pygments 0.1.2 jupyterlab_server 2.25.1 jupyterlab-widgets 3.0.10 lightning-utilities 0.11.2 linkify-it-py 2.0.3 loralib 0.1.2 markdown-it-py 3.0.0 MarkupSafe 2.1.3 matplotlib-inline 0.1.6 mdit-py-plugins 0.4.0 mdurl 0.1.2 memray 1.12.0 mistune 2.0.4 mkl-fft 1.3.8 mkl-random 1.2.4 mkl-service 2.4.0 mpi4py 3.1.4 mpmath 1.3.0 msgpack 1.0.8 multidict 6.0.5 multiprocess 0.70.16 nbclient 0.8.0 nbconvert 7.10.0 nbformat 5.9.2 nest-asyncio 1.6.0 networkx 3.1 ninja 1.11.1.1 notebook 7.0.8 notebook_shim 0.2.3 numpy 1.26.4 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 8.9.2.26 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-nccl-cu12 2.20.5 nvidia-nvjitlink-cu12 12.4.127 nvidia-nvtx-cu12 12.1.105 opencensus 0.11.4 opencensus-context 0.1.3 openrlhf 0.2.6 optimum 1.19.1 overrides 7.4.0 packaging 23.2 pandas 2.2.2 pandocfilters 1.5.0 parso 0.8.3 peft 0.10.0 pexpect 4.8.0 pillow 10.2.0 pip 23.3.1 platformdirs 3.10.0 ply 3.11 prometheus-client 0.14.1 prompt-toolkit 3.0.43 proto-plus 1.23.0 protobuf 4.25.3 psutil 5.9.0 ptyprocess 0.7.0 pure-eval 0.2.2 py-cpuinfo 9.0.0 py-spy 0.3.14 pyarrow 16.0.0 pyarrow-hotfix 0.6 pyasn1 0.6.0 pyasn1_modules 0.4.0 pycparser 2.21 pydantic 2.7.1 pydantic_core 2.18.2 Pygments 2.15.1 pynvml 11.5.0 PyQt5 5.15.10 PyQt5-sip 12.13.0 PySocks 1.7.1 python-dateutil 2.8.2 python-json-logger 2.0.7 pytz 2024.1 PyYAML 6.0.1 pyzmq 25.1.2 qtconsole 5.5.1 QtPy 2.4.1 ray 2.12.0 referencing 0.35.0 regex 2024.4.16 requests 2.31.0 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rich 13.7.1 rpds-py 0.10.6 rsa 4.9 safetensors 0.4.3 Send2Trash 1.8.2 sentencepiece 0.2.0 sentry-sdk 2.0.1 setproctitle 1.3.3 setuptools 68.2.2 sip 6.7.12 six 1.16.0 smart-open 7.0.4 smmap 5.0.1 sniffio 1.3.0 soupsieve 2.5 stack-data 0.2.0 sympy 1.12 terminado 0.17.1 textual 0.58.0 tinycss2 1.2.1 tokenizers 0.15.2 tomli 2.0.1 torch 2.3.0 torchaudio 2.3.0 torchmetrics 1.3.2 torchvision 0.18.0 tornado 6.3.3 tqdm 4.66.2 traitlets 5.7.1 transformers 4.38.2 transformers-stream-generator 0.0.5 triton 2.3.0 typing_extensions 4.11.0 tzdata 2024.1 uc-micro-py 1.0.3 urllib3 2.1.0 virtualenv 20.26.0 wandb 0.16.6 wcwidth 0.2.5 webencodings 0.5.1 websocket-client 0.58.0 wheel 0.41.3 widgetsnbextension 4.0.10 wrapt 1.16.0 xxhash 3.4.1 yarl 1.9.4

hijkzzz commented 1 month ago

could you try the docker container?

Zeyuan-Liu commented 1 month ago

could you try the docker container? When I try to run bash docker_run.sh build, the following issues happens:

Dockerfile:1

1 | >>> FROM nvcr.io/nvidia/pytorch:23.12-py3 2 |
3 | WORKDIR /app

ERROR: failed to solve: nvcr.io/nvidia/pytorch:23.12-py3: failed to resolve source metadata for nvcr.io/nvidia/pytorch:23.12-py3: failed to do request: Head "https://nvcr.io/v2/nvidia/pytorch/manifests/23.12-py3": dial tcp 54.148.47.228:443: connect: network is unreachable

hijkzzz commented 1 month ago

network issue, can you access nvcr.io/nvidia/pytorch:23.12-py3 ?

Zeyuan-Liu commented 1 month ago

When I click on the aforementioned link nvcr.io/nvidia/pytorch:23.12-py3, I can get access to

https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags

but can not open

https://nvcr.io/nvidia/pytorch:23.12-py3