search

lean-dojo / ReProver

Retrieval-Augmented Theorem Provers for Lean
https://leandojo.org
MIT License
229 stars 53 forks source link

"ValueError: Tensors must be contiguous" when training ReProver (CUDA 12.4) #66

Closed realharryhero closed 3 months ago

realharryhero commented 3 months ago

Hello,

When I follow all the instructions (including setting up a conda environment), I recieve the following error:

Log

``` [2024-07-26 19:20:55,224] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [WARNING] async_io requires the dev libaio .so object and headers but these were not found. [WARNING] async_io: please install the libaio-dev package with apt [WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible 2024-07-26 19:20:55.838 | INFO | __main__:main:19 - PID: 108346 Seed set to 3407 2024-07-26 19:20:56.872 | INFO | common:__init__:200 - Building the corpus from data/leandojo_benchmark_4/corpus.jsonl 2024-07-26 19:21:10.642 | INFO | generation.datamodule:__init__:147 - Without retrieval data GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores HPU available: False, using: 0 HPUs [rank: 0] Seed set to 3407 initializing deepspeed distributed: GLOBAL_RANK: 0, MEMBER: 1/1 wandb: Currently logged in as: [REDACTED] ([REDACTED]). Use `wandb login --relogin` to force relogin wandb: WARNING Path logs/train_generator_novel_premises/wandb/ wasn't writable, using system temp directory. wandb: WARNING Path logs/train_generator_novel_premises/wandb/ wasn't writable, using system temp directory wandb: Tracking run with wandb version 0.17.5 wandb: Run data is saved locally in /tmp/wandb/run-20240726_192111-iykqoyqi wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run train_generator_novel_premises wandb: โญ๏ธ View project at https://wandb.ai/[REDACTED]/lightning_logs wandb: ๐Ÿš€ View run at https://wandb.ai/[REDACTED]/lightning_logs/runs/iykqoyqi 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 118517/118517 [00:00<00:00, 574149.48it/s] 2024-07-26 19:21:21.894 | INFO | generation.datamodule:_load_data:60 - 246714 examples loaded 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 2000/2000 [00:00<00:00, 527088.16it/s] 2024-07-26 19:21:21.956 | INFO | generation.datamodule:_load_data:60 - 6240 examples loaded Enabling DeepSpeed BF16. Model parameters and inputs will be cast to `bfloat16`. LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2024-07-26 19:21:22.041 | INFO | common:get_optimizers:392 - Optimizing with FusedAdam Using /home/mfan/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /home/mfan/.cache/torch_extensions/py310_cu121/fused_adam/build.ninja... /home/mfan/ReProver/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. warnings.warn( Building extension module fused_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/2] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output multi_tensor_adam.cuda.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/ops/csrc/adam -isystem /home/mfan/ReProver/lib/python3.10/site-packages/torch/include -isystem /home/mfan/ReProver/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/mfan/ReProver/lib/python3.10/site-packages/torch/include/TH -isystem /home/mfan/ReProver/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -lineinfo --use_fast_math -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_86,code=compute_86 -DBF16_AVAILABLE -U__CUDA_NO_BFLOAT16_OPERATORS__ -U__CUDA_NO_BFLOAT162_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ -std=c++17 -c /home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu -o multi_tensor_adam.cuda.o [2/2] c++ fused_adam_frontend.o multi_tensor_adam.cuda.o -shared -L/home/mfan/ReProver/lib/python3.10/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o fused_adam.so Loading extension module fused_adam... Time to load fused_adam op: 14.073747634887695 seconds Traceback (most recent call last): File "/home/mfan/ReProver/generation/main.py", line 25, in main() File "/home/mfan/ReProver/generation/main.py", line 20, in main cli = CLI(RetrievalAugmentedGenerator, GeneratorDataModule) File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/cli.py", line 394, in __init__ self._run_subcommand(self.subcommand) File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/cli.py", line 701, in _run_subcommand fn(**fn_kwargs) File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 543, in fit call._call_and_handle_interrupt( File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs) File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 105, in launch return function(*args, **kwargs) File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl self._run(model, ckpt_path=ckpt_path) File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in _run self.strategy.setup(self) File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/strategies/deepspeed.py", line 350, in setup self.init_deepspeed() File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/strategies/deepspeed.py", line 451, in init_deepspeed self._initialize_deepspeed_train(self.model) File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/strategies/deepspeed.py", line 487, in _initialize_deepspeed_train model, deepspeed_optimizer = self._setup_model_and_optimizer(model, optimizer, scheduler) File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/strategies/deepspeed.py", line 423, in _setup_model_and_optimizer deepspeed_engine, deepspeed_optimizer, _, _ = deepspeed.initialize( File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/__init__.py", line 181, in initialize engine = DeepSpeedEngine(args=args, File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 262, in __init__ self._configure_distributed_model(model) File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1148, in _configure_distributed_model self._broadcast_model() File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1068, in _broadcast_model dist.broadcast(p.data, groups._get_broadcast_src_rank(), group=self.seq_data_parallel_group) File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/comm/comm.py", line 117, in log_wrapper return func(*args, **kwargs) File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/comm/comm.py", line 224, in broadcast return cdb.broadcast(tensor=tensor, src=src, group=group, async_op=async_op) File "/home/mfan/ReProver/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn return fn(*args, **kwargs) File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 199, in broadcast return torch.distributed.broadcast(tensor=tensor, src=src, group=group, async_op=async_op) File "/home/mfan/ReProver/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper return func(*args, **kwargs) File "/home/mfan/ReProver/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2140, in broadcast work = group.broadcast([tensor], opts) ValueError: Tensors must be contiguous [rank0]: Traceback (most recent call last): [rank0]: File "/home/mfan/ReProver/generation/main.py", line 25, in [rank0]: main() [rank0]: File "/home/mfan/ReProver/generation/main.py", line 20, in main [rank0]: cli = CLI(RetrievalAugmentedGenerator, GeneratorDataModule) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/cli.py", line 394, in __init__ [rank0]: self._run_subcommand(self.subcommand) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/cli.py", line 701, in _run_subcommand [rank0]: fn(**fn_kwargs) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 543, in fit [rank0]: call._call_and_handle_interrupt( [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt [rank0]: return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 105, in launch [rank0]: return function(*args, **kwargs) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl [rank0]: self._run(model, ckpt_path=ckpt_path) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in _run [rank0]: self.strategy.setup(self) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/strategies/deepspeed.py", line 350, in setup [rank0]: self.init_deepspeed() [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/strategies/deepspeed.py", line 451, in init_deepspeed [rank0]: self._initialize_deepspeed_train(self.model) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/strategies/deepspeed.py", line 487, in _initialize_deepspeed_train [rank0]: model, deepspeed_optimizer = self._setup_model_and_optimizer(model, optimizer, scheduler) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/pytorch_lightning/strategies/deepspeed.py", line 423, in _setup_model_and_optimizer [rank0]: deepspeed_engine, deepspeed_optimizer, _, _ = deepspeed.initialize( [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/__init__.py", line 181, in initialize [rank0]: engine = DeepSpeedEngine(args=args, [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 262, in __init__ [rank0]: self._configure_distributed_model(model) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1148, in _configure_distributed_model [rank0]: self._broadcast_model() [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1068, in _broadcast_model [rank0]: dist.broadcast(p.data, groups._get_broadcast_src_rank(), group=self.seq_data_parallel_group) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/comm/comm.py", line 117, in log_wrapper [rank0]: return func(*args, **kwargs) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/comm/comm.py", line 224, in broadcast [rank0]: return cdb.broadcast(tensor=tensor, src=src, group=group, async_op=async_op) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn [rank0]: return fn(*args, **kwargs) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/deepspeed/comm/torch.py", line 199, in broadcast [rank0]: return torch.distributed.broadcast(tensor=tensor, src=src, group=group, async_op=async_op) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper [rank0]: return func(*args, **kwargs) [rank0]: File "/home/mfan/ReProver/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2140, in broadcast [rank0]: work = group.broadcast([tensor], opts) [rank0]: ValueError: Tensors must be contiguous wandb: ๐Ÿš€ View run train_generator_novel_premises at: https://wandb.ai/[REDACTED]/lightning_logs/runs/iykqoyqi wandb: โญ๏ธ View project at: https://wandb.ai/[REDACTED]/lightning_logs wandb: Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s) wandb: Find logs at: /tmp/wandb/run-20240726_192111-iykqoyqi/logs wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with `wandb.require("core")`! See https://wandb.me/wandb-core for more information. ```

I'm using CUDA 12.4, so that could be an issue. Otherwise, a result on StackOverflow seems to summarize what's occurring. If so, where would the .contiguous() function go? Thanks!

yangky11 commented 3 months ago

What was the command that triggered this error?

realharryhero commented 3 months ago

python generation/main.py fit --config generation/confs/cli_lean4_novel_premises.yaml --trainer.logger.name train_generator_novel_premises --trainer.logger.save_dir logs/train_generator_novel_premises # LeanDojo Benchmark 4, `novel_premises` split

yangky11 commented 3 months ago

I wasn't able to reproduce the error. From the logs, it looks like the error is in PyTorch Lightning or DeepSpeed. So I don't think adding a . contiguous() in ReProver would actually help. Have you tried creating a new conda environment and re-install those dependencies?

realharryhero commented 3 months ago

The error still remains, even when using CUDA 12.1 (and python 3.11). It does seem to be a common issue with DeepSpeed according to this link, but I'm still not sure why it then only doesn't work for me. The full logs, after installing PyTorch but before installing DeepSpeed, are below:

Logs

``` (ReProver) mfan@localhost:~/ReProver$ pip install tqdm loguru deepspeed "pytorch-lightning[extra]" transformers wandb openai rank_bm25 lean-dojo vllm Collecting tqdm Using cached tqdm-4.66.4-py3-none-any.whl.metadata (57 kB) Collecting loguru Using cached loguru-0.7.2-py3-none-any.whl.metadata (23 kB) Collecting deepspeed Using cached deepspeed-0.14.4-py3-none-any.whl Collecting transformers Using cached transformers-4.43.3-py3-none-any.whl.metadata (43 kB) Collecting wandb Using cached wandb-0.17.5-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (10 kB) Collecting openai Using cached openai-1.37.1-py3-none-any.whl.metadata (22 kB) Collecting rank_bm25 Using cached rank_bm25-0.2.2-py3-none-any.whl.metadata (3.2 kB) Collecting lean-dojo Using cached lean_dojo-2.0.3-py3-none-any.whl.metadata (7.9 kB) Collecting vllm Using cached vllm-0.5.3.post1-cp38-abi3-manylinux1_x86_64.whl.metadata (1.8 kB) Collecting pytorch-lightning[extra] Using cached pytorch_lightning-2.3.3-py3-none-any.whl.metadata (21 kB) Collecting hjson (from deepspeed) Using cached hjson-3.1.0-py3-none-any.whl.metadata (2.6 kB) Collecting ninja (from deepspeed) Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB) Requirement already satisfied: numpy in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from deepspeed) (2.0.1) Collecting nvidia-ml-py (from deepspeed) Using cached nvidia_ml_py-12.555.43-py3-none-any.whl.metadata (8.6 kB) Collecting packaging>=20.0 (from deepspeed) Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB) Collecting psutil (from deepspeed) Using cached psutil-6.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB) Collecting py-cpuinfo (from deepspeed) Using cached py_cpuinfo-9.0.0-py3-none-any.whl.metadata (794 bytes) Collecting pydantic (from deepspeed) Using cached pydantic-2.8.2-py3-none-any.whl.metadata (125 kB) Requirement already satisfied: torch in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from deepspeed) (2.4.0) Collecting PyYAML>=5.4 (from pytorch-lightning[extra]) Using cached PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB) Requirement already satisfied: fsspec>=2022.5.0 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from fsspec[http]>=2022.5.0->pytorch-lightning[extra]) (2024.6.1) Collecting torchmetrics>=0.7.0 (from pytorch-lightning[extra]) Using cached torchmetrics-1.4.0.post0-py3-none-any.whl.metadata (19 kB) Requirement already satisfied: typing-extensions>=4.4.0 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from pytorch-lightning[extra]) (4.11.0) Collecting lightning-utilities>=0.10.0 (from pytorch-lightning[extra]) Using cached lightning_utilities-0.11.6-py3-none-any.whl.metadata (5.2 kB) Collecting matplotlib>3.1 (from pytorch-lightning[extra]) Using cached matplotlib-3.9.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB) Collecting omegaconf>=2.2.3 (from pytorch-lightning[extra]) Using cached omegaconf-2.3.0-py3-none-any.whl.metadata (3.9 kB) Collecting hydra-core>=1.2.0 (from pytorch-lightning[extra]) Using cached hydra_core-1.3.2-py3-none-any.whl.metadata (5.5 kB) Collecting jsonargparse>=4.27.7 (from jsonargparse[signatures]>=4.27.7; extra == "extra"->pytorch-lightning[extra]) Using cached jsonargparse-4.32.0-py3-none-any.whl.metadata (12 kB) Collecting rich>=12.3.0 (from pytorch-lightning[extra]) Using cached rich-13.7.1-py3-none-any.whl.metadata (18 kB) Collecting tensorboardX>=2.2 (from pytorch-lightning[extra]) Using cached tensorboardX-2.6.2.2-py2.py3-none-any.whl.metadata (5.8 kB) Collecting bitsandbytes>=0.42.0 (from pytorch-lightning[extra]) Downloading bitsandbytes-0.43.3-py3-none-manylinux_2_24_x86_64.whl.metadata (3.5 kB) Requirement already satisfied: filelock in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from transformers) (3.15.4) Collecting huggingface-hub<1.0,>=0.23.2 (from transformers) Downloading huggingface_hub-0.24.5-py3-none-any.whl.metadata (13 kB) Collecting regex!=2019.12.17 (from transformers) Using cached regex-2024.7.24-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB) Collecting requests (from transformers) Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB) Collecting safetensors>=0.4.1 (from transformers) Using cached safetensors-0.4.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB) Collecting tokenizers<0.20,>=0.19 (from transformers) Using cached tokenizers-0.19.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB) Collecting click!=8.0.0,>=7.1 (from wandb) Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB) Collecting docker-pycreds>=0.4.0 (from wandb) Using cached docker_pycreds-0.4.0-py2.py3-none-any.whl.metadata (1.8 kB) Collecting gitpython!=3.1.29,>=1.0.0 (from wandb) Using cached GitPython-3.1.43-py3-none-any.whl.metadata (13 kB) Collecting platformdirs (from wandb) Using cached platformdirs-4.2.2-py3-none-any.whl.metadata (11 kB) Collecting protobuf!=4.21.0,<6,>=3.19.0 (from wandb) Using cached protobuf-5.27.2-cp38-abi3-manylinux2014_x86_64.whl.metadata (592 bytes) Collecting sentry-sdk>=1.0.0 (from wandb) Downloading sentry_sdk-2.12.0-py2.py3-none-any.whl.metadata (9.8 kB) Collecting setproctitle (from wandb) Using cached setproctitle-1.3.3-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.9 kB) Requirement already satisfied: setuptools in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from wandb) (69.5.1) Collecting anyio<5,>=3.5.0 (from openai) Using cached anyio-4.4.0-py3-none-any.whl.metadata (4.6 kB) Collecting distro<2,>=1.7.0 (from openai) Using cached distro-1.9.0-py3-none-any.whl.metadata (6.8 kB) Collecting httpx<1,>=0.23.0 (from openai) Using cached httpx-0.27.0-py3-none-any.whl.metadata (7.2 kB) Collecting sniffio (from openai) Using cached sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB) Collecting lxml (from lean-dojo) Using cached lxml-5.2.2-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (3.4 kB) Requirement already satisfied: networkx in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from lean-dojo) (3.3) Requirement already satisfied: pexpect in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from lean-dojo) (4.8.0) Collecting pygithub (from lean-dojo) Using cached PyGithub-2.3.0-py3-none-any.whl.metadata (3.8 kB) Collecting python-dotenv (from lean-dojo) Using cached python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB) Collecting ray>=2.8 (from ray[default]>=2.8->lean-dojo) Downloading ray-2.34.0-cp311-cp311-manylinux2014_x86_64.whl.metadata (13 kB) Collecting toml (from lean-dojo) Using cached toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) Collecting types-psutil (from lean-dojo) Using cached types_psutil-6.0.0.20240621-py3-none-any.whl.metadata (1.6 kB) Collecting types-toml (from lean-dojo) Using cached types_toml-0.10.8.20240310-py3-none-any.whl.metadata (1.5 kB) Collecting cmake>=3.21 (from vllm) Using cached cmake-3.30.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.1 kB) Collecting sentencepiece (from vllm) Using cached sentencepiece-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB) Collecting numpy (from deepspeed) Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB) โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 61.0/61.0 kB 2.8 MB/s eta 0:00:00 Collecting fastapi (from vllm) Using cached fastapi-0.111.1-py3-none-any.whl.metadata (26 kB) Collecting aiohttp (from vllm) Downloading aiohttp-3.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.5 kB) Collecting uvicorn[standard] (from vllm) Using cached uvicorn-0.30.3-py3-none-any.whl.metadata (6.5 kB) Requirement already satisfied: pillow in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from vllm) (10.4.0) Collecting prometheus-client>=0.18.0 (from vllm) Using cached prometheus_client-0.20.0-py3-none-any.whl.metadata (1.8 kB) Collecting prometheus-fastapi-instrumentator>=7.0.0 (from vllm) Using cached prometheus_fastapi_instrumentator-7.0.0-py3-none-any.whl.metadata (13 kB) Collecting tiktoken>=0.6.0 (from vllm) Using cached tiktoken-0.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB) Collecting lm-format-enforcer==0.10.3 (from vllm) Using cached lm_format_enforcer-0.10.3-py3-none-any.whl.metadata (16 kB) Collecting outlines<0.1,>=0.0.43 (from vllm) Using cached outlines-0.0.46-py3-none-any.whl.metadata (15 kB) Collecting pyzmq (from vllm) Using cached pyzmq-26.0.3-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (6.1 kB) Collecting torch (from deepspeed) Using cached torch-2.3.1-cp311-cp311-manylinux1_x86_64.whl.metadata (26 kB) Collecting torchvision==0.18.1 (from vllm) Using cached torchvision-0.18.1-cp311-cp311-manylinux1_x86_64.whl.metadata (6.6 kB) Collecting xformers==0.0.27 (from vllm) Using cached xformers-0.0.27-cp311-cp311-manylinux2014_x86_64.whl.metadata (1.0 kB) Collecting vllm-flash-attn==2.5.9.post1 (from vllm) Using cached vllm_flash_attn-2.5.9.post1-cp311-cp311-manylinux1_x86_64.whl.metadata (482 bytes) Collecting interegular>=0.3.2 (from lm-format-enforcer==0.10.3->vllm) Using cached interegular-0.3.3-py37-none-any.whl.metadata (3.0 kB) Requirement already satisfied: sympy in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from torch->deepspeed) (1.13.1) Requirement already satisfied: jinja2 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from torch->deepspeed) (3.1.4) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from torch->deepspeed) (12.1.105) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from torch->deepspeed) (12.1.105) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from torch->deepspeed) (12.1.105) Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch->deepspeed) Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from torch->deepspeed) (12.1.3.1) Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from torch->deepspeed) (11.0.2.54) Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from torch->deepspeed) (10.3.2.106) Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from torch->deepspeed) (11.4.5.107) Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from torch->deepspeed) (12.1.0.106) Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from torch->deepspeed) (2.20.5) Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from torch->deepspeed) (12.1.105) Collecting triton==2.3.1 (from torch->deepspeed) Using cached triton-2.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.4 kB) Requirement already satisfied: nvidia-nvjitlink-cu12 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch->deepspeed) (12.5.82) Collecting idna>=2.8 (from anyio<5,>=3.5.0->openai) Using cached idna-3.7-py3-none-any.whl.metadata (9.9 kB) Requirement already satisfied: six>=1.4.0 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from docker-pycreds>=0.4.0->wandb) (1.16.0) Collecting aiohappyeyeballs>=2.3.0 (from aiohttp->vllm) Downloading aiohappyeyeballs-2.3.2-py3-none-any.whl.metadata (5.5 kB) Collecting aiosignal>=1.1.2 (from aiohttp->vllm) Using cached aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB) Collecting attrs>=17.3.0 (from aiohttp->vllm) Using cached attrs-23.2.0-py3-none-any.whl.metadata (9.5 kB) Collecting frozenlist>=1.1.1 (from aiohttp->vllm) Using cached frozenlist-1.4.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB) Collecting multidict<7.0,>=4.5 (from aiohttp->vllm) Using cached multidict-6.0.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB) Collecting yarl<2.0,>=1.0 (from aiohttp->vllm) Using cached yarl-1.9.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB) Collecting gitdb<5,>=4.0.1 (from gitpython!=3.1.29,>=1.0.0->wandb) Using cached gitdb-4.0.11-py3-none-any.whl.metadata (1.2 kB) Collecting certifi (from httpx<1,>=0.23.0->openai) Using cached certifi-2024.7.4-py3-none-any.whl.metadata (2.2 kB) Collecting httpcore==1.* (from httpx<1,>=0.23.0->openai) Using cached httpcore-1.0.5-py3-none-any.whl.metadata (20 kB) Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx<1,>=0.23.0->openai) Using cached h11-0.14.0-py3-none-any.whl.metadata (8.2 kB) Collecting antlr4-python3-runtime==4.9.* (from hydra-core>=1.2.0->pytorch-lightning[extra]) Using cached antlr4_python3_runtime-4.9.3-py3-none-any.whl Collecting docstring-parser>=0.15 (from jsonargparse[signatures]>=4.27.7; extra == "extra"->pytorch-lightning[extra]) Using cached docstring_parser-0.16-py3-none-any.whl.metadata (3.0 kB) Collecting typeshed-client>=2.1.0 (from jsonargparse[signatures]>=4.27.7; extra == "extra"->pytorch-lightning[extra]) Using cached typeshed_client-2.7.0-py3-none-any.whl.metadata (7.9 kB) Collecting contourpy>=1.0.1 (from matplotlib>3.1->pytorch-lightning[extra]) Using cached contourpy-1.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.8 kB) Collecting cycler>=0.10 (from matplotlib>3.1->pytorch-lightning[extra]) Using cached cycler-0.12.1-py3-none-any.whl.metadata (3.8 kB) Collecting fonttools>=4.22.0 (from matplotlib>3.1->pytorch-lightning[extra]) Using cached fonttools-4.53.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (162 kB) Collecting kiwisolver>=1.3.1 (from matplotlib>3.1->pytorch-lightning[extra]) Using cached kiwisolver-1.4.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.4 kB) Collecting pyparsing>=2.3.1 (from matplotlib>3.1->pytorch-lightning[extra]) Using cached pyparsing-3.1.2-py3-none-any.whl.metadata (5.1 kB) Collecting python-dateutil>=2.7 (from matplotlib>3.1->pytorch-lightning[extra]) Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB) Collecting lark (from outlines<0.1,>=0.0.43->vllm) Using cached lark-1.1.9-py3-none-any.whl.metadata (1.9 kB) Collecting nest-asyncio (from outlines<0.1,>=0.0.43->vllm) Using cached nest_asyncio-1.6.0-py3-none-any.whl.metadata (2.8 kB) Collecting cloudpickle (from outlines<0.1,>=0.0.43->vllm) Using cached cloudpickle-3.0.0-py3-none-any.whl.metadata (7.0 kB) Collecting diskcache (from outlines<0.1,>=0.0.43->vllm) Using cached diskcache-5.6.3-py3-none-any.whl.metadata (20 kB) Collecting numba (from outlines<0.1,>=0.0.43->vllm) Using cached numba-0.60.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.7 kB) Collecting referencing (from outlines<0.1,>=0.0.43->vllm) Using cached referencing-0.35.1-py3-none-any.whl.metadata (2.8 kB) Collecting jsonschema (from outlines<0.1,>=0.0.43->vllm) Using cached jsonschema-4.23.0-py3-none-any.whl.metadata (7.9 kB) Collecting datasets (from outlines<0.1,>=0.0.43->vllm) Using cached datasets-2.20.0-py3-none-any.whl.metadata (19 kB) Collecting pycountry (from outlines<0.1,>=0.0.43->vllm) Using cached pycountry-24.6.1-py3-none-any.whl.metadata (12 kB) Collecting pyairports (from outlines<0.1,>=0.0.43->vllm) Using cached pyairports-2.1.1-py3-none-any.whl.metadata (1.7 kB) Collecting starlette<1.0.0,>=0.30.0 (from prometheus-fastapi-instrumentator>=7.0.0->vllm) Downloading starlette-0.38.2-py3-none-any.whl.metadata (5.9 kB) Collecting annotated-types>=0.4.0 (from pydantic->deepspeed) Using cached annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB) Collecting pydantic-core==2.20.1 (from pydantic->deepspeed) Using cached pydantic_core-2.20.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB) Collecting msgpack<2.0.0,>=1.0.0 (from ray>=2.8->ray[default]>=2.8->lean-dojo) Using cached msgpack-1.0.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.1 kB) Collecting aiohttp-cors (from ray[default]>=2.8->lean-dojo) Using cached aiohttp_cors-0.7.0-py3-none-any.whl.metadata (20 kB) Collecting colorful (from ray[default]>=2.8->lean-dojo) Using cached colorful-0.5.6-py2.py3-none-any.whl.metadata (16 kB) Collecting py-spy>=0.2.0 (from ray[default]>=2.8->lean-dojo) Using cached py_spy-0.3.14-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (16 kB) Collecting opencensus (from ray[default]>=2.8->lean-dojo) Using cached opencensus-0.11.4-py2.py3-none-any.whl.metadata (12 kB) Collecting smart-open (from ray[default]>=2.8->lean-dojo) Using cached smart_open-7.0.4-py3-none-any.whl.metadata (23 kB) Collecting virtualenv!=20.21.1,>=20.0.24 (from ray[default]>=2.8->lean-dojo) Using cached virtualenv-20.26.3-py3-none-any.whl.metadata (4.5 kB) Collecting grpcio>=1.42.0 (from ray[default]>=2.8->lean-dojo) Downloading grpcio-1.65.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.3 kB) Collecting memray (from ray[default]>=2.8->lean-dojo) Using cached memray-1.13.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) Collecting charset-normalizer<4,>=2 (from requests->transformers) Using cached charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB) Collecting urllib3<3,>=1.21.1 (from requests->transformers) Using cached urllib3-2.2.2-py3-none-any.whl.metadata (6.4 kB) Collecting markdown-it-py>=2.2.0 (from rich>=12.3.0->pytorch-lightning[extra]) Using cached markdown_it_py-3.0.0-py3-none-any.whl.metadata (6.9 kB) Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from rich>=12.3.0->pytorch-lightning[extra]) (2.15.1) Collecting starlette<1.0.0,>=0.30.0 (from prometheus-fastapi-instrumentator>=7.0.0->vllm) Using cached starlette-0.37.2-py3-none-any.whl.metadata (5.9 kB) Collecting fastapi-cli>=0.0.2 (from fastapi->vllm) Using cached fastapi_cli-0.0.4-py3-none-any.whl.metadata (7.0 kB) Collecting python-multipart>=0.0.7 (from fastapi->vllm) Using cached python_multipart-0.0.9-py3-none-any.whl.metadata (2.5 kB) Collecting email_validator>=2.0.0 (from fastapi->vllm) Using cached email_validator-2.2.0-py3-none-any.whl.metadata (25 kB) Collecting httptools>=0.5.0 (from uvicorn[standard]->vllm) Using cached httptools-0.6.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.6 kB) Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn[standard]->vllm) Using cached uvloop-0.19.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB) Collecting watchfiles>=0.13 (from uvicorn[standard]->vllm) Using cached watchfiles-0.22.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB) Collecting websockets>=10.4 (from uvicorn[standard]->vllm) Using cached websockets-12.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB) Requirement already satisfied: ptyprocess>=0.5 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from pexpect->lean-dojo) (0.7.0) Collecting pynacl>=1.4.0 (from pygithub->lean-dojo) Using cached PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl.metadata (8.6 kB) Collecting pyjwt>=2.4.0 (from pyjwt[crypto]>=2.4.0->pygithub->lean-dojo) Using cached PyJWT-2.8.0-py3-none-any.whl.metadata (4.2 kB) Collecting Deprecated (from pygithub->lean-dojo) Using cached Deprecated-1.2.14-py2.py3-none-any.whl.metadata (5.4 kB) Collecting dnspython>=2.0.0 (from email_validator>=2.0.0->fastapi->vllm) Using cached dnspython-2.6.1-py3-none-any.whl.metadata (5.8 kB) Collecting typer>=0.12.3 (from fastapi-cli>=0.0.2->fastapi->vllm) Using cached typer-0.12.3-py3-none-any.whl.metadata (15 kB) Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython!=3.1.29,>=1.0.0->wandb) Using cached smmap-5.0.1-py3-none-any.whl.metadata (4.3 kB) Requirement already satisfied: MarkupSafe>=2.0 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from jinja2->torch->deepspeed) (2.1.5) Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich>=12.3.0->pytorch-lightning[extra]) Using cached mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB) Collecting cryptography>=3.4.0 (from pyjwt[crypto]>=2.4.0->pygithub->lean-dojo) Using cached cryptography-43.0.0-cp39-abi3-manylinux_2_28_x86_64.whl.metadata (5.4 kB) Collecting cffi>=1.4.1 (from pynacl>=1.4.0->pygithub->lean-dojo) Using cached cffi-1.16.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB) Collecting importlib-resources>=1.4.0 (from typeshed-client>=2.1.0->jsonargparse[signatures]>=4.27.7; extra == "extra"->pytorch-lightning[extra]) Using cached importlib_resources-6.4.0-py3-none-any.whl.metadata (3.9 kB) Collecting distlib<1,>=0.3.7 (from virtualenv!=20.21.1,>=20.0.24->ray[default]>=2.8->lean-dojo) Using cached distlib-0.3.8-py2.py3-none-any.whl.metadata (5.1 kB) Collecting pyarrow>=15.0.0 (from datasets->outlines<0.1,>=0.0.43->vllm) Using cached pyarrow-17.0.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (3.3 kB) Collecting pyarrow-hotfix (from datasets->outlines<0.1,>=0.0.43->vllm) Using cached pyarrow_hotfix-0.6-py3-none-any.whl.metadata (3.6 kB) Collecting dill<0.3.9,>=0.3.0 (from datasets->outlines<0.1,>=0.0.43->vllm) Using cached dill-0.3.8-py3-none-any.whl.metadata (10 kB) Collecting pandas (from datasets->outlines<0.1,>=0.0.43->vllm) Using cached pandas-2.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) Collecting xxhash (from datasets->outlines<0.1,>=0.0.43->vllm) Using cached xxhash-3.4.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB) Collecting multiprocess (from datasets->outlines<0.1,>=0.0.43->vllm) Using cached multiprocess-0.70.16-py311-none-any.whl.metadata (7.2 kB) INFO: pip is looking at multiple versions of datasets to determine which version is compatible with other requirements. This could take a while. Collecting datasets (from outlines<0.1,>=0.0.43->vllm) Downloading datasets-2.19.2-py3-none-any.whl.metadata (19 kB) Downloading datasets-2.19.1-py3-none-any.whl.metadata (19 kB) Downloading datasets-2.19.0-py3-none-any.whl.metadata (19 kB) Downloading datasets-2.18.0-py3-none-any.whl.metadata (20 kB) Downloading datasets-2.17.1-py3-none-any.whl.metadata (20 kB) Downloading datasets-2.17.0-py3-none-any.whl.metadata (20 kB) Downloading datasets-2.16.1-py3-none-any.whl.metadata (20 kB) Collecting dill<0.3.8,>=0.3.0 (from datasets->outlines<0.1,>=0.0.43->vllm) Using cached dill-0.3.7-py3-none-any.whl.metadata (9.9 kB) INFO: pip is still looking at multiple versions of datasets to determine which version is compatible with other requirements. This could take a while. Collecting datasets (from outlines<0.1,>=0.0.43->vllm) Downloading datasets-2.16.0-py3-none-any.whl.metadata (20 kB) Using cached datasets-2.15.0-py3-none-any.whl.metadata (20 kB) Downloading datasets-2.14.7-py3-none-any.whl.metadata (19 kB) Using cached datasets-2.14.6-py3-none-any.whl.metadata (19 kB) Downloading datasets-2.14.5-py3-none-any.whl.metadata (19 kB) INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C. Downloading datasets-2.14.4-py3-none-any.whl.metadata (19 kB) Collecting wrapt<2,>=1.10 (from Deprecated->pygithub->lean-dojo) Using cached wrapt-1.16.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB) Collecting jsonschema-specifications>=2023.03.6 (from jsonschema->outlines<0.1,>=0.0.43->vllm) Using cached jsonschema_specifications-2023.12.1-py3-none-any.whl.metadata (3.0 kB) Collecting rpds-py>=0.7.1 (from jsonschema->outlines<0.1,>=0.0.43->vllm) Using cached rpds_py-0.19.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB) Collecting textual>=0.41.0 (from memray->ray[default]>=2.8->lean-dojo) Using cached textual-0.74.0-py3-none-any.whl.metadata (5.6 kB) Collecting llvmlite<0.44,>=0.43.0dev0 (from numba->outlines<0.1,>=0.0.43->vllm) Using cached llvmlite-0.43.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.8 kB) Collecting opencensus-context>=0.1.3 (from opencensus->ray[default]>=2.8->lean-dojo) Using cached opencensus_context-0.1.3-py2.py3-none-any.whl.metadata (3.3 kB) Collecting google-api-core<3.0.0,>=1.0.0 (from opencensus->ray[default]>=2.8->lean-dojo) Using cached google_api_core-2.19.1-py3-none-any.whl.metadata (2.7 kB) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages (from sympy->torch->deepspeed) (1.3.0) Collecting pycparser (from cffi>=1.4.1->pynacl>=1.4.0->pygithub->lean-dojo) Using cached pycparser-2.22-py3-none-any.whl.metadata (943 bytes) Collecting googleapis-common-protos<2.0.dev0,>=1.56.2 (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.8->lean-dojo) Using cached googleapis_common_protos-1.63.2-py2.py3-none-any.whl.metadata (1.5 kB) Collecting proto-plus<2.0.0dev,>=1.22.3 (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.8->lean-dojo) Using cached proto_plus-1.24.0-py3-none-any.whl.metadata (2.2 kB) Collecting google-auth<3.0.dev0,>=2.14.1 (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.8->lean-dojo) Using cached google_auth-2.32.0-py2.py3-none-any.whl.metadata (4.7 kB) Collecting shellingham>=1.3.0 (from typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm) Using cached shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB) INFO: pip is looking at multiple versions of multiprocess to determine which version is compatible with other requirements. This could take a while. Collecting multiprocess (from datasets->outlines<0.1,>=0.0.43->vllm) Downloading multiprocess-0.70.15-py311-none-any.whl.metadata (7.2 kB) Collecting pytz>=2020.1 (from pandas->datasets->outlines<0.1,>=0.0.43->vllm) Using cached pytz-2024.1-py2.py3-none-any.whl.metadata (22 kB) Collecting tzdata>=2022.7 (from pandas->datasets->outlines<0.1,>=0.0.43->vllm) Using cached tzdata-2024.1-py2.py3-none-any.whl.metadata (1.4 kB) Collecting cachetools<6.0,>=2.0.0 (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.8->lean-dojo) Using cached cachetools-5.4.0-py3-none-any.whl.metadata (5.3 kB) Collecting pyasn1-modules>=0.2.1 (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.8->lean-dojo) Using cached pyasn1_modules-0.4.0-py3-none-any.whl.metadata (3.4 kB) Collecting rsa<5,>=3.1.4 (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.8->lean-dojo) Using cached rsa-4.9-py3-none-any.whl.metadata (4.2 kB) Collecting linkify-it-py<3,>=1 (from markdown-it-py[linkify,plugins]>=2.1.0->textual>=0.41.0->memray->ray[default]>=2.8->lean-dojo) Using cached linkify_it_py-2.0.3-py3-none-any.whl.metadata (8.5 kB) Collecting mdit-py-plugins (from markdown-it-py[linkify,plugins]>=2.1.0->textual>=0.41.0->memray->ray[default]>=2.8->lean-dojo) Using cached mdit_py_plugins-0.4.1-py3-none-any.whl.metadata (2.8 kB) Collecting uc-micro-py (from linkify-it-py<3,>=1->markdown-it-py[linkify,plugins]>=2.1.0->textual>=0.41.0->memray->ray[default]>=2.8->lean-dojo) Using cached uc_micro_py-1.0.3-py3-none-any.whl.metadata (2.0 kB) Collecting pyasn1<0.7.0,>=0.4.6 (from pyasn1-modules>=0.2.1->google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.8->lean-dojo) Using cached pyasn1-0.6.0-py2.py3-none-any.whl.metadata (8.3 kB) Using cached tqdm-4.66.4-py3-none-any.whl (78 kB) Using cached loguru-0.7.2-py3-none-any.whl (62 kB) Using cached transformers-4.43.3-py3-none-any.whl (9.4 MB) Using cached wandb-0.17.5-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.8 MB) Using cached openai-1.37.1-py3-none-any.whl (337 kB) Using cached rank_bm25-0.2.2-py3-none-any.whl (8.6 kB) Using cached lean_dojo-2.0.3-py3-none-any.whl (50 kB) Using cached vllm-0.5.3.post1-cp38-abi3-manylinux1_x86_64.whl (158.3 MB) Using cached lm_format_enforcer-0.10.3-py3-none-any.whl (43 kB) Using cached torch-2.3.1-cp311-cp311-manylinux1_x86_64.whl (779.2 MB) Using cached torchvision-0.18.1-cp311-cp311-manylinux1_x86_64.whl (7.0 MB) Using cached vllm_flash_attn-2.5.9.post1-cp311-cp311-manylinux1_x86_64.whl (37.1 MB) Using cached xformers-0.0.27-cp311-cp311-manylinux2014_x86_64.whl (164.2 MB) Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB) Using cached triton-2.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (168.1 MB) Using cached anyio-4.4.0-py3-none-any.whl (86 kB) Downloading bitsandbytes-0.43.3-py3-none-manylinux_2_24_x86_64.whl (137.5 MB) โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 137.5/137.5 MB 14.4 MB/s eta 0:00:00 Using cached click-8.1.7-py3-none-any.whl (97 kB) Using cached cmake-3.30.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.9 MB) Using cached distro-1.9.0-py3-none-any.whl (20 kB) Using cached docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB) Downloading aiohttp-3.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB) โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 1.3/1.3 MB 3.9 MB/s eta 0:00:00 Using cached GitPython-3.1.43-py3-none-any.whl (207 kB) Using cached httpx-0.27.0-py3-none-any.whl (75 kB) Using cached httpcore-1.0.5-py3-none-any.whl (77 kB) Downloading huggingface_hub-0.24.5-py3-none-any.whl (417 kB) โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 417.5/417.5 kB 18.6 MB/s eta 0:00:00 Using cached hydra_core-1.3.2-py3-none-any.whl (154 kB) Using cached jsonargparse-4.32.0-py3-none-any.whl (206 kB) Using cached lightning_utilities-0.11.6-py3-none-any.whl (26 kB) Using cached matplotlib-3.9.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.3 MB) โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 18.3/18.3 MB 17.7 MB/s eta 0:00:00 Using cached omegaconf-2.3.0-py3-none-any.whl (79 kB) Using cached outlines-0.0.46-py3-none-any.whl (101 kB) Using cached packaging-24.1-py3-none-any.whl (53 kB) Using cached prometheus_client-0.20.0-py3-none-any.whl (54 kB) Using cached prometheus_fastapi_instrumentator-7.0.0-py3-none-any.whl (19 kB) Using cached protobuf-5.27.2-cp38-abi3-manylinux2014_x86_64.whl (309 kB) Using cached psutil-6.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (290 kB) Using cached pydantic-2.8.2-py3-none-any.whl (423 kB) Using cached pydantic_core-2.20.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB) Using cached PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (757 kB) Downloading ray-2.34.0-cp311-cp311-manylinux2014_x86_64.whl (65.0 MB) โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 65.0/65.0 MB 13.5 MB/s eta 0:00:00 Using cached regex-2024.7.24-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (786 kB) Using cached requests-2.32.3-py3-none-any.whl (64 kB) Using cached rich-13.7.1-py3-none-any.whl (240 kB) Using cached safetensors-0.4.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB) Downloading sentry_sdk-2.12.0-py2.py3-none-any.whl (301 kB) โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 301.8/301.8 kB 13.1 MB/s eta 0:00:00 Using cached sniffio-1.3.1-py3-none-any.whl (10 kB) Using cached tensorboardX-2.6.2.2-py2.py3-none-any.whl (101 kB) Using cached tiktoken-0.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB) Using cached tokenizers-0.19.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB) Using cached torchmetrics-1.4.0.post0-py3-none-any.whl (868 kB) Using cached fastapi-0.111.1-py3-none-any.whl (92 kB) Using cached python_dotenv-1.0.1-py3-none-any.whl (19 kB) Using cached hjson-3.1.0-py3-none-any.whl (54 kB) Using cached lxml-5.2.2-cp311-cp311-manylinux_2_28_x86_64.whl (5.0 MB) Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB) Using cached nvidia_ml_py-12.555.43-py3-none-any.whl (39 kB) Using cached platformdirs-4.2.2-py3-none-any.whl (18 kB) Using cached py_cpuinfo-9.0.0-py3-none-any.whl (22 kB) Using cached PyGithub-2.3.0-py3-none-any.whl (354 kB) Using cached pytorch_lightning-2.3.3-py3-none-any.whl (812 kB) Using cached pyzmq-26.0.3-cp311-cp311-manylinux_2_28_x86_64.whl (919 kB) Using cached sentencepiece-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB) Using cached setproctitle-1.3.3-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (31 kB) Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB) Using cached types_psutil-6.0.0.20240621-py3-none-any.whl (18 kB) Using cached types_toml-0.10.8.20240310-py3-none-any.whl (4.8 kB) Downloading aiohappyeyeballs-2.3.2-py3-none-any.whl (11 kB) Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB) Using cached annotated_types-0.7.0-py3-none-any.whl (13 kB) Using cached attrs-23.2.0-py3-none-any.whl (60 kB) Using cached certifi-2024.7.4-py3-none-any.whl (162 kB) Using cached charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (140 kB) Using cached contourpy-1.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (306 kB) Using cached cycler-0.12.1-py3-none-any.whl (8.3 kB) Using cached docstring_parser-0.16-py3-none-any.whl (36 kB) Using cached email_validator-2.2.0-py3-none-any.whl (33 kB) Using cached fastapi_cli-0.0.4-py3-none-any.whl (9.5 kB) Using cached fonttools-4.53.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.9 MB) Using cached frozenlist-1.4.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (272 kB) Using cached gitdb-4.0.11-py3-none-any.whl (62 kB) Downloading grpcio-1.65.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.7 MB) โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 5.7/5.7 MB 16.4 MB/s eta 0:00:00 Using cached h11-0.14.0-py3-none-any.whl (58 kB) Using cached httptools-0.6.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (318 kB) Using cached idna-3.7-py3-none-any.whl (66 kB) Using cached interegular-0.3.3-py37-none-any.whl (23 kB) Using cached kiwisolver-1.4.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB) Using cached markdown_it_py-3.0.0-py3-none-any.whl (87 kB) Using cached msgpack-1.0.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (409 kB) Using cached multidict-6.0.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (128 kB) Using cached py_spy-0.3.14-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (3.0 MB) Using cached PyJWT-2.8.0-py3-none-any.whl (22 kB) Using cached PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (856 kB) Using cached pyparsing-3.1.2-py3-none-any.whl (103 kB) Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) Using cached python_multipart-0.0.9-py3-none-any.whl (22 kB) Using cached starlette-0.37.2-py3-none-any.whl (71 kB) Using cached typeshed_client-2.7.0-py3-none-any.whl (624 kB) Using cached urllib3-2.2.2-py3-none-any.whl (121 kB) Using cached uvicorn-0.30.3-py3-none-any.whl (62 kB) Using cached uvloop-0.19.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB) Using cached virtualenv-20.26.3-py3-none-any.whl (5.7 MB) Using cached watchfiles-0.22.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB) Using cached websockets-12.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (130 kB) Using cached yarl-1.9.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (328 kB) Using cached aiohttp_cors-0.7.0-py3-none-any.whl (27 kB) Using cached cloudpickle-3.0.0-py3-none-any.whl (20 kB) Using cached colorful-0.5.6-py2.py3-none-any.whl (201 kB) Downloading datasets-2.14.4-py3-none-any.whl (519 kB) โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 519.3/519.3 kB 19.0 MB/s eta 0:00:00 Using cached Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB) Using cached diskcache-5.6.3-py3-none-any.whl (45 kB) Using cached jsonschema-4.23.0-py3-none-any.whl (88 kB) Using cached referencing-0.35.1-py3-none-any.whl (26 kB) Using cached lark-1.1.9-py3-none-any.whl (111 kB) Using cached memray-1.13.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.4 MB) Using cached nest_asyncio-1.6.0-py3-none-any.whl (5.2 kB) Using cached numba-0.60.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.7 MB) Using cached opencensus-0.11.4-py2.py3-none-any.whl (128 kB) Using cached pyairports-2.1.1-py3-none-any.whl (371 kB) Using cached pycountry-24.6.1-py3-none-any.whl (6.3 MB) Using cached smart_open-7.0.4-py3-none-any.whl (61 kB) Using cached cffi-1.16.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (464 kB) Using cached cryptography-43.0.0-cp39-abi3-manylinux_2_28_x86_64.whl (4.0 MB) Using cached dill-0.3.7-py3-none-any.whl (115 kB) Using cached distlib-0.3.8-py2.py3-none-any.whl (468 kB) Using cached dnspython-2.6.1-py3-none-any.whl (307 kB) Using cached google_api_core-2.19.1-py3-none-any.whl (139 kB) Using cached importlib_resources-6.4.0-py3-none-any.whl (38 kB) Using cached jsonschema_specifications-2023.12.1-py3-none-any.whl (18 kB) Using cached llvmlite-0.43.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (43.9 MB) Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB) Using cached opencensus_context-0.1.3-py2.py3-none-any.whl (5.1 kB) Using cached pyarrow-17.0.0-cp311-cp311-manylinux_2_28_x86_64.whl (39.9 MB) Using cached rpds_py-0.19.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (355 kB) Using cached smmap-5.0.1-py3-none-any.whl (24 kB) Using cached textual-0.74.0-py3-none-any.whl (566 kB) Using cached typer-0.12.3-py3-none-any.whl (47 kB) Using cached wrapt-1.16.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (80 kB) Downloading multiprocess-0.70.15-py311-none-any.whl (135 kB) โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ” 135.4/135.4 kB 12.9 MB/s eta 0:00:00 Using cached pandas-2.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.0 MB) Using cached xxhash-3.4.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB) Using cached google_auth-2.32.0-py2.py3-none-any.whl (195 kB) Using cached googleapis_common_protos-1.63.2-py2.py3-none-any.whl (220 kB) Using cached proto_plus-1.24.0-py3-none-any.whl (50 kB) Using cached pytz-2024.1-py2.py3-none-any.whl (505 kB) Using cached shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB) Using cached tzdata-2024.1-py2.py3-none-any.whl (345 kB) Using cached pycparser-2.22-py3-none-any.whl (117 kB) Using cached cachetools-5.4.0-py3-none-any.whl (9.5 kB) Using cached linkify_it_py-2.0.3-py3-none-any.whl (19 kB) Using cached pyasn1_modules-0.4.0-py3-none-any.whl (181 kB) Using cached rsa-4.9-py3-none-any.whl (34 kB) Using cached mdit_py_plugins-0.4.1-py3-none-any.whl (54 kB) Using cached pyasn1-0.6.0-py2.py3-none-any.whl (85 kB) Using cached uc_micro_py-1.0.3-py3-none-any.whl (6.2 kB) Installing collected packages: sentencepiece, pytz, pyairports, py-spy, py-cpuinfo, opencensus-context, nvidia-ml-py, ninja, hjson, distlib, colorful, antlr4-python3-runtime, xxhash, wrapt, websockets, uvloop, urllib3, uc-micro-py, tzdata, types-toml, types-psutil, triton, tqdm, toml, sniffio, smmap, shellingham, setproctitle, safetensors, rpds-py, regex, pyzmq, PyYAML, python-multipart, python-dotenv, python-dateutil, pyparsing, pyjwt, pydantic-core, pycparser, pycountry, pyasn1, psutil, protobuf, prometheus-client, platformdirs, packaging, nvidia-cudnn-cu12, numpy, nest-asyncio, multidict, msgpack, mdurl, lxml, loguru, llvmlite, lark, kiwisolver, interegular, importlib-resources, idna, httptools, h11, grpcio, frozenlist, fonttools, docstring-parser, docker-pycreds, dnspython, distro, diskcache, dill, cycler, cmake, cloudpickle, click, charset-normalizer, certifi, cachetools, attrs, annotated-types, aiohappyeyeballs, yarl, virtualenv, uvicorn, typeshed-client, tensorboardX, smart-open, sentry-sdk, rsa, requests, referencing, rank_bm25, pydantic, pyasn1-modules, pyarrow, proto-plus, pandas, omegaconf, numba, multiprocess, markdown-it-py, linkify-it-py, lightning-utilities, jsonargparse, httpcore, googleapis-common-protos, gitdb, email_validator, Deprecated, contourpy, cffi, anyio, aiosignal, watchfiles, torch, tiktoken, starlette, rich, pynacl, mdit-py-plugins, matplotlib, lm-format-enforcer, jsonschema-specifications, hydra-core, huggingface-hub, httpx, google-auth, gitpython, cryptography, aiohttp, xformers, wandb, vllm-flash-attn, typer, torchvision, torchmetrics, tokenizers, prometheus-fastapi-instrumentator, openai, jsonschema, google-api-core, deepspeed, bitsandbytes, aiohttp-cors, transformers, textual, ray, pytorch-lightning, pygithub, opencensus, fastapi-cli, datasets, outlines, memray, fastapi, vllm, lean-dojo Attempting uninstall: triton Found existing installation: triton 3.0.0 Uninstalling triton-3.0.0: Successfully uninstalled triton-3.0.0 Attempting uninstall: nvidia-cudnn-cu12 Found existing installation: nvidia-cudnn-cu12 9.1.0.70 Uninstalling nvidia-cudnn-cu12-9.1.0.70: Successfully uninstalled nvidia-cudnn-cu12-9.1.0.70 Attempting uninstall: numpy Found existing installation: numpy 2.0.1 Uninstalling numpy-2.0.1: Successfully uninstalled numpy-2.0.1 Attempting uninstall: torch Found existing installation: torch 2.4.0 Uninstalling torch-2.4.0: Successfully uninstalled torch-2.4.0 Attempting uninstall: torchvision Found existing installation: torchvision 0.19.0 Uninstalling torchvision-0.19.0: Successfully uninstalled torchvision-0.19.0 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. torchaudio 2.4.0 requires torch==2.4.0, but you have torch 2.3.1 which is incompatible. Successfully installed Deprecated-1.2.14 PyYAML-6.0.1 aiohappyeyeballs-2.3.2 aiohttp-3.10.0 aiohttp-cors-0.7.0 aiosignal-1.3.1 annotated-types-0.7.0 antlr4-python3-runtime-4.9.3 anyio-4.4.0 attrs-23.2.0 bitsandbytes-0.43.3 cachetools-5.4.0 certifi-2024.7.4 cffi-1.16.0 charset-normalizer-3.3.2 click-8.1.7 cloudpickle-3.0.0 cmake-3.30.1 colorful-0.5.6 contourpy-1.2.1 cryptography-43.0.0 cycler-0.12.1 datasets-2.14.4 deepspeed-0.14.4 dill-0.3.7 diskcache-5.6.3 distlib-0.3.8 distro-1.9.0 dnspython-2.6.1 docker-pycreds-0.4.0 docstring-parser-0.16 email_validator-2.2.0 fastapi-0.111.1 fastapi-cli-0.0.4 fonttools-4.53.1 frozenlist-1.4.1 gitdb-4.0.11 gitpython-3.1.43 google-api-core-2.19.1 google-auth-2.32.0 googleapis-common-protos-1.63.2 grpcio-1.65.2 h11-0.14.0 hjson-3.1.0 httpcore-1.0.5 httptools-0.6.1 httpx-0.27.0 huggingface-hub-0.24.5 hydra-core-1.3.2 idna-3.7 importlib-resources-6.4.0 interegular-0.3.3 jsonargparse-4.32.0 jsonschema-4.23.0 jsonschema-specifications-2023.12.1 kiwisolver-1.4.5 lark-1.1.9 lean-dojo-2.0.3 lightning-utilities-0.11.6 linkify-it-py-2.0.3 llvmlite-0.43.0 lm-format-enforcer-0.10.3 loguru-0.7.2 lxml-5.2.2 markdown-it-py-3.0.0 matplotlib-3.9.1 mdit-py-plugins-0.4.1 mdurl-0.1.2 memray-1.13.4 msgpack-1.0.8 multidict-6.0.5 multiprocess-0.70.15 nest-asyncio-1.6.0 ninja-1.11.1.1 numba-0.60.0 numpy-1.26.4 nvidia-cudnn-cu12-8.9.2.26 nvidia-ml-py-12.555.43 omegaconf-2.3.0 openai-1.37.1 opencensus-0.11.4 opencensus-context-0.1.3 outlines-0.0.46 packaging-24.1 pandas-2.2.2 platformdirs-4.2.2 prometheus-client-0.20.0 prometheus-fastapi-instrumentator-7.0.0 proto-plus-1.24.0 protobuf-5.27.2 psutil-6.0.0 py-cpuinfo-9.0.0 py-spy-0.3.14 pyairports-2.1.1 pyarrow-17.0.0 pyasn1-0.6.0 pyasn1-modules-0.4.0 pycountry-24.6.1 pycparser-2.22 pydantic-2.8.2 pydantic-core-2.20.1 pygithub-2.3.0 pyjwt-2.8.0 pynacl-1.5.0 pyparsing-3.1.2 python-dateutil-2.9.0.post0 python-dotenv-1.0.1 python-multipart-0.0.9 pytorch-lightning-2.3.3 pytz-2024.1 pyzmq-26.0.3 rank_bm25-0.2.2 ray-2.34.0 referencing-0.35.1 regex-2024.7.24 requests-2.32.3 rich-13.7.1 rpds-py-0.19.1 rsa-4.9 safetensors-0.4.3 sentencepiece-0.2.0 sentry-sdk-2.12.0 setproctitle-1.3.3 shellingham-1.5.4 smart-open-7.0.4 smmap-5.0.1 sniffio-1.3.1 starlette-0.37.2 tensorboardX-2.6.2.2 textual-0.74.0 tiktoken-0.7.0 tokenizers-0.19.1 toml-0.10.2 torch-2.3.1 torchmetrics-1.4.0.post0 torchvision-0.18.1 tqdm-4.66.4 transformers-4.43.3 triton-2.3.1 typer-0.12.3 types-psutil-6.0.0.20240621 types-toml-0.10.8.20240310 typeshed-client-2.7.0 tzdata-2024.1 uc-micro-py-1.0.3 urllib3-2.2.2 uvicorn-0.30.3 uvloop-0.19.0 virtualenv-20.26.3 vllm-0.5.3.post1 vllm-flash-attn-2.5.9.post1 wandb-0.17.5 watchfiles-0.22.0 websockets-12.0 wrapt-1.16.0 xformers-0.0.27 xxhash-3.4.1 yarl-1.9.4 (ReProver) mfan@localhost:~/ReProver$ export PYTHONPATH=$PYTHONPATH:/home/mfan/ReProver (ReProver) mfan@localhost:~/ReProver$ python scripts/download_data.py 2024-07-31 15:00:20.559 | INFO | __main__:main:39 - Namespace(data_path='data') 2024-07-31 15:00:20.560 | INFO | __main__:main:45 - Downloading https://zenodo.org/records/12740403/files/leandojo_benchmark_4.tar.gz?download=1 --2024-07-31 15:00:20-- https://zenodo.org/records/12740403/files/leandojo_benchmark_4.tar.gz?download=1 Resolving zenodo.org (zenodo.org)... 188.184.98.238, 188.185.79.172, 188.184.103.159, ... Connecting to zenodo.org (zenodo.org)|188.184.98.238|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 68141936 (65M) [application/octet-stream] Saving to: โ€˜data/leandojo_benchmark_4.tar.gz?download=1โ€™ data/leandojo_benchmark_4. 100%[======================================>] 64.98M 514KB/s in 2m 23s 2024-07-31 15:02:43 (466 KB/s) - โ€˜data/leandojo_benchmark_4.tar.gz?download=1โ€™ saved [68141936/68141936] 2024-07-31 15:02:43.999 | INFO | __main__:main:51 - Extracting data/leandojo_benchmark_4.tar.gz?download=1 2024-07-31 15:02:46.035 | INFO | __main__:main:54 - Removing data/leandojo_benchmark_4.tar.gz?download=1 2024-07-31 15:02:46.038 | INFO | __main__:main:57 - Done! (ReProver) mfan@localhost:~/ReProver$ python scripts/trace_repos.py 2024-07-31 15:04:07.132 | INFO | __main__:main:12 - Namespace(data_path='data') 2024-07-31 15:04:14.821 | INFO | __main__:main:26 - Repos to trace: set() (ReProver) mfan@localhost:~/ReProver$ wandb login wandb: Currently logged in as: greathero (greathero1user). Use `wandb login --relogin` to force relogin (ReProver) mfan@localhost:~/ReProver$ python generation/main.py fit --config generation/confs/cli_lean4_novel_premises.yaml --trainer.logger.name train_generator_novel_premises --trainer.logger.save_dir logs/train_generator_novel_premises # LeanDojo Benchmark 4, `novel_premises` split [2024-07-31 15:04:30,247] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [WARNING] async_io requires the dev libaio .so object and headers but these were not found. [WARNING] async_io: please install the libaio-dev package with apt [WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible 2024-07-31 15:04:30.832 | INFO | __main__:main:19 - PID: 4634 Seed set to 3407 2024-07-31 15:04:31.673 | INFO | common:__init__:200 - Building the corpus from data/leandojo_benchmark_4/corpus.jsonl 2024-07-31 15:04:43.509 | INFO | generation.datamodule:__init__:147 - Without retrieval data GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores HPU available: False, using: 0 HPUs [rank: 0] Seed set to 3407 initializing deepspeed distributed: GLOBAL_RANK: 0, MEMBER: 1/1 wandb: Currently logged in as: greathero (greathero1user). Use `wandb login --relogin` to force relogin wandb: WARNING Path logs/train_generator_novel_premises/wandb/ wasn't writable, using system temp directory. wandb: WARNING Path logs/train_generator_novel_premises/wandb/ wasn't writable, using system temp directory wandb: Tracking run with wandb version 0.17.5 wandb: Run data is saved locally in /tmp/wandb/run-20240731_150444-e4q048jk wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run train_generator_novel_premises wandb: โญ๏ธ View project at https://wandb.ai/greathero1user/lightning_logs wandb: ๐Ÿš€ View run at https://wandb.ai/greathero1user/lightning_logs/runs/e4q048jk 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 118517/118517 [00:00<00:00, 621183.28it/s] 2024-07-31 15:04:58.114 | INFO | generation.datamodule:_load_data:60 - 246714 examples loaded 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 2000/2000 [00:00<00:00, 559987.18it/s] 2024-07-31 15:04:58.174 | INFO | generation.datamodule:_load_data:60 - 6240 examples loaded Enabling DeepSpeed BF16. Model parameters and inputs will be cast to `bfloat16`. LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 2024-07-31 15:04:58.242 | INFO | common:get_optimizers:392 - Optimizing with FusedAdam Using /home/mfan/.cache/torch_extensions/py311_cu121 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /home/mfan/.cache/torch_extensions/py311_cu121/fused_adam/build.ninja... /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. warnings.warn( Building extension module fused_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/3] /usr/local/cuda-12.1/bin/nvcc --generate-dependencies-with-compile --dependency-output multi_tensor_adam.cuda.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/ops/csrc/adam -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include/TH -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda-12.1/include -isystem /home/mfan/miniconda3/envs/ReProver/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -lineinfo --use_fast_math -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_86,code=compute_86 -DBF16_AVAILABLE -U__CUDA_NO_BFLOAT16_OPERATORS__ -U__CUDA_NO_BFLOAT162_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ -std=c++17 -c /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu -o multi_tensor_adam.cuda.o [2/3] c++ -MMD -MF fused_adam_frontend.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/ops/csrc/adam -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include/TH -isystem /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda-12.1/include -isystem /home/mfan/miniconda3/envs/ReProver/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DBF16_AVAILABLE -c /home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -o fused_adam_frontend.o [3/3] c++ fused_adam_frontend.o multi_tensor_adam.cuda.o -shared -L/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/local/cuda-12.1/lib64 -lcudart -o fused_adam.so Loading extension module fused_adam... Time to load fused_adam op: 16.516267776489258 seconds Traceback (most recent call last): File "/home/mfan/ReProver/generation/main.py", line 25, in main() File "/home/mfan/ReProver/generation/main.py", line 20, in main cli = CLI(RetrievalAugmentedGenerator, GeneratorDataModule) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/cli.py", line 394, in __init__ self._run_subcommand(self.subcommand) File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/cli.py", line 701, in _run_subcommand fn(**fn_kwargs) File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 543, in fit call._call_and_handle_interrupt( File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 105, in launch return function(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl self._run(model, ckpt_path=ckpt_path) File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in _run self.strategy.setup(self) File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/deepspeed.py", line 350, in setup self.init_deepspeed() File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/deepspeed.py", line 451, in init_deepspeed self._initialize_deepspeed_train(self.model) File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/deepspeed.py", line 487, in _initialize_deepspeed_train model, deepspeed_optimizer = self._setup_model_and_optimizer(model, optimizer, scheduler) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/deepspeed.py", line 423, in _setup_model_and_optimizer deepspeed_engine, deepspeed_optimizer, _, _ = deepspeed.initialize( ^^^^^^^^^^^^^^^^^^^^^ File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/__init__.py", line 181, in initialize engine = DeepSpeedEngine(args=args, ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 262, in __init__ self._configure_distributed_model(model) File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1148, in _configure_distributed_model self._broadcast_model() File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1068, in _broadcast_model dist.broadcast(p.data, groups._get_broadcast_src_rank(), group=self.seq_data_parallel_group) File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 117, in log_wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 224, in broadcast return cdb.broadcast(tensor=tensor, src=src, group=group, async_op=async_op) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/comm/torch.py", line 199, in broadcast return torch.distributed.broadcast(tensor=tensor, src=src, group=group, async_op=async_op) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2140, in broadcast work = group.broadcast([tensor], opts) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: Tensors must be contiguous [rank0]: Traceback (most recent call last): [rank0]: File "/home/mfan/ReProver/generation/main.py", line 25, in [rank0]: main() [rank0]: File "/home/mfan/ReProver/generation/main.py", line 20, in main [rank0]: cli = CLI(RetrievalAugmentedGenerator, GeneratorDataModule) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/cli.py", line 394, in __init__ [rank0]: self._run_subcommand(self.subcommand) [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/cli.py", line 701, in _run_subcommand [rank0]: fn(**fn_kwargs) [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 543, in fit [rank0]: call._call_and_handle_interrupt( [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt [rank0]: return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 105, in launch [rank0]: return function(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl [rank0]: self._run(model, ckpt_path=ckpt_path) [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in _run [rank0]: self.strategy.setup(self) [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/deepspeed.py", line 350, in setup [rank0]: self.init_deepspeed() [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/deepspeed.py", line 451, in init_deepspeed [rank0]: self._initialize_deepspeed_train(self.model) [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/deepspeed.py", line 487, in _initialize_deepspeed_train [rank0]: model, deepspeed_optimizer = self._setup_model_and_optimizer(model, optimizer, scheduler) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/pytorch_lightning/strategies/deepspeed.py", line 423, in _setup_model_and_optimizer [rank0]: deepspeed_engine, deepspeed_optimizer, _, _ = deepspeed.initialize( [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/__init__.py", line 181, in initialize [rank0]: engine = DeepSpeedEngine(args=args, [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 262, in __init__ [rank0]: self._configure_distributed_model(model) [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1148, in _configure_distributed_model [rank0]: self._broadcast_model() [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1068, in _broadcast_model [rank0]: dist.broadcast(p.data, groups._get_broadcast_src_rank(), group=self.seq_data_parallel_group) [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 117, in log_wrapper [rank0]: return func(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 224, in broadcast [rank0]: return cdb.broadcast(tensor=tensor, src=src, group=group, async_op=async_op) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn [rank0]: return fn(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/deepspeed/comm/torch.py", line 199, in broadcast [rank0]: return torch.distributed.broadcast(tensor=tensor, src=src, group=group, async_op=async_op) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper [rank0]: return func(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/home/mfan/miniconda3/envs/ReProver/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2140, in broadcast [rank0]: work = group.broadcast([tensor], opts) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: ValueError: Tensors must be contiguous wandb: ๐Ÿš€ View run train_generator_novel_premises at: https://wandb.ai/greathero1user/lightning_logs/runs/e4q048jk wandb: โญ๏ธ View project at: https://wandb.ai/greathero1user/lightning_logs wandb: Synced 6 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s) wandb: Find logs at: /tmp/wandb/run-20240731_150444-e4q048jk/logs wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with `wandb.require("core")`! See https://wandb.me/wandb-core for more information ```

I'll search for a solution in the Github issue for DeepSpeed.

(I was previously using python 3.10; I installed CUDA 12.1 after installing driver version 545 instead of 530, due to bugs with 530 as in this link.)

realharryhero commented 3 months ago

According to this fix, I uninstalled DeepSpeed, cloned the DeepSpeed repository, made a change to broadcast a continuous tensor, and then downloaded DeepSpeed again from the cloned repository, and it works. Thank you!

huolongguo1O commented 3 months ago

I wasn't able to reproduce the error. From the logs, it looks like the error is in PyTorch Lightning or DeepSpeed. So I don't think adding a . contiguous() in ReProver would actually help. Have you tried creating a new conda environment and re-install those dependencies?

ๅœจ4090ๅ’Œ3080TiไธŠ้‡ๅˆฐๅŒๆ ท็š„้—ฎ้ข˜๏ผŒ่ƒฝไธ่ƒฝๆ”ฏๆŒไธไฝฟ็”จdeepspeed

yifan123 commented 2 months ago

for param in model.parameters(): param.data = param.data.contiguous() https://github.com/huggingface/transformers/issues/28293

After loading model, add the above line to convert weight to contiguous, it works for me!