search

KoboldAI / KoboldAI-Client

For GGUF support, see KoboldCPP: https://github.com/LostRuins/koboldcpp
https://koboldai.com
GNU Affero General Public License v3.0
3.48k stars 747 forks source link

kobold ai does not start #246

Open Myuless opened 1 year ago

Myuless commented 1 year ago

Displays this text Found TPU at: grpc://10.85.230.122:8470 Now we will need your Google Drive to store settings and saves, you must login with the same account you used for Colab. Mounted at /content/drive/ --2023-02-24 06:10:37-- https://koboldai.org/ckds Resolving koboldai.org (koboldai.org)... 104.21.21.176, 172.67.199.170, 2606:4700:3036::ac43:c7aa, ... Connecting to koboldai.org (koboldai.org)|104.21.21.176|:443... connected. HTTP request sent, awaiting response... 302 Moved Temporarily Location: https://raw.githubusercontent.com/henk717/KoboldAI/united/colabkobold.sh [following] --2023-02-24 06:10:37-- https://raw.githubusercontent.com/henk717/KoboldAI/united/colabkobold.sh Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.108.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 6675 (6.5K) [text/plain] Saving to: ‘STDOUT’

2023-02-24 06:10:37 (75.4 MB/s) - written to stdout [6675/6675]

mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/stories/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/models/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/settings/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/softprompts/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/userscripts/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/presets/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/themes/’: File exists Initialized empty Git repository in /content/KoboldAI-Client/.git/ fatal: No such remote: 'origin' Fetching origin remote: Enumerating objects: 6491, done. remote: Total 6491 (delta 0), reused 0 (delta 0), pack-reused 6491 Receiving objects: 100% (6491/6491), 14.04 MiB | 18.96 MiB/s, done. Resolving deltas: 100% (4098/4098), done. From https://github.com/koboldai/KoboldAI-Client

henk717 commented 1 year ago

Known issue with the TPU's https://github.com/KoboldAI/KoboldAI-Client/issues/232

You can use https://koboldai.org/gpucolab which still works

Myuless commented 1 year ago

ok thanks

RowanSkie commented 1 year ago

Hi, I'm getting this issue. @henk717

henk717 commented 1 year ago

Yes they broke it again

DFXLuna commented 1 year ago

Copying my post on discord:

Made some serious progress with TPU stuff, got it to load with V2 of the tpu driver! It worked with the GPTJ 6B model, but it took a long time to load tensors(~11 minutes). However, when trying to run a larger model like Erebus 13B runs out of HBM memory when trying to do an XLA compile after loading the tensors

Here's a notebook that uses my kobold fork: https://colab.research.google.com/drive/15eGJVEtg5zPz_2mR_1j0zfZnr767jC4_?usp=sharing

Here's my kobold fork: https://github.com/DFXLuna/KoboldAI

Here's the log output from the note book:

log text ```javascript Found TPU at: grpc://10.55.173.74:8470 Now we will need your Google Drive to store settings and saves, you must login with the same account you used for Colab. --2023-03-22 19:17:21-- https://raw.githubusercontent.com/DFXLuna/KoboldAI/united/colabkobold.sh Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 6289 (6.1K) [text/plain] Saving to: ‘STDOUT’ - 100%[===================>] 6.14K --.-KB/s in 0s 2023-03-22 19:17:21 (83.3 MB/s) - written to stdout [6289/6289] mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/stories/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/models/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/settings/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/softprompts/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/userscripts/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/presets/’: File exists mkdir: cannot create directory ‘/content/drive/MyDrive/KoboldAI/themes/’: File exists mkdir: cannot create directory ‘/content/KoboldAI-Client’: File exists Reinitialized existing Git repository in /content/KoboldAI-Client/.git/ Fetching origin remote: Enumerating objects: 5, done. remote: Counting objects: 100% (5/5), done. remote: Compressing objects: 100% (1/1), done. remote: Total 3 (delta 2), reused 3 (delta 2), pack-reused 0 Unpacking objects: 100% (3/3), 980 bytes | 980.00 KiB/s, done. From https://github.com/DFXLuna/KoboldAI * [new branch] united -> origin/united Already on 'united' HEAD is now at 07c71ba lower the version to the one that accidentally worked Submodule path 'KoboldAI-Horde-Bridge': checked out '7a7327804ff10182adf8cda48e97784958699a49' cp: cannot stat 'softprompts/*': No such file or directory rm: cannot remove 'stories': Is a directory rm: cannot remove 'userscripts': Is a directory rm: cannot remove 'models': Is a directory rm: cannot remove 'presets': Is a directory rm: cannot remove 'themes': Is a directory ln: failed to create symbolic link 'settings/settings': File exists Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Collecting git+https://github.com/DFXLuna/mesh-transformer-jax@ck (from -r requirements_mtj.txt (line 11)) Cloning https://github.com/DFXLuna/mesh-transformer-jax (to revision ck) to /tmp/pip-req-build-cr648g23 Running command git clone --filter=blob:none --quiet https://github.com/DFXLuna/mesh-transformer-jax /tmp/pip-req-build-cr648g23 Running command git checkout -b ck --track origin/ck Switched to a new branch 'ck' Branch 'ck' set up to track remote branch 'ck' from 'origin'. Resolved https://github.com/DFXLuna/mesh-transformer-jax to commit 89577e3d49aef94179a138260f6fbd6ca2964327 Preparing metadata (setup.py) ... done Requirement already satisfied: torch<1.13,>=1.9 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 1)) (1.12.1) Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 2)) (1.24.2) Requirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 3)) (4.65.0) Requirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 4)) (2.27.1) Collecting dm-haiku==0.0.5 Using cached dm_haiku-0.0.5-py3-none-any.whl (287 kB) Collecting jax==0.3.25 Using cached jax-0.3.25-py3-none-any.whl Collecting jaxlib==0.3.25 Using cached jaxlib-0.3.25-cp39-cp39-manylinux2014_x86_64.whl (71.2 MB) Requirement already satisfied: transformers==4.25.1 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 8)) (4.25.1) Requirement already satisfied: huggingface_hub==0.12.1 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 9)) (0.12.1) Requirement already satisfied: progressbar2 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 10)) (3.38.0) Requirement already satisfied: Flask==2.2.3 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 12)) (2.2.3) Requirement already satisfied: Flask-SocketIO==5.3.2 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 13)) (5.3.2) Requirement already satisfied: python-socketio==5.7.2 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 14)) (5.7.2) Requirement already satisfied: flask-cloudflared==0.0.10 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 15)) (0.0.10) Requirement already satisfied: flask-ngrok in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 16)) (0.0.25) Requirement already satisfied: flask-cors in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 17)) (3.0.10) Requirement already satisfied: eventlet==0.33.3 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 18)) (0.33.3) Requirement already satisfied: dnspython==2.2.1 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 19)) (2.2.1) Requirement already satisfied: lupa==1.10 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 20)) (1.10) Requirement already satisfied: markdown in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 21)) (3.4.1) Requirement already satisfied: bleach==4.1.0 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 22)) (4.1.0) Requirement already satisfied: flask-session==0.4.0 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 23)) (0.4.0) Requirement already satisfied: marshmallow>=3.13 in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 24)) (3.19.0) Requirement already satisfied: apispec-webframeworks in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 25)) (0.5.2) Requirement already satisfied: loguru in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 26)) (0.6.0) Requirement already satisfied: Pillow in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 27)) (8.4.0) Requirement already satisfied: diffusers in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 28)) (0.14.0) Requirement already satisfied: psutil in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 29)) (5.9.4) Requirement already satisfied: ansi2html in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 30)) (1.8.0) Requirement already satisfied: flask_compress in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 31)) (1.13) Requirement already satisfied: ijson in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 32)) (3.2.0.post0) Requirement already satisfied: ftfy in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 33)) (6.1.1) Requirement already satisfied: pydub in /usr/local/lib/python3.9/dist-packages (from -r requirements_mtj.txt (line 34)) (0.25.1) Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.9/dist-packages (from dm-haiku==0.0.5->-r requirements_mtj.txt (line 5)) (0.8.10) Requirement already satisfied: absl-py>=0.7.1 in /usr/local/lib/python3.9/dist-packages (from dm-haiku==0.0.5->-r requirements_mtj.txt (line 5)) (1.4.0) Requirement already satisfied: jmp>=0.0.2 in /usr/local/lib/python3.9/dist-packages (from dm-haiku==0.0.5->-r requirements_mtj.txt (line 5)) (0.0.4) Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from jax==0.3.25->-r requirements_mtj.txt (line 6)) (4.5.0) Requirement already satisfied: opt-einsum in /usr/local/lib/python3.9/dist-packages (from jax==0.3.25->-r requirements_mtj.txt (line 6)) (3.3.0) Requirement already satisfied: scipy>=1.5 in /usr/local/lib/python3.9/dist-packages (from jax==0.3.25->-r requirements_mtj.txt (line 6)) (1.10.1) Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.9/dist-packages (from transformers==4.25.1->-r requirements_mtj.txt (line 8)) (2022.10.31) Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from transformers==4.25.1->-r requirements_mtj.txt (line 8)) (3.10.0) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.9/dist-packages (from transformers==4.25.1->-r requirements_mtj.txt (line 8)) (23.0) Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.9/dist-packages (from transformers==4.25.1->-r requirements_mtj.txt (line 8)) (6.0) Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.9/dist-packages (from transformers==4.25.1->-r requirements_mtj.txt (line 8)) (0.13.2) Requirement already satisfied: Jinja2>=3.0 in /usr/local/lib/python3.9/dist-packages (from Flask==2.2.3->-r requirements_mtj.txt (line 12)) (3.1.2) Requirement already satisfied: click>=8.0 in /usr/local/lib/python3.9/dist-packages (from Flask==2.2.3->-r requirements_mtj.txt (line 12)) (8.1.3) Requirement already satisfied: importlib-metadata>=3.6.0 in /usr/local/lib/python3.9/dist-packages (from Flask==2.2.3->-r requirements_mtj.txt (line 12)) (6.1.0) Requirement already satisfied: itsdangerous>=2.0 in /usr/local/lib/python3.9/dist-packages (from Flask==2.2.3->-r requirements_mtj.txt (line 12)) (2.1.2) Requirement already satisfied: Werkzeug>=2.2.2 in /usr/local/lib/python3.9/dist-packages (from Flask==2.2.3->-r requirements_mtj.txt (line 12)) (2.2.3) Requirement already satisfied: bidict>=0.21.0 in /usr/local/lib/python3.9/dist-packages (from python-socketio==5.7.2->-r requirements_mtj.txt (line 14)) (0.22.1) Requirement already satisfied: python-engineio>=4.3.0 in /usr/local/lib/python3.9/dist-packages (from python-socketio==5.7.2->-r requirements_mtj.txt (line 14)) (4.4.0) Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.9/dist-packages (from eventlet==0.33.3->-r requirements_mtj.txt (line 18)) (1.16.0) Requirement already satisfied: greenlet>=0.3 in /usr/local/lib/python3.9/dist-packages (from eventlet==0.33.3->-r requirements_mtj.txt (line 18)) (2.0.2) Requirement already satisfied: webencodings in /usr/local/lib/python3.9/dist-packages (from bleach==4.1.0->-r requirements_mtj.txt (line 22)) (0.5.1) Requirement already satisfied: cachelib in /usr/local/lib/python3.9/dist-packages (from flask-session==0.4.0->-r requirements_mtj.txt (line 23)) (0.10.2) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->-r requirements_mtj.txt (line 4)) (1.26.15) Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests->-r requirements_mtj.txt (line 4)) (2.0.12) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->-r requirements_mtj.txt (line 4)) (3.4) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->-r requirements_mtj.txt (line 4)) (2022.12.7) Requirement already satisfied: python-utils>=2.3.0 in /usr/local/lib/python3.9/dist-packages (from progressbar2->-r requirements_mtj.txt (line 10)) (3.5.2) Requirement already satisfied: apispec[yaml]>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from apispec-webframeworks->-r requirements_mtj.txt (line 25)) (6.3.0) Requirement already satisfied: brotli in /usr/local/lib/python3.9/dist-packages (from flask_compress->-r requirements_mtj.txt (line 31)) (1.0.9) Requirement already satisfied: wcwidth>=0.2.5 in /usr/local/lib/python3.9/dist-packages (from ftfy->-r requirements_mtj.txt (line 33)) (0.2.6) Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.9/dist-packages (from importlib-metadata>=3.6.0->Flask==2.2.3->-r requirements_mtj.txt (line 12)) (3.15.0) Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.9/dist-packages (from Jinja2>=3.0->Flask==2.2.3->-r requirements_mtj.txt (line 12)) (2.1.2) Installing collected packages: jaxlib, jax, dm-haiku Attempting uninstall: jaxlib Found existing installation: jaxlib 0.4.1 Uninstalling jaxlib-0.4.1: Successfully uninstalled jaxlib-0.4.1 Attempting uninstall: jax Found existing installation: jax 0.4.1 Uninstalling jax-0.4.1: Successfully uninstalled jax-0.4.1 Attempting uninstall: dm-haiku Found existing installation: dm-haiku 0.0.9 Uninstalling dm-haiku-0.0.9: Successfully uninstalled dm-haiku-0.0.9 Successfully installed dm-haiku-0.0.5 jax-0.3.25 jaxlib-0.3.25 Reading package lists... Done Building dependency tree Reading state information... Done netbase is already the newest version (6.1). aria2 is already the newest version (1.35.0-1build1). 0 upgraded, 0 newly installed, 0 to remove and 23 not upgraded. /tools/node/bin/lt -> /tools/node/lib/node_modules/localtunnel/bin/lt.js + localtunnel@2.0.2 updated 1 package in 0.871s Launching KoboldAI with the following options : python3 aiserver.py --model KoboldAI/OPT-13B-Erebus -v --colab 2023-03-22 19:17:45.743956: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-22 19:17:47.707664: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 2023-03-22 19:17:47.707898: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 2023-03-22 19:17:47.707964: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. Colab Check: True MESSAGE | Welcome to KoboldAI! MESSAGE | You have selected the following Model: KoboldAI/OPT-13B-Erebus INIT | Starting | Flask INIT | OK | Flask INIT | Starting | Webserver INIT | OK | Webserver /usr/local/lib/python3.9/dist-packages/haiku/_src/data_structures.py:37: FutureWarning: jax.tree_structure is deprecated, and will be removed in a future release. Use jax.tree_util.tree_structure instead. PyTreeDef = type(jax.tree_structure(None)) Initializing Mesh Transformer JAX, please wait... Using driver version: tpu_driver0.2 Connecting to your Colab instance's TPU Elapsed Time: 0:00:12 [ █ ] to line 1246 1.2636184692382812e-05s MESSAGE | KoboldAI has finished loading and is available at the following link for UI 1: https://vietnam-autos-arrived-canvas.trycloudflare.com/ MESSAGE | KoboldAI has finished loading and is available at the following link for UI 2: https://vietnam-autos-arrived-canvas.trycloudflare.com/new_ui 8 TPU cores will be used to run the model. Running in OPT compatibility mode. Downloading (…)okenizer_config.json: 100% 848/848 [00:00<00:00, 23.2kB/s] Downloading (…)olve/main/vocab.json: 0% 0.00/999k [00:00= 8.49G: reserved 530.00M program 7.97G arguments 0B Output size 0B; shares 0B with arguments. Program hbm requirement 7.97G: global 16.11M HLO temp 7.96G (93.1% utilization: Unpadded (7.41G) Padded (7.95G), 0.0% fragmentation (0B)) Largest program allocations in hbm: 1. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6170 = fusion(fusion.2297, bitcast.1020, bitcast.1014), kind=kOutput, calls=fused_computation.5438 Allocation type: HLO temp ========================== 2. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6169 = fusion(fusion.2297, bitcast.1001, bitcast.995), kind=kOutput, calls=fused_computation.5437 Allocation type: HLO temp ========================== 3. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6207 = fusion(fusion.2297, bitcast.1723, bitcast.1717), kind=kOutput, calls=fused_computation.5475 Allocation type: HLO temp ========================== 4. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6206 = fusion(fusion.2297, bitcast.1704, bitcast.1698), kind=kOutput, calls=fused_computation.5474 Allocation type: HLO temp ========================== 5. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6171 = fusion(fusion.2297, bitcast.1039, bitcast.1033), kind=kOutput, calls=fused_computation.5439 Allocation type: HLO temp ========================== 6. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6172 = fusion(fusion.2297, bitcast.1058, bitcast.1052), kind=kOutput, calls=fused_computation.5440 Allocation type: HLO temp ========================== 7. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6173 = fusion(fusion.2297, bitcast.1077, bitcast.1071), kind=kOutput, calls=fused_computation.5441 Allocation type: HLO temp ========================== 8. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6174 = fusion(fusion.2297, bitcast.1096, bitcast.1090), kind=kOutput, calls=fused_computation.5442 Allocation type: HLO temp ========================== 9. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6175 = fusion(fusion.2297, bitcast.1115, bitcast.1109), kind=kOutput, calls=fused_computation.5443 Allocation type: HLO temp ========================== 10. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6176 = fusion(fusion.2297, bitcast.1134, bitcast.1128), kind=kOutput, calls=fused_computation.5444 Allocation type: HLO temp ========================== 11. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6177 = fusion(fusion.2297, bitcast.1153, bitcast.1147), kind=kOutput, calls=fused_computation.5445 Allocation type: HLO temp ========================== 12. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6178 = fusion(fusion.2297, bitcast.1172, bitcast.1166), kind=kOutput, calls=fused_computation.5446 Allocation type: HLO temp ========================== 13. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6179 = fusion(fusion.2297, bitcast.1191, bitcast.1185), kind=kOutput, calls=fused_computation.5447 Allocation type: HLO temp ========================== 14. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6180 = fusion(fusion.2297, bitcast.1210, bitcast.1204), kind=kOutput, calls=fused_computation.5448 Allocation type: HLO temp ========================== 15. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6181 = fusion(fusion.2297, bitcast.1229, bitcast.1223), kind=kOutput, calls=fused_computation.5449 Allocation type: HLO temp ========================== 16. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6182 = fusion(fusion.2297, bitcast.1248, bitcast.1242), kind=kOutput, calls=fused_computation.5450 Allocation type: HLO temp ========================== 17. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6183 = fusion(fusion.2297, bitcast.1267, bitcast.1261), kind=kOutput, calls=fused_computation.5451 Allocation type: HLO temp ========================== 18. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6184 = fusion(fusion.2297, bitcast.1286, bitcast.1280), kind=kOutput, calls=fused_computation.5452 Allocation type: HLO temp ========================== 19. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6185 = fusion(fusion.2297, bitcast.1305, bitcast.1299), kind=kOutput, calls=fused_computation.5453 Allocation type: HLO temp ========================== 20. Size: 80.00M Operator: op_name="xmap(generate_static)/jit(main)/while/body/thd,Thd->htT/jit(_einsum)/dot_general[dimension_numbers=(((2,), (2,)), ((1,), (1,))) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.9/dist-packages/mesh_transformer/layers.py" source_line=563 Shape: f32[5,2047,2047]{1,2,0:T(8,128)} Unpadded size: 79.92M Extra memory due to padding: 80.0K (1.0x expansion) XLA label: fusion.6186 = fusion(fusion.2297, bitcast.1324, bitcast.1318), kind=kOutput, calls=fused_computation.5454 Allocation type: HLO temp ========================== INIT | Closed | Webserver ```