gfx906 (AMD MI60) is failing on run_and_save_benchmarks.sh and llama.cpp

Said-Akbar commented 6 days ago

Hi @lamikr,

I built rocm_sdk_builder on a freshly installed Ubuntu 24.04.1. It took 5 hours, 120GB of storage and many hours of fixing small issues during building the repo (reference: https://github.com/lamikr/rocm_sdk_builder/issues/175). Also, I chose gfx906 from ./babs.sh -c.

When I ran ./run_and_save_benchmarks.sh, I got this message.

./run_and_save_benchmarks.sh
Timestamp for benchmark results: 20241121_190404
Saving to file: 20241121_190404_cpu_vs_gpu_simple.txt
Benchmarking CPU and GPUs
Pytorch version: 2.4.1
ROCM HIP version: 6.1.40093-61a06a2f8
       Device:  AMD Ryzen 9 5950X 16-Core Processor
    'CPU time: 26.503 sec
       Device: AMD Radeon Graphics
    'GPU time: 0.399 sec
       Device: AMD Radeon Graphics
    'GPU time: 0.353 sec
Benchmark ready

Saving to file: 20241121_190404_pytorch_dot_products.txt
Pytorch version: 2.4.1
dot product calculation test
tensor([[[ 0.2042, -0.5683,  0.5711,  1.5666, -0.8859, -0.4255, -0.6103,
          -0.5932],
         [-0.1816, -1.0552,  0.3676,  2.1399, -0.8622,  0.1185, -0.4614,
          -0.4577],
         [ 0.2491, -0.5238,  0.5873,  1.5027, -0.8808, -0.4906, -0.6309,
          -0.6083]],

        [[-0.0812,  0.5027, -0.0134, -0.1771, -1.6389,  0.0154, -1.1964,
          -0.3948],
         [-0.3459, -0.4265,  0.0969,  0.0608, -0.9923, -0.4199, -0.7190,
          -0.0208],
         [-0.2615, -0.6958,  0.1066, -0.1948, -1.2152, -0.1223, -0.6278,
           0.1627]]], device='cuda:0')

Benchmarking cuda and cpu with Default, Math, Flash Attention amd Memory pytorch backends
Device: AMD Radeon Graphics / cuda:0
    Default benchmark:
:0:/home/saidp/Downloads/rocm_sdk_builder/src_projects/clr/hipamd/src/hip_global.cpp:114 : 8471950880 us: [pid:454884 tid:0x7ad2a9db0b80] Cannot find Symbol with name: Cijk_Alik_Bljk_HHS_BH_MT128x64x16_SE_APM1_AF0EM2_AF1EM1_AMAS3_ASAE01_ASCE01_ASEM2_BL1_BS1_DTLA0_DTLB0_EPS1_FL1_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA1_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT8_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG16_16_1_WGM1

Note the error at the bottom 'Cannot find Symbol with name'. I thought this would not be an issue with llama.cpp. However, I got a similar error in llama.cpp as well (I built it using ./babs.sh -b binfo/extra/ai_tools.blist).

source /opt/rocm_sdk_612/bin/env_rocm.sh
llama-server -m /media/saidp/datasets/text_generation/models/Qwen2.5-7B-Instruct-Q8_0/Qwen2.5-7B-Instruct-Q8_0.gguf -c 2048 -ngl 99 --metrics
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon Graphics, compute capability 9.0, VMM: no
  Device 1: AMD Radeon Graphics, compute capability 9.0, VMM: no
build: 3901 (49f4671b) with cc (Ubuntu 13.2.0-23ubuntu4) 13.2.0 for x86_64-linux-gnu
system info: n_threads = 16, n_threads_batch = 16, total_threads = 32

system_info: n_threads = 16 (n_threads_batch = 16) / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | 

main: HTTP server is listening, hostname: 127.0.0.1, port: 8080, http threads: 31
main: loading model
llama_model_loader: loaded meta data with 38 key-value pairs and 339 tensors from /media/saidp/datasets/text_generation/models/Qwen2.5-7B-Instruct-Q8_0/Qwen2.5-7B-Instruct-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 7B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Qwen2.5
llama_model_loader: - kv   5:                         general.size_label str              = 7B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                       general.license.link str              = https://huggingface.co/Qwen/Qwen2.5-7...
llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
llama_model_loader: - kv   9:                  general.base_model.0.name str              = Qwen2.5 7B
llama_model_loader: - kv  10:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  11:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen2.5-7B
llama_model_loader: - kv  12:                               general.tags arr[str,2]       = ["chat", "text-generation"]
llama_model_loader: - kv  13:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv  14:                          qwen2.block_count u32              = 28
llama_model_loader: - kv  15:                       qwen2.context_length u32              = 32768
llama_model_loader: - kv  16:                     qwen2.embedding_length u32              = 3584
llama_model_loader: - kv  17:                  qwen2.feed_forward_length u32              = 18944
llama_model_loader: - kv  18:                 qwen2.attention.head_count u32              = 28
llama_model_loader: - kv  19:              qwen2.attention.head_count_kv u32              = 4
llama_model_loader: - kv  20:                       qwen2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  21:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  22:                          general.file_type u32              = 7
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  30:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  33:               general.quantization_version u32              = 2
llama_model_loader: - kv  34:                      quantize.imatrix.file str              = /models_out/Qwen2.5-7B-Instruct-GGUF/...
llama_model_loader: - kv  35:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt
llama_model_loader: - kv  36:             quantize.imatrix.entries_count i32              = 196
llama_model_loader: - kv  37:              quantize.imatrix.chunks_count i32              = 128
llama_model_loader: - type  f32:  141 tensors
llama_model_loader: - type q8_0:  198 tensors
llm_load_vocab: special tokens cache size = 22
llm_load_vocab: token to piece cache size = 0.9310 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = qwen2
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 152064
llm_load_print_meta: n_merges         = 151387
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 3584
llm_load_print_meta: n_layer          = 28
llm_load_print_meta: n_head           = 28
llm_load_print_meta: n_head_kv        = 4
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 7
llm_load_print_meta: n_embd_k_gqa     = 512
llm_load_print_meta: n_embd_v_gqa     = 512
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 18944
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q8_0
llm_load_print_meta: model params     = 7.62 B
llm_load_print_meta: model size       = 7.54 GiB (8.50 BPW) 
llm_load_print_meta: general.name     = Qwen2.5 7B Instruct
llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOS token        = 151645 '<|im_end|>'
llm_load_print_meta: PAD token        = 151643 '<|endoftext|>'
llm_load_print_meta: LF token         = 148848 'ÄĬ'
llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
llm_load_print_meta: EOG token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOG token        = 151645 '<|im_end|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: ggml ctx size =    0.45 MiB
llm_load_tensors: offloading 28 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 29/29 layers to GPU
llm_load_tensors:      ROCm0 buffer size =  3542.78 MiB
llm_load_tensors:      ROCm1 buffer size =  3622.66 MiB
llm_load_tensors:        CPU buffer size =   552.23 MiB
......................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      ROCm0 KV buffer size =    60.00 MiB
llama_kv_cache_init:      ROCm1 KV buffer size =    52.00 MiB
llama_new_context_with_model: KV self size  =  112.00 MiB, K (f16):   56.00 MiB, V (f16):   56.00 MiB
llama_new_context_with_model:  ROCm_Host  output buffer size =     1.16 MiB
llama_new_context_with_model: pipeline parallelism enabled (n_copies=4)
llama_new_context_with_model:      ROCm0 compute buffer size =   184.01 MiB
llama_new_context_with_model:      ROCm1 compute buffer size =   348.02 MiB
llama_new_context_with_model:  ROCm_Host compute buffer size =    23.02 MiB
llama_new_context_with_model: graph nodes  = 986
llama_new_context_with_model: graph splits = 3
llama_init_from_gpt_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
:0:/home/saidp/Downloads/rocm_sdk_builder/src_projects/clr/hipamd/src/hip_global.cpp:114 : 10662878012 us: [pid:465832 tid:0x7268ce2a2c40] Cannot find Symbol with name: Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1
Aborted (core dumped)

llama.cpp is failing with a similar error. Note that this llama.cpp worked with the CPU when I do not set the ngl parameter (layer offloading). Please let me know if there is a fix.

Said-Akbar commented 5 days ago

@lamikr , That error line comes from https://github.com/ROCm/clr/blob/rocm-6.1.x/hipamd/src/hip_global.cpp#L114 .

But I am not sure how to fix my issue above. Please, let me know if you have time to review this today. Thanks!

lamikr commented 4 days ago

Hi, unfortunately I do not have myself the gfx906 for debug, so I only added added some patches that would be needed at least to get it build and start testing and added it's support as an experimental.

About your error, I have not never seen that kind of error, but it could be some kind of misconfiguration in rocBLAS related to src_projects/rocBLAS/library/src/blas3/Tensile/Logic/asm_full/vega10/vega10_Cijk_Alik_Bljk_HB_GB.yaml

But let's try to check first couple of basic issues step by step so I get basic info.

1) Can you paste me first the output of rocminfo command? I am interested in whether it detects your gpu and what information it shows from it.

2) Then are you able to build and run this test these test apps:

/opt/rocm_sdk_612/docs/examples/hipcc/hello_world /opt/rocm_sdk_612/docs/examples/opencl/check_opencl_caps

Said-Akbar commented 4 days ago

Hello @lamikr, Sure, here is the output of rocminfo.

``` rocminfo ROCk module is loaded ===================== HSA System Attributes ===================== Runtime Version: 1.1 Runtime Ext Version: 1.4 System Timestamp Freq.: 1000.000000MHz Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) Machine Model: LARGE System Endianness: LITTLE Mwaitx: DISABLED DMAbuf Support: YES ========== HSA Agents ========== ******* Agent 1 ******* Name: AMD Ryzen 9 5950X 16-Core Processor Uuid: CPU-XX Marketing Name: AMD Ryzen 9 5950X 16-Core Processor Vendor Name: CPU Feature: None specified Profile: FULL_PROFILE Float Round Mode: NEAR Max Queue Number: 0(0x0) Queue Min Size: 0(0x0) Queue Max Size: 0(0x0) Queue Type: MULTI Node: 0 Device Type: CPU Cache Info: L1: 32768(0x8000) KB Chip ID: 0(0x0) ASIC Revision: 0(0x0) Cacheline Size: 64(0x40) Max Clock Freq. (MHz): 3400 BDFID: 0 Internal Node ID: 0 Compute Unit: 32 SIMDs per CU: 0 Shader Engines: 0 Shader Arrs. per Eng.: 0 WatchPts on Addr. Ranges:1 Features: None Pool Info: Pool 1 Segment: GLOBAL; FLAGS: FINE GRAINED Size: 98773496(0x5e329f8) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:4KB Alloc Alignment: 4KB Accessible by all: TRUE Pool 2 Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED Size: 98773496(0x5e329f8) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:4KB Alloc Alignment: 4KB Accessible by all: TRUE Pool 3 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 98773496(0x5e329f8) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:4KB Alloc Alignment: 4KB Accessible by all: TRUE ISA Info: ******* Agent 2 ******* Name: gfx906 Uuid: GPU-161620e172e17d3d Marketing Name: AMD Radeon Graphics Vendor Name: AMD Feature: KERNEL_DISPATCH Profile: BASE_PROFILE Float Round Mode: NEAR Max Queue Number: 128(0x80) Queue Min Size: 64(0x40) Queue Max Size: 131072(0x20000) Queue Type: MULTI Node: 1 Device Type: GPU Cache Info: L1: 16(0x10) KB L2: 8192(0x2000) KB Chip ID: 26273(0x66a1) ASIC Revision: 1(0x1) Cacheline Size: 64(0x40) Max Clock Freq. (MHz): 1800 BDFID: 3328 Internal Node ID: 1 Compute Unit: 64 SIMDs per CU: 4 Shader Engines: 4 Shader Arrs. per Eng.: 1 WatchPts on Addr. Ranges:4 Coherent Host Access: FALSE Features: KERNEL_DISPATCH Fast F16 Operation: TRUE Wavefront Size: 64(0x40) Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Max Waves Per CU: 40(0x28) Max Work-item Per CU: 2560(0xa00) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) Max fbarriers/Workgrp: 32 Packet Processor uCode:: 471 SDMA engine uCode:: 145 IOMMU Support:: None Pool Info: Pool 1 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 33538048(0x1ffc000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:2048KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 2 Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED Size: 33538048(0x1ffc000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:2048KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 3 Segment: GROUP Size: 64(0x40) KB Allocatable: FALSE Alloc Granule: 0KB Alloc Recommended Granule:0KB Alloc Alignment: 0KB Accessible by all: FALSE ISA Info: ISA 1 Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- Machine Models: HSA_MACHINE_MODEL_LARGE Profiles: HSA_PROFILE_BASE Default Rounding Mode: NEAR Default Rounding Mode: NEAR Fast f16: TRUE Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) FBarrier Max Size: 32 ******* Agent 3 ******* Name: gfx906 Uuid: GPU-915e294172fd62d2 Marketing Name: AMD Radeon Graphics Vendor Name: AMD Feature: KERNEL_DISPATCH Profile: BASE_PROFILE Float Round Mode: NEAR Max Queue Number: 128(0x80) Queue Min Size: 64(0x40) Queue Max Size: 131072(0x20000) Queue Type: MULTI Node: 2 Device Type: GPU Cache Info: L1: 16(0x10) KB L2: 8192(0x2000) KB Chip ID: 26273(0x66a1) ASIC Revision: 1(0x1) Cacheline Size: 64(0x40) Max Clock Freq. (MHz): 1800 BDFID: 4096 Internal Node ID: 2 Compute Unit: 64 SIMDs per CU: 4 Shader Engines: 4 Shader Arrs. per Eng.: 1 WatchPts on Addr. Ranges:4 Coherent Host Access: FALSE Features: KERNEL_DISPATCH Fast F16 Operation: TRUE Wavefront Size: 64(0x40) Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Max Waves Per CU: 40(0x28) Max Work-item Per CU: 2560(0xa00) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) Max fbarriers/Workgrp: 32 Packet Processor uCode:: 471 SDMA engine uCode:: 145 IOMMU Support:: None Pool Info: Pool 1 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 33538048(0x1ffc000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:2048KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 2 Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED Size: 33538048(0x1ffc000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:2048KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 3 Segment: GROUP Size: 64(0x40) KB Allocatable: FALSE Alloc Granule: 0KB Alloc Recommended Granule:0KB Alloc Alignment: 0KB Accessible by all: FALSE ISA Info: ISA 1 Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- Machine Models: HSA_MACHINE_MODEL_LARGE Profiles: HSA_PROFILE_BASE Default Rounding Mode: NEAR Default Rounding Mode: NEAR Fast f16: TRUE Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) FBarrier Max Size: 32 *** Done *** ```

Said-Akbar commented 4 days ago

tests:

cd /opt/rocm_sdk_612/docs/examples/hipcc/hello_world/
./build.sh 
rm -f ./hello_world
rm -f hello_world.o
rm -f /opt/rocm_sdk_612/src/*.o
/opt/rocm_sdk_612/bin/hipcc -g -fPIE   -c -o hello_world.o hello_world.cpp
/opt/rocm_sdk_612/bin/hipcc hello_world.o -fPIE -o hello_world
./hello_world
 System minor: 0
 System major: 9
 Agent name: AMD Radeon Graphics
Kernel input: GdkknVnqkc
Expecting that kernel increases each character from input string by one
Kernel output string: HelloWorld
Output string matched with HelloWorld
Test ok!

Said-Akbar commented 4 days ago

Opencl test:

cd /opt/rocm_sdk_612/docs/examples/opencl/check_opencl_caps
make
/check_opencl_caps 
number of opencl platform devices: 1
==============================
Platform id: 0
AMD Accelerated Parallel Processing
Advanced Micro Devices, Inc.
OpenCL 2.1 AMD-APP (3614.0)
FULL_PROFILE
cl_khr_icd cl_amd_event_callback 
Number of devices found for platform: 2
    ---------------------------
    Device id: 0
    CL_DEVICE_VENDOR_ID: 0x1002
    CL_DEVICE_TYPE:  GPU
    CL_DEVICE_VENDOR_ID: 0x1002
    CL_DEVICE_MAX_COMPUTE_UNITS: 0x40
    CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS: 0x3
    CL_DEVICE_MAX_WORK_GROUP_SIZE: 0x3
    CL_DEVICE_PREFERRED_VECTOR_WIDTH_CHAR: 0x4
    CL_DEVICE_PREFERRED_VECTOR_WIDTH_SHORT: 0x2
    todo more information...
   ---------------------------
    ---------------------------
    Device id: 1
    CL_DEVICE_VENDOR_ID: 0x1002
    CL_DEVICE_TYPE:  GPU
    CL_DEVICE_VENDOR_ID: 0x1002
    CL_DEVICE_MAX_COMPUTE_UNITS: 0x40
    CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS: 0x3
    CL_DEVICE_MAX_WORK_GROUP_SIZE: 0x3
    CL_DEVICE_PREFERRED_VECTOR_WIDTH_CHAR: 0x4
    CL_DEVICE_PREFERRED_VECTOR_WIDTH_SHORT: 0x2
    todo more information...
   ---------------------------
==============================

Said-Akbar commented 4 days ago

by the way, gfx906 has 'Vega 20' GPUs, but not 'Vega 10' GPUs. Not sure if some instruction that does not exist in gfx906 is being called from llama.cpp.

Said-Akbar commented 4 days ago

Here is the app crash log :

``` cat /var/crash/_opt_rocm_sdk_612_bin_llama-server.1000.crash ApportVersion: 2.28.1-0ubuntu3.1 CasperMD5CheckResult: pass Disassembly: => 0x7b73e609eb1c <__GI___pthread_kill+284>: mov %eax,%r14d 0x7b73e609eb1f <__GI___pthread_kill+287>: neg %r14d 0x7b73e609eb22 <__GI___pthread_kill+290>: cmp $0xfffff000,%eax 0x7b73e609eb27 <__GI___pthread_kill+295>: mov $0x0,%eax 0x7b73e609eb2c <__GI___pthread_kill+300>: cmovbe %eax,%r14d 0x7b73e609eb30 <__GI___pthread_kill+304>: jmp 0x7b73e609eab0 <__GI___pthread_kill+176> 0x7b73e609eb35 <__GI___pthread_kill+309>: nopl (%rax) 0x7b73e609eb38 <__GI___pthread_kill+312>: mov %r13,%rdi 0x7b73e609eb3b <__GI___pthread_kill+315>: call 0x7b73e6098ed0 <__GI___lll_lock_wait_private> 0x7b73e609eb40 <__GI___pthread_kill+320>: jmp 0x7b73e609ea7e <__GI___pthread_kill+126> 0x7b73e609eb45 <__GI___pthread_kill+325>: nopl (%rax) 0x7b73e609eb48 <__GI___pthread_kill+328>: mov %r13,%rdi 0x7b73e609eb4b <__GI___pthread_kill+331>: call 0x7b73e6098f90 <__GI___lll_lock_wake_private> 0x7b73e609eb50 <__GI___pthread_kill+336>: jmp 0x7b73e609ea99 <__GI___pthread_kill+153> 0x7b73e609eb55 <__GI___pthread_kill+341>: call 0x7b73e6137e90 <__stack_chk_fail> 0x7b73e609eb5a: nopw 0x0(%rax,%rax,1) InstallationDate: Installed on 2024-11-20 (4 days ago) InstallationMedia: Ubuntu 24.04.1 LTS "Noble Numbat" - Release amd64 (20240827.1) JournalErrors: -- No entries -- ProcCpuinfoMinimal: processor : 31 vendor_id : AuthenticAMD cpu family : 25 model : 33 model name : AMD Ryzen 9 5950X 16-Core Processor stepping : 0 microcode : 0xa201016 cpu MHz : 2200.000 cache size : 512 KB physical id : 0 siblings : 32 core id : 15 cpu cores : 16 apicid : 31 initial apicid : 31 fpu : yes fpu_exception : yes cpuid level : 16 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm debug_swap bugs : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso bogomips : 6799.86 TLB size : 2560 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14] ProcVersionSignature: Ubuntu 6.8.0-49.49-generic 6.8.12 Registers: rax 0x0 0 rbx 0x189e 6302 rcx 0x7b73e609eb1c 135737710865180 rdx 0x6 6 rsi 0x189e 6302 rdi 0x189e 6302 rbp 0x7ffe465f42b0 0x7ffe465f42b0 rsp 0x7ffe465f4270 0x7ffe465f4270 r8 0x57 87 r9 0x0 0 r10 0x8 8 r11 0x246 582 r12 0x6 6 r13 0x0 0 r14 0x16 22 r15 0x627599932160 108257227252064 rip 0x7b73e609eb1c 0x7b73e609eb1c <__GI___pthread_kill+284> eflags 0x246 [ PF ZF IF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 fs_base 0x7b73e628ec40 135737712897088 gs_base 0x0 0 Stacktrace: #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=) at ./nptl/pthread_kill.c:44 tid = ret = 0 pd = old_mask = {__val = {0}} ret = pd = old_mask = ret = tid = ret = resultvar = resultvar = __arg3 = __arg2 = __arg1 = _a3 = _a2 = _a1 = __futex = resultvar = __arg3 = __arg2 = __arg1 = _a3 = _a2 = _a1 = __futex = __private = __oldval = #1 __pthread_kill_internal (signo=6, threadid=) at ./nptl/pthread_kill.c:78 No locals. #2 __GI___pthread_kill (threadid=, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 No locals. #3 0x00007b73e604526e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 ret = #4 0x00007b73e60288ff in __GI_abort () at ./stdlib/abort.c:79 save_stage = 1 act = {__sigaction_handler = {sa_handler = 0x20, sa_sigaction = 0x20}, sa_mask = {__val = {108257077776080, 59, 14422071311227648676, 140730079068992, 135737623843607, 135737712327360, 18446744073709551512, 108257227250632, 303, 108257227129664, 303, 303, 2, 14, 6983489619661282816, 140730079069200}}, sa_flags = -1718412448, sa_restorer = 0x7ffe465f4410} #5 0x00007b73e0a2e0ff in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6 No symbol table info available. #6 0x00007b73e0b2b201 in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6 No symbol table info available. #7 0x00007b73e0ad5983 in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6 No symbol table info available. #8 0x00007b73e0c9f9ed in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6 No symbol table info available. #9 0x00007b73e0c799df in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6 No symbol table info available. #10 0x00007b73e204161e in Tensile::hip::SolutionAdapter::getKernel(ihipModuleSymbol_t*&, std::__cxx11::basic_string, std::allocator > const&) () from /opt/rocm_sdk_612/lib64/librocblas.so.4 No symbol table info available. #11 0x00007b73e2042257 in Tensile::hip::SolutionAdapter::launchKernel(Tensile::KernelInvocation const&, ihipStream_t*, ihipEvent_t*, ihipEvent_t*) () from /opt/rocm_sdk_612/lib64/librocblas.so.4 No symbol table info available. #12 0x00007b73e2042a68 in Tensile::hip::SolutionAdapter::launchKernels(std::vector > const&, ihipStream_t*, ihipEvent_t*, ihipEvent_t*) () from /opt/rocm_sdk_612/lib64/librocblas.so.4 No symbol table info available. #13 0x00007b73e184224f in rocblas_status_ runContractionProblem<_Float16, _Float16, _Float16, _Float16, _Float16, _Float16>(RocblasContractionProblem<_Float16, _Float16, _Float16, _Float16, _Float16, _Float16> const&, rocblas_gemm_algo_, int) () from /opt/rocm_sdk_612/lib64/librocblas.so.4 No symbol table info available. #14 0x00007b73e1968b6c in rocblas_status_ gemm_ex_batched_template<_Float16, _Float16, _Float16>(_rocblas_handle*, rocblas_operation_, rocblas_operation_, int, int, int, _Float16 const*, _Float16 const* const*, long, int, long, _Float16 const* const*, long, int, long, _Float16 const*, _Float16 const* const*, long, int, long, _Float16* const*, long, int, long, int, rocblas_gemm_algo_, int, rocblas_gemm_flags_) () from /opt/rocm_sdk_612/lib64/librocblas.so.4 No symbol table info available. #15 0x00007b73e1966f0d in rocblas_status_ gemm_ex_typecasting(_rocblas_handle*, rocblas_operation_, rocblas_operation_, int, int, int, void const*, void const*, long, int, long, void const*, long, int, long, void const*, void const*, long, int, long, void*, long, int, long, int, rocblas_gemm_algo_, int, rocblas_gemm_flags_) () from /opt/rocm_sdk_612/lib64/librocblas.so.4 No symbol table info available. #16 0x00007b73e1960db2 in rocblas_status_ rocblas_gemm_ex_template(_rocblas_handle*, rocblas_operation_, rocblas_operation_, int, int, int, void const*, void const*, rocblas_datatype_, long, int, long, void const*, rocblas_datatype_, long, int, long, void const*, void const*, rocblas_datatype_, long, int, long, void*, rocblas_datatype_, long, int, long, int, rocblas_datatype_, rocblas_gemm_algo_, int, unsigned int) () from /opt/rocm_sdk_612/lib64/librocblas.so.4 No symbol table info available. #17 0x00007b73e195ff43 in rocblas_gemm_batched_ex () from /opt/rocm_sdk_612/lib64/librocblas.so.4 No symbol table info available. #18 0x00007b73e67b3e38 in hipblasGemmBatchedEx () from /opt/rocm_sdk_612/lib64/libhipblas.so.2 No symbol table info available. #19 0x00007b73e6903068 in ggml_cuda_mul_mat_batched_cublas(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor*) () from /opt/rocm_sdk_612/lib64/libggml.so No symbol table info available. #20 0x00007b73e68f33b9 in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /opt/rocm_sdk_612/lib64/libggml.so No symbol table info available. #21 0x00007b73e6873703 in ggml_backend_sched_graph_compute_async () from /opt/rocm_sdk_612/lib64/libggml.so No symbol table info available. #22 0x00007b73e8d6dfd2 in llama_decode () from /opt/rocm_sdk_612/lib64/libllama.so No symbol table info available. #23 0x0000627586b5d704 in llama_init_from_gpt_params(gpt_params&) () No symbol table info available. #24 0x0000627586af0822 in server_context::load_model(gpt_params const&) () No symbol table info available. #25 0x0000627586aa2820 in main () No symbol table info available. StacktraceAddressSignature: /opt/rocm_sdk_612/bin/llama-server:6:/usr/lib/x86_64-linux-gnu/libc.so.6+1d26e:/usr/lib/x86_64-linux-gnu/libc.so.6+8ff:/opt/rocm_sdk_612/lib64/libamdhip64.so.6.1.40093-61a06a2f8+b0ff:/opt/rocm_sdk_612/lib64/libamdhip64.so.6.1.40093-61a06a2f8+108201:/opt/rocm_sdk_612/lib64/libamdhip64.so.6.1.40093-61a06a2f8+b2983:/opt/rocm_sdk_612/lib64/libamdhip64.so.6.1.40093-61a06a2f8+27c9ed:/opt/rocm_sdk_612/lib64/libamdhip64.so.6.1.40093-61a06a2f8+2569df:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+fae61e:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+faf257:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+fafa68:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+7af24f:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+8d5b6c:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+8d3f0d:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+8cddb2:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+8ccf43 StacktraceTop: ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6 ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6 ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6 ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6 ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6 Tags: noble wayland-session ThreadStacktrace: . Thread 35 (Thread 0x7b71930006c0 (LWP 6336)): #0 0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = resultvar = __arg6 = __arg5 = __arg4 = __arg3 = __arg2 = __arg1 = _a6 = _a5 = _a4 = _a3 = _a2 = _a1 = #1 __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87 err = clockbit = 256 op = 393 err = clockbit = op = #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 No locals. #3 0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b7192fff8c0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 56, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0} err = g = 0 flags = g1_start = maxspin = 0 signals = result = 0 wseq = 56 seq = 28 private = 0 maxspin = err = result = wseq = g = seq = flags = private = signals = done = g1_start = spin = buffer = cbuffer = s = #4 ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627 No locals. #5 0x0000627586ad07ca in std::thread::_State_impl > >::_M_run() () No symbol table info available. #6 0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 No symbol table info available. #7 0x00007b73e609ca94 in start_thread (arg=) at ./nptl/pthread_create.c:447 ret = pd = out = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727727773376, 1370407099822688401, 135727727773376, -160, 0, 135728109450704, 1370407099742996625, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = #8 0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 No locals. . Thread 34 (Thread 0x7b71926006c0 (LWP 6337)): #0 0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = resultvar = __arg6 = __arg5 = __arg4 = __arg3 = __arg2 = __arg1 = _a6 = _a5 = _a4 = _a3 = _a2 = _a1 = #1 __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87 err = clockbit = 256 op = 393 err = clockbit = op = #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 No locals. #3 0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71925ff8c0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 58, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0} err = g = 0 flags = g1_start = maxspin = 0 signals = result = 0 wseq = 58 seq = 29 private = 0 maxspin = err = result = wseq = g = seq = flags = private = signals = done = g1_start = spin = buffer = cbuffer = s = #4 ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627 No locals. #5 0x0000627586ad07ca in std::thread::_State_impl > >::_M_run() () No symbol table info available. #6 0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 No symbol table info available. #7 0x00007b73e609ca94 in start_thread (arg=) at ./nptl/pthread_create.c:447 ret = pd = out = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727717287616, 1370405725433153681, 135727717287616, -160, 0, 135728109450704, 1370405725353461905, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = #8 0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 No locals. . Thread 33 (Thread 0x7b7194e006c0 (LWP 6333)): #0 0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = resultvar = __arg6 = __arg5 = __arg4 = __arg3 = __arg2 = __arg1 = _a6 = _a5 = _a4 = _a3 = _a2 = _a1 = #1 __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87 err = clockbit = 256 op = 393 err = clockbit = op = #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 No locals. #3 0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b7194dff8c0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 50, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0} err = g = 0 flags = g1_start = maxspin = 0 signals = result = 0 wseq = 50 seq = 25 private = 0 maxspin = err = result = wseq = g = seq = flags = private = signals = done = g1_start = spin = buffer = cbuffer = s = #4 ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627 No locals. #5 0x0000627586ad07ca in std::thread::_State_impl > >::_M_run() () No symbol table info available. #6 0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 No symbol table info available. #7 0x00007b73e609ca94 in start_thread (arg=) at ./nptl/pthread_create.c:447 ret = pd = out = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727759230656, 1370402426898270353, 135727759230656, -160, 0, 135728109450704, 1370402426818578577, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = #8 0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 No locals. . Thread 32 (Thread 0x7b7196c006c0 (LWP 6330)): #0 0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = resultvar = __arg6 = __arg5 = __arg4 = __arg3 = __arg2 = __arg1 = _a6 = _a5 = _a4 = _a3 = _a2 = _a1 = #1 __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87 err = clockbit = 256 op = 393 err = clockbit = op = #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 No locals. #3 0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b7196bff8c0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 44, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0} err = g = 0 flags = g1_start = maxspin = 0 signals = result = 0 wseq = 44 seq = 22 private = 0 maxspin = err = result = wseq = g = seq = flags = private = signals = done = g1_start = spin = buffer = cbuffer = s = #4 ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627 No locals. #5 0x0000627586ad07ca in std::thread::_State_impl > >::_M_run() () No symbol table info available. #6 0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 No symbol table info available. #7 0x00007b73e609ca94 in start_thread (arg=) at ./nptl/pthread_create.c:447 ret = pd = out = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727790687936, 1370397753973852305, 135727790687936, -160, 0, 135728109450704, 1370397753894160529, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = #8 0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 No locals. . Thread 31 (Thread 0x7b71958006c0 (LWP 6332)): #0 0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = resultvar = __arg6 = __arg5 = __arg4 = __arg3 = __arg2 = __arg1 = _a6 = _a5 = _a4 = _a3 = _a2 = _a1 = #1 __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87 err = clockbit = 256 op = 393 err = clockbit = op = #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 No locals. #3 0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71957ff8c0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 48, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0} err = g = 0 flags = g1_start = maxspin = 0 signals = result = 0 wseq = 48 seq = 24 private = 0 maxspin = err = result = wseq = g = seq = flags = private = signals = done = g1_start = spin = buffer = cbuffer = s = #4 ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627 No locals. #5 0x0000627586ad07ca in std::thread::_State_impl > >::_M_run() () No symbol table info available. #6 0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 No symbol table info available. #7 0x00007b73e609ca94 in start_thread (arg=) at ./nptl/pthread_create.c:447 ret = pd = out = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727769716416, 1370403801287805073, 135727769716416, -160, 0, 135728109450704, 1370403801208113297, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = #8 0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 No locals. . Thread 30 (Thread 0x7b71962006c0 (LWP 6331)): #0 0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = resultvar = __arg6 = __arg5 = __arg4 = __arg3 = __arg2 = __arg1 = _a6 = _a5 = _a4 = _a3 = _a2 = _a1 = #1 __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87 err = clockbit = 256 op = 393 err = clockbit = op = #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 No locals. #3 0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71961ff8c0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 46, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0} err = g = 0 flags = g1_start = maxspin = 0 signals = result = 0 wseq = 46 seq = 23 private = 0 maxspin = err = result = wseq = g = seq = flags = private = signals = done = g1_start = spin = buffer = cbuffer = s = #4 ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627 No locals. #5 0x0000627586ad07ca in std::thread::_State_impl > >::_M_run() () No symbol table info available. #6 0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 No symbol table info available. #7 0x00007b73e609ca94 in start_thread (arg=) at ./nptl/pthread_create.c:447 ret = pd = out = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727780202176, 1370396379584317585, 135727780202176, -160, 0, 135728109450704, 1370396379504625809, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = #8 0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 No locals. . Thread 29 (Thread 0x7b7193a006c0 (LWP 6335)): #0 0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = resultvar = __arg6 = __arg5 = __arg4 = __arg3 = __arg2 = __arg1 = _a6 = _a5 = _a4 = _a3 = _a2 = _a1 = #1 __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87 err = clockbit = 256 op = 393 err = clockbit = op = #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 No locals. #3 0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71939ff8c0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 52, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0} err = g = 0 flags = g1_start = maxspin = 0 signals = result = 0 wseq = 52 seq = 26 private = 0 maxspin = err = result = wseq = g = seq = flags = private = signals = done = g1_start = spin = buffer = cbuffer = s = #4 ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627 No locals. #5 0x0000627586ad07ca in std::thread::_State_impl > >::_M_run() () No symbol table info available. #6 0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 No symbol table info available. #7 0x00007b73e609ca94 in start_thread (arg=) at ./nptl/pthread_create.c:447 ret = pd = out = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727738259136, 1370408474212223121, 135727738259136, -160, 0, 135728109450704, 1370408474132531345, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = #8 0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 No locals. . Thread 28 (Thread 0x7b7191c006c0 (LWP 6338)): #0 0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = resultvar = __arg6 = __arg5 = __arg4 = __arg3 = __arg2 = __arg1 = _a6 = _a5 = _a4 = _a3 = _a2 = _a1 = #1 __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87 err = clockbit = 256 op = 393 err = clockbit = op = #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 No locals. #3 0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b7191bff8c0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 60, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0} err = g = 0 flags = g1_start = maxspin = 0 signals = result = 0 wseq = 60 seq = 30 private = 0 maxspin = err = result = wseq = g = seq = flags = private = signals = done = g1_start = spin = buffer = cbuffer = s = #4 ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627 No locals. #5 0x0000627586ad07ca in std::thread::_State_impl > >::_M_run() () No symbol table info available. #6 0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 No symbol table info available. #7 0x00007b73e609ca94 in start_thread (arg=) at ./nptl/pthread_create.c:447 ret = pd = out = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727706801856, 1370413147136641169, 135727706801856, -160, 0, 135728109450704, 1370413147056949393, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = #8 0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 No locals. . Thread 27 (Thread 0x7b71944006c0 (LWP 6334)): #0 0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57 sc_cancel_oldtype = 0 sc_ret = resultvar = __arg6 = __arg5 = __arg4 = __arg3 = __arg2 = __arg1 = _a6 = _a5 = _a4 = _a3 = _a2 = _a1 = #1 __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87 err = clockbit = 256 op = 393 err = clockbit = op = #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 No locals. #3 0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71943ff8c0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 54, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0} err =

Said-Akbar commented 4 days ago

Based on app crash logs, I see that rocm is not able to find the symbol table 'No symbol table info available.' Not sure what that means. Let me know. Thanks!

lamikr commented 3 days ago

Thanks, good to see that the the basic applications works. I will start my gfx906 build and try to check if I can figure out fix for those build errors with llama.cpp.

Said-Akbar commented 3 days ago

Thank you! Looking forward to your updates.

lamikr commented 2 days ago

Hi, I added some more trace to clr component that is responsible for loading the so-files that can contain CO-data. Also some other small changes related to vega. You should get the updated files and then start the build with commands:

git fetch
./babs.sh -up wip/rocm_sdk_builder_612_vega_testing
./babs.sh -b

After that then this command should printout much more debug to see what's going on:

AMD_COMGR_SAVE_TEMPS=1 AMD_COMGR_REDIRECT_LOGS=stdout AMD_COMGR_EMIT_VERBOSE_LOGS=1 ROCM_SDK_PRINTOUT_DEBUG_MESSAGES=1 llama-server -m /opt/rocm_sdk_models/Qwen2.5-7B-Instruct-Q8_0/Qwen2.5-7B-Instruct-Q8_0.gguf -c 2048 -ngl 99 --metrics I have tested that this commands works on gfx1030,gfx1010,gfx1102 and gfx1103.

And I did the gfx906 build and can find the string causing problem on these files

$cd /opt/rocm_sdk_612/lib64/rocblas/library
$ grep -R Cijk_Alik_Bljk_HB_GB
grep: TensileLibrary_Type_HH_Contraction_l_Alik_Bljk_Cijk_Dijk_fallback.dat: binary file matches
grep: TensileLibrary_Type_HH_Contraction_l_Alik_Bljk_Cijk_Dijk_gfx906.dat: binary file matches
grep: TensileLibrary_Type_HH_Contraction_l_Alik_Bljk_Cijk_Dijk_gfx906.co: binary file matches
grep: TensileLibrary_Type_HH_Contraction_l_Alik_Bljk_Cijk_Dijk_fallback_gfx906.hsaco: binary file matches

$ strings TensileLibrary_Type_HH_Contraction_l_Alik_Bljk_Cijk_Dijk_gfx906.co | grep Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1

Said-Akbar commented 2 days ago

@lamikr , thanks! I will run above commands when I get back home.

Regarding new changes in the repo, will I have to build everything from scratch or only build only specific files in wip/rocm_sdk_builder_612_vega_testing ? I spent 3 days and over 10 hours building the last version of this repo. I hope this change will not require building everything from scratch.

Thanks!

Said-Akbar commented 2 days ago

ok, this time it took 1 hour to build. I am still seeing the llama.cpp error. This time it has all the error logs as you explained above. Attaching the output error here.

AMD_COMGR_SAVE_TEMPS=1 AMD_COMGR_REDIRECT_LOGS=stdout AMD_COMGR_EMIT_VERBOSE_LOGS=1 ROCM_SDK_PRINTOUT_DEBUG_MESSAGES=1 llama-server -m /media/saidp/datasets/text_generation/models/Qwen2.5-7B-Instruct-Q8_0/Qwen2.5-7B-Instruct-Q8_0.gguf -c 2048 -ngl 99 --metrics >>sdk_error_output.txt 2>&1

sdk_error_output.txt

Said-Akbar commented 2 days ago

Here is the error logs from run_and_save_benchmarks.sh.

` cd benchmarks/

AMD_COMGR_SAVE_TEMPS=1 AMD_COMGR_REDIRECT_LOGS=stdout AMD_COMGR_EMIT_VERBOSE_LOGS=1 ROCM_SDK_PRINTOUT_DEBUG_MESSAGES=1 ./run_and_save_benchmarks.sh >>benchmark_error_output.txt 2>&1 `

benchmark_error_output.txt

lamikr commented 2 days ago

No need to re-build everyhing. If you have a working build and run the

"./babs.sh -up ", it will check

which projects have changes either in binfo file or in patches directory
check-out and-reapply patches for those changed projects
clean the build directory for those changed projects

So when you then run the ./babs.sh -b next time, it will only re-build and install the changed projects.

Said-Akbar commented 2 days ago

@lamikr , yes, I used the commands you shared above.

git fetch
./babs.sh -up wip/rocm_sdk_builder_612_vega_testing
./babs.sh -b

It took 1 hour to compile. I am still seeing the same 'symbol not found' errors. Please, refer to my comments above for detailed error logs.

lamikr commented 2 days ago

In comparison, here is my log for successful lllama_coo launch with gfx1030 and same parameters. The output is pretty-similar except in the the very end. I was expecting to see some erros in your case on those clang or lld build commands that it executes to build the model, but even those looked pretty same.

` INFO [ main] build info | tid="140451167387904" timestamp=1732592140 build=3407 commit="dab1e48c" INFO [ main] system info | tid="140451167387904" timestamp=1732592140 n_threads=8 n_threads_batch=-1 total_threads=16 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | " llama_model_loader: loaded meta data with 38 key-value pairs and 339 tensors from /opt/rocm_sdk_models/Qwen2.5-7B-Instruct-Q8_0/Qwen2.5-7B-Instruct-Q8_0.gguf (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = qwen2 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Qwen2.5 7B Instruct llama_model_loader: - kv 3: general.finetune str = Instruct llama_model_loader: - kv 4: general.basename str = Qwen2.5 llama_model_loader: - kv 5: general.size_label str = 7B llama_model_loader: - kv 6: general.license str = apache-2.0 llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen2.5-7... llama_model_loader: - kv 8: general.base_model.count u32 = 1 llama_model_loader: - kv 9: general.base_model.0.name str = Qwen2.5 7B llama_model_loader: - kv 10: general.base_model.0.organization str = Qwen llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2.5-7B llama_model_loader: - kv 12: general.tags arr[str,2] = ["chat", "text-generation"] llama_model_loader: - kv 13: general.languages arr[str,1] = ["en"] llama_model_loader: - kv 14: qwen2.block_count u32 = 28 llama_model_loader: - kv 15: qwen2.context_length u32 = 32768 llama_model_loader: - kv 16: qwen2.embedding_length u32 = 3584 llama_model_loader: - kv 17: qwen2.feed_forward_length u32 = 18944 llama_model_loader: - kv 18: qwen2.attention.head_count u32 = 28 llama_model_loader: - kv 19: qwen2.attention.head_count_kv u32 = 4 llama_model_loader: - kv 20: qwen2.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 21: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 22: general.file_type u32 = 7 llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2 llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645 llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643 llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643 llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false llama_model_loader: - kv 32: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>... llama_model_loader: - kv 33: general.quantization_version u32 = 2 llama_model_loader: - kv 34: quantize.imatrix.file str = /models_out/Qwen2.5-7B-Instruct-GGUF/... llama_model_loader: - kv 35: quantize.imatrix.dataset str = /training_dir/calibration_datav3.txt llama_model_loader: - kv 36: quantize.imatrix.entries_count i32 = 196 llama_model_loader: - kv 37: quantize.imatrix.chunks_count i32 = 128 llama_model_loader: - type f32: 141 tensors llama_model_loader: - type q8_0: 198 tensors llm_load_vocab: special tokens cache size = 22 llm_load_vocab: token to piece cache size = 0.9310 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = qwen2 llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 152064 llm_load_print_meta: n_merges = 151387 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 3584 llm_load_print_meta: n_layer = 28 llm_load_print_meta: n_head = 28 llm_load_print_meta: n_head_kv = 4 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 7 llm_load_print_meta: n_embd_k_gqa = 512 llm_load_print_meta: n_embd_v_gqa = 512 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-06 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 18944 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 2 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = ?B llm_load_print_meta: model ftype = Q8_0 llm_load_print_meta: model params = 7.62 B llm_load_print_meta: model size = 7.54 GiB (8.50 BPW) llm_load_print_meta: general.name = Qwen2.5 7B Instruct llm_load_print_meta: BOS token = 151643 '<|endoftext|>' llm_load_print_meta: EOS token = 151645 '<|im_end|>' llm_load_print_meta: PAD token = 151643 '<|endoftext|>' llm_load_print_meta: LF token = 148848 'ÄĬ' llm_load_print_meta: EOT token = 151645 '<|im_end|>' llm_load_print_meta: max token length = 256 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 ROCm devices: Device 0: AMD Radeon RX 6800, compute capability 10.3, VMM: no llm_load_tensors: ggml ctx size = 0.30 MiB llm_load_tensors: offloading 28 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 29/29 layers to GPU llm_load_tensors: ROCm0 buffer size = 7165.44 MiB llm_load_tensors: CPU buffer size = 552.23 MiB amd_comgr_do_action: ActionKind: AMD_COMGR_ACTION_ADD_PRECOMPILED_HEADERS IsaName: amdgcn-amd-amdhsa--gfx1030 Options: "-O3" "-cl-kernel-arg-info" "-DOPENCL_VERSION=200" "-DIMAGE_SUPPORT=1" "-Xclang" "-cl-ext=+cl_khr_fp64,+cl_khr_global_int32_base_atomics,+cl_khr_global_int32_extended_atomics,+cl_khr_local_int32_base_atomics,+cl_khr_local_int32_extended_atomics,+cl_khr_int64_base_atomics,+cl_khr_int64_extended_atomics,+cl_khr_3d_image_writes,+cl_khr_byte_addressable_store,+cl_khr_fp16,+cl_khr_gl_sharing,+cl_amd_device_attribute_query,+cl_amd_media_ops,+cl_amd_media_ops2,+cl_khr_image2d_from_buffer,+cl_khr_subgroups,+cl_amd_copy_buffer_p2p,+cl_amd_assembly_program" "-mllvm" "-amdgpu-prelink" "-mcode-object-version=5" Path: Language: AMD_COMGR_LANGUAGE_OPENCL_1_2 Comgr Branch-Commit: HEAD-72e3209e9ecf LLVM Commit: 72e3209e9ecf09af59f32bde15867048d6410e3b ReturnStatus: AMD_COMGR_STATUS_SUCCESS

amd_comgr_do_action: ActionKind: AMD_COMGR_ACTION_COMPILE_SOURCE_TO_BC IsaName: amdgcn-amd-amdhsa--gfx1030 Options: "-O3" "-cl-kernel-arg-info" "-DOPENCL_VERSION=200" "-DIMAGE_SUPPORT=1" "-Xclang" "-cl-ext=+cl_khr_fp64,+cl_khr_global_int32_base_atomics,+cl_khr_global_int32_extended_atomics,+cl_khr_local_int32_base_atomics,+cl_khr_local_int32_extended_atomics,+cl_khr_int64_base_atomics,+cl_khr_int64_extended_atomics,+cl_khr_3d_image_writes,+cl_khr_byte_addressable_store,+cl_khr_fp16,+cl_khr_gl_sharing,+cl_amd_device_attribute_query,+cl_amd_media_ops,+cl_amd_media_ops2,+cl_khr_image2d_from_buffer,+cl_khr_subgroups,+cl_amd_copy_buffer_p2p,+cl_amd_assembly_program" "-mllvm" "-amdgpu-prelink" "-mcode-object-version=5" Path: Language: AMD_COMGR_LANGUAGE_OPENCL_1_2 Comgr Branch-Commit: HEAD-72e3209e9ecf LLVM Commit: 72e3209e9ecf09af59f32bde15867048d6410e3b Compilation Args: "-target" "amdgcn-amd-amdhsa" "-mcpu=gfx1030" "-I" "/tmp/comgr-9ef550/include" "-include-pch" "/tmp/comgr-9ef550/include/opencl1.2-c.pch" "-Xclang" "-fno-validate-pch" "-x" "cl" "-std=cl1.2" "-cl-no-stdinc" "-c" "-emit-llvm" "-O3" "-cl-kernel-arg-info" "-DOPENCL_VERSION=200" "-DIMAGE_SUPPORT=1" "-Xclang" "-cl-ext=+cl_khr_fp64,+cl_khr_global_int32_base_atomics,+cl_khr_global_int32_extended_atomics,+cl_khr_local_int32_base_atomics,+cl_khr_local_int32_extended_atomics,+cl_khr_int64_base_atomics,+cl_khr_int64_extended_atomics,+cl_khr_3d_image_writes,+cl_khr_byte_addressable_store,+cl_khr_fp16,+cl_khr_gl_sharing,+cl_amd_device_attribute_query,+cl_amd_media_ops,+cl_amd_media_ops2,+cl_khr_image2d_from_buffer,+cl_khr_subgroups,+cl_amd_copy_buffer_p2p,+cl_amd_assembly_program" "-mllvm" "-amdgpu-prelink" "-mcode-object-version=5" "-nogpulib" "/tmp/comgr-9ef550/input/CompileSource" "-o" "/tmp/comgr-9ef550/output/CompileSource.bc" Driver Job Args: clang "-cc1" "-mcode-object-version=5" "-mllvm" "--amdhsa-code-object-version=5" "-triple" "amdgcn-amd-amdhsa" "-emit-llvm-bc" "-emit-llvm-uselists" "-clear-ast-before-backend" "-main-file-name" "CompileSource" "-mrelocation-model" "pic" "-pic-level" "2" "-fhalf-no-semantic-interposition" "-mframe-pointer=none" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-fvisibility=hidden" "-fapply-global-visibility-to-externs" "-target-cpu" "gfx1030" "-debugger-tuning=gdb" "-resource-dir" "/opt/rocm_sdk_612/llvm/lib/clang/17" "-c-isystem" "/opt/rocm_sdk_612/llvm/include/gpu-none-llvm" "-include-pch" "/tmp/comgr-9ef550/include/opencl1.2-c.pch" "-I" "/tmp/comgr-9ef550/include" "-D" "OPENCL_VERSION=200" "-D" "IMAGE_SUPPORT=1" "-O3" "-std=cl1.2" "-fdebug-compilation-dir=/home/lamikr" "-ferror-limit" "19" "-cl-kernel-arg-info" "-nogpulib" "-fno-threadsafe-statics" "-vectorize-loops" "-vectorize-slp" "-fno-validate-pch" "-cl-ext=+cl_khr_fp64,+cl_khr_global_int32_base_atomics,+cl_khr_global_int32_extended_atomics,+cl_khr_local_int32_base_atomics,+cl_khr_local_int32_extended_atomics,+cl_khr_int64_base_atomics,+cl_khr_int64_extended_atomics,+cl_khr_3d_image_writes,+cl_khr_byte_addressable_store,+cl_khr_fp16,+cl_khr_gl_sharing,+cl_amd_device_attribute_query,+cl_amd_media_ops,+cl_amd_media_ops2,+cl_khr_image2d_from_buffer,+cl_khr_subgroups,+cl_amd_copy_buffer_p2p,+cl_amd_assembly_program" "-mllvm" "-amdgpu-prelink" "-faddrsig" "-o" "/tmp/comgr-9ef550/output/CompileSource.bc" "-x" "cl" "/tmp/comgr-9ef550/input/CompileSource" ReturnStatus: AMD_COMGR_STATUS_SUCCESS

amd_comgr_do_action: ActionKind: AMD_COMGR_ACTION_ADD_DEVICE_LIBRARIES IsaName: amdgcn-amd-amdhsa--gfx1030 Options: "code_object_v5" Path: Language: AMD_COMGR_LANGUAGE_OPENCL_1_2 Comgr Branch-Commit: HEAD-72e3209e9ecf LLVM Commit: 72e3209e9ecf09af59f32bde15867048d6410e3b ReturnStatus: AMD_COMGR_STATUS_SUCCESS

amd_comgr_do_action: ActionKind: AMD_COMGR_ACTION_LINK_BC_TO_BC IsaName: amdgcn-amd-amdhsa--gfx1030 Options: "code_object_v5" Path: Language: AMD_COMGR_LANGUAGE_OPENCL_1_2 Comgr Branch-Commit: HEAD-72e3209e9ecf LLVM Commit: 72e3209e9ecf09af59f32bde15867048d6410e3b Linking Bitcode: /tmp/comgr-bebd29/input/LLVM Binary Linking Bitcode: /tmp/comgr-bebd29/input/opencl_lib.bc Linking Bitcode: /tmp/comgr-bebd29/input/ocml_lib.bc Linking Bitcode: /tmp/comgr-bebd29/input/ockl_lib.bc Linking Bitcode: /tmp/comgr-bebd29/input/oclc_isa_version_1030.bc Linking Bitcode: /tmp/comgr-bebd29/input/oclc_correctly_rounded_sqrt_off_lib.bc Linking Bitcode: /tmp/comgr-bebd29/input/oclc_daz_opt_off_lib.bc Linking Bitcode: /tmp/comgr-bebd29/input/oclc_finite_only_off_lib.bc Linking Bitcode: /tmp/comgr-bebd29/input/oclc_unsafe_math_off_lib.bc Linking Bitcode: /tmp/comgr-bebd29/input/oclc_wavefrontsize64_off_lib.bc Linking Bitcode: /tmp/comgr-bebd29/input/oclc_abi_version_500_lib.bc ReturnStatus: AMD_COMGR_STATUS_SUCCESS

amd_comgr_do_action: ActionKind: AMD_COMGR_ACTION_CODEGEN_BC_TO_RELOCATABLE IsaName: amdgcn-amd-amdhsa--gfx1030 Options: "-O3" "-cl-kernel-arg-info" "-mllvm" "-amdgpu-internalize-symbols" "-mcode-object-version=5" Path: Language: AMD_COMGR_LANGUAGE_NONE Comgr Branch-Commit: HEAD-72e3209e9ecf LLVM Commit: 72e3209e9ecf09af59f32bde15867048d6410e3b Compilation Args: "-target" "amdgcn-amd-amdhsa" "-mcpu=gfx1030" "-c" "-mllvm" "-amdgpu-internalize-symbols" "-O3" "-cl-kernel-arg-info" "-mllvm" "-amdgpu-internalize-symbols" "-mcode-object-version=5" "-nogpulib" "/tmp/comgr-6e691f/input/linked.bc" "-o" "/tmp/comgr-6e691f/output/linked.bc.o" Driver Job Args: clang "-cc1" "-mcode-object-version=5" "-mllvm" "--amdhsa-code-object-version=5" "-triple" "amdgcn-amd-amdhsa" "-emit-obj" "-clear-ast-before-backend" "-main-file-name" "linked.bc" "-mrelocation-model" "pic" "-pic-level" "2" "-fhalf-no-semantic-interposition" "-mframe-pointer=none" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-fvisibility=hidden" "-fapply-global-visibility-to-externs" "-target-cpu" "gfx1030" "-debugger-tuning=gdb" "-resource-dir" "/opt/rocm_sdk_612/llvm/lib/clang/17" "-O3" "-fdebug-compilation-dir=/home/lamikr" "-ferror-limit" "19" "-cl-kernel-arg-info" "-nogpulib" "-vectorize-loops" "-vectorize-slp" "-mllvm" "-amdgpu-internalize-symbols" "-mllvm" "-amdgpu-internalize-symbols" "-faddrsig" "-o" "/tmp/comgr-6e691f/output/linked.bc.o" "-x" "ir" "/tmp/comgr-6e691f/input/linked.bc" ReturnStatus: AMD_COMGR_STATUS_SUCCESS

amd_comgr_do_action: ActionKind: AMD_COMGR_ACTION_LINK_RELOCATABLE_TO_EXECUTABLE IsaName: amdgcn-amd-amdhsa--gfx1030 Options: Path: Language: AMD_COMGR_LANGUAGE_NONE Comgr Branch-Commit: HEAD-72e3209e9ecf LLVM Commit: 72e3209e9ecf09af59f32bde15867048d6410e3b Compilation Args: "-target" "amdgcn-amd-amdhsa" "-mcpu=gfx1030" "/tmp/comgr-3e9e36/input/linked.bc.o" "-o" "/tmp/comgr-3e9e36/output/a.so" Driver Job Args: lld "/tmp/comgr-3e9e36/input/linked.bc.o" "-plugin-opt=mcpu=gfx1030" "--no-undefined" "-shared" "-o" "/tmp/comgr-3e9e36/output/a.so" ....................................................................................... llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: n_batch = 2048 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 1000000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: ROCm0 KV buffer size = 112.00 MiB llama_new_context_with_model: KV self size = 112.00 MiB, K (f16): 56.00 MiB, V (f16): 56.00 MiB llama_new_context_with_model: ROCm_Host output buffer size = 1.16 MiB llama_new_context_with_model: ROCm0 compute buffer size = 304.00 MiB llama_new_context_with_model: ROCm_Host compute buffer size = 11.01 MiB llama_new_context_with_model: graph nodes = 986 llama_new_context_with_model: graph splits = 2 lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsparse.so.1.0.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/libggml.so hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102 hip_fatbin.cpp: Found CO for device amdgcn-amd-amdhsa--gfx1030, file: /opt/rocm_sdk_612/lib64/librocsolver.so.0.1.60102 INFO [ init] initializing slots | tid="140451167387904" timestamp=1732592145 n_slots=1 INFO [ init] new slot | tid="140451167387904" timestamp=1732592145 id_slot=0 n_ctx_slot=2048 INFO [ main] model loaded | tid="140451167387904" timestamp=1732592145 INFO [ main] chat template | tid="140451167387904" timestamp=1732592145 chat_example="<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHi there<|im_end|>\n<|im_start|>user\nHow are you?<|im_end|>\n<|im_start|>assistant\n" built_in=true INFO [ main] HTTP server listening | tid="140451167387904" timestamp=1732592145 n_threads_http="15" port="8080" hostname="127.0.0.1" INFO [ update_slots] all slots are idle | tid="140451167387904" timestamp=1732592145 INFO [ update_slots] all slots are idle | tid="140451167387904" timestamp=1732592162 ReturnStatus: AMD_COMGR_STATUS_SUCCESS

`

lamikr commented 2 days ago

How about those grep commands on /opt/rocm_sdk_612/lib64/rocblas/library Do you have same files matched when searching this Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1?

One possibility could be that this is some kind of xnack+/xnack- type error. I have not had needed to debug that kind of problem by myself but basically the gpu can run code in xnack- mode and then something else is build on xnack+ mode, then those are not compatible. I need to investigate this more.

Said-Akbar commented 2 days ago

sure, here is the string matches:

strings TensileLibrary_Type_HH_Contraction_l_Alik_Bljk_Cijk_Dijk_gfx906.co | grep Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1
/Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1
2Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1.kd
/Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM8
2Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM8.kd
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1.kd
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM8
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM8.kd
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1_preloaded
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM8_preloaded
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1.kd
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM8
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM8.kd

and llama.cpp symbol matches:

strings TensileLibrary_Type_HH_Contraction_l_Alik_Bljk_Cijk_Dijk_gfx906.co | grep Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1
/Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1
2Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1.kd
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1.kd
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1_preloaded
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1
Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1.kd

Said-Akbar commented 2 days ago

Is there a way to switch to xnack+ mode in rocm/amd MI60 GPUs? If not, then I will wait for your update. Thanks!

EDIT: Based on this page (https://rocm.docs.amd.com/en/latest/conceptual/gpu-memory.html), enabling xnack should be possible (HSA_XNACK=1) but did not work for me. Interesting fact from that page is 'Compiled kernels will run regardless if XNACK is enabled or is disabled. ' e.g. hipcc --offload-arch=gfx906 will run regardless if XNACK+ or XNACK-.

lamikr commented 2 days ago

Sorry, as I do not remember seeing this kind of error my self, this is little bit quess work for now to try to isolate the problem. So disabling comgr and hiprt would be again something to try next. Got idea from: https://github.com/ROCm/MIOpen/issues/2851

So, if you have time, can you try to build MIOpen with following options to see if anything changes.

-DMIOPEN_USE_COMGR=Off -DMIOPEN_USE_HIPRTC=Off

It can be done by opening binfo/core/034_miopen.binfo and adding following line BINFO_APP_CMAKE_CFG="${BINFO_APP_CMAKE_CFG} -DMIOPEN_USE_COMGR=OFF -DMIOPEN_USE_HIPRTC=OFF" for example after the BINFO_APP_CMAKE_CFG="${BINFO_APP_CMAKE_CFG} -DCMAKE_INSTALL_LIBDIR=lib64"

And then rebuilding MIOpen

./babs.sh --clean binfo/core/034_miopen.binfo ./babs.sh -b

Said-Akbar commented 1 day ago

Thanks! I will try it today when I get back home.

Said-Akbar commented 1 day ago

ok, I was impatient and tried buidling MIOpen with above changes.

``` install ok: MIOpen /home/saidp/Downloads/rocm_sdk_builder/builddir/034_miopen [77] Post-installing: MIOpen post-install MIOpen, post install command 0 unset CXX post-install cmd ok: MIOpen post-install ok: MIOpen LIST_BINFO_FILE_FULLNAME[78]: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/035_AMDMIGraphX.binfo APP_INFO_FULL_NAME: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/035_AMDMIGraphX.binfo --------------------------- [78] BINFO_APP_NAME: AMDMIGraphX BINFO FILE: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/035_AMDMIGraphX.binfo BINFO_APP_SRC_SUBDIR_BASENAME: BINFO_APP_SRC_TOPDIR_BASENAME: AMDMIGraphX BINFO_APP_SRC_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/AMDMIGraphX BINFO_APP_SRC_CLONE_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/AMDMIGraphX BINFO_APP_BUILD_DIR: /home/saidp/Downloads/rocm_sdk_builder/builddir/035_AMDMIGraphX HIP_PATH: /opt/rocm_sdk_612 INSTALL_DIR: /opt/rocm_sdk_612 HIP_PLATFORM: amd TASK_RESULT_FILE_INSTALL: /home/saidp/Downloads/rocm_sdk_builder/builddir/035_AMDMIGraphX/.result_install --------------------------- LIST_BINFO_FILE_FULLNAME[79]: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/036_rocWMMA.binfo APP_INFO_FULL_NAME: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/036_rocWMMA.binfo --------------------------- [79] BINFO_APP_NAME: rocWMMA BINFO FILE: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/036_rocWMMA.binfo BINFO_APP_SRC_SUBDIR_BASENAME: BINFO_APP_SRC_TOPDIR_BASENAME: rocWMMA BINFO_APP_SRC_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/rocWMMA BINFO_APP_SRC_CLONE_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/rocWMMA BINFO_APP_BUILD_DIR: /home/saidp/Downloads/rocm_sdk_builder/builddir/036_rocWMMA HIP_PATH: /opt/rocm_sdk_612 INSTALL_DIR: /opt/rocm_sdk_612 HIP_PLATFORM: amd TASK_RESULT_FILE_INSTALL: /home/saidp/Downloads/rocm_sdk_builder/builddir/036_rocWMMA/.result_install --------------------------- LIST_BINFO_FILE_FULLNAME[80]: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/037_magma.binfo APP_INFO_FULL_NAME: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/037_magma.binfo --------------------------- [80] BINFO_APP_NAME: magma BINFO FILE: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/037_magma.binfo BINFO_APP_SRC_SUBDIR_BASENAME: BINFO_APP_SRC_TOPDIR_BASENAME: magma BINFO_APP_SRC_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/magma BINFO_APP_SRC_CLONE_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/magma BINFO_APP_BUILD_DIR: /home/saidp/Downloads/rocm_sdk_builder/builddir/037_magma HIP_PATH: /opt/rocm_sdk_612 INSTALL_DIR: /opt/rocm_sdk_612 HIP_PLATFORM: amd TASK_RESULT_FILE_INSTALL: /home/saidp/Downloads/rocm_sdk_builder/builddir/037_magma/.result_install --------------------------- LIST_BINFO_FILE_FULLNAME[81]: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/038_aotriton.binfo APP_INFO_FULL_NAME: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/038_aotriton.binfo --------------------------- [81] BINFO_APP_NAME: aotriton BINFO FILE: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/038_aotriton.binfo BINFO_APP_SRC_SUBDIR_BASENAME: BINFO_APP_SRC_TOPDIR_BASENAME: aotriton BINFO_APP_SRC_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/aotriton BINFO_APP_SRC_CLONE_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/aotriton BINFO_APP_BUILD_DIR: /home/saidp/Downloads/rocm_sdk_builder/builddir/038_aotriton HIP_PATH: /opt/rocm_sdk_612 INSTALL_DIR: /opt/rocm_sdk_612 HIP_PLATFORM: amd TASK_RESULT_FILE_INSTALL: /home/saidp/Downloads/rocm_sdk_builder/builddir/038_aotriton/.result_install --------------------------- LIST_BINFO_FILE_FULLNAME[82]: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_01_pytorch_dependencies.binfo APP_INFO_FULL_NAME: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_01_pytorch_dependencies.binfo --------------------------- [82] BINFO_APP_NAME: pytorch_dependencies BINFO FILE: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_01_pytorch_dependencies.binfo BINFO_APP_SRC_SUBDIR_BASENAME: BINFO_APP_SRC_TOPDIR_BASENAME: pytorch_dependencies BINFO_APP_SRC_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/pytorch_dependencies BINFO_APP_SRC_CLONE_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/pytorch_dependencies BINFO_APP_BUILD_DIR: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_01_pytorch_dependencies HIP_PATH: /opt/rocm_sdk_612 INSTALL_DIR: /opt/rocm_sdk_612 HIP_PLATFORM: amd TASK_RESULT_FILE_INSTALL: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_01_pytorch_dependencies/.result_install --------------------------- LIST_BINFO_FILE_FULLNAME[83]: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_02_pytorch.binfo APP_INFO_FULL_NAME: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_02_pytorch.binfo --------------------------- [83] BINFO_APP_NAME: pytorch BINFO FILE: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_02_pytorch.binfo BINFO_APP_SRC_SUBDIR_BASENAME: BINFO_APP_SRC_TOPDIR_BASENAME: pytorch BINFO_APP_SRC_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/pytorch BINFO_APP_SRC_CLONE_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/pytorch BINFO_APP_BUILD_DIR: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_02_pytorch HIP_PATH: /opt/rocm_sdk_612 INSTALL_DIR: /opt/rocm_sdk_612 HIP_PLATFORM: amd TASK_RESULT_FILE_INSTALL: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_02_pytorch/.result_install --------------------------- LIST_BINFO_FILE_FULLNAME[84]: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_03_pytorch_vision.binfo APP_INFO_FULL_NAME: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_03_pytorch_vision.binfo --------------------------- [84] BINFO_APP_NAME: pytorch_vision BINFO FILE: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_03_pytorch_vision.binfo BINFO_APP_SRC_SUBDIR_BASENAME: BINFO_APP_SRC_TOPDIR_BASENAME: pytorch_vision BINFO_APP_SRC_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/pytorch_vision BINFO_APP_SRC_CLONE_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/pytorch_vision BINFO_APP_BUILD_DIR: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_03_pytorch_vision HIP_PATH: /opt/rocm_sdk_612 INSTALL_DIR: /opt/rocm_sdk_612 HIP_PLATFORM: amd TASK_RESULT_FILE_INSTALL: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_03_pytorch_vision/.result_install --------------------------- LIST_BINFO_FILE_FULLNAME[85]: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_04_pytorch_audio.binfo APP_INFO_FULL_NAME: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_04_pytorch_audio.binfo --------------------------- [85] BINFO_APP_NAME: pytorch_audio BINFO FILE: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_04_pytorch_audio.binfo BINFO_APP_SRC_SUBDIR_BASENAME: BINFO_APP_SRC_TOPDIR_BASENAME: pytorch_audio BINFO_APP_SRC_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/pytorch_audio BINFO_APP_SRC_CLONE_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/pytorch_audio BINFO_APP_BUILD_DIR: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_04_pytorch_audio HIP_PATH: /opt/rocm_sdk_612 INSTALL_DIR: /opt/rocm_sdk_612 HIP_PLATFORM: amd TASK_RESULT_FILE_INSTALL: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_04_pytorch_audio/.result_install --------------------------- LIST_BINFO_FILE_FULLNAME[86]: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_05_torch_migraphx.binfo APP_INFO_FULL_NAME: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_05_torch_migraphx.binfo --------------------------- [86] BINFO_APP_NAME: torch_migraphx BINFO FILE: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_05_torch_migraphx.binfo BINFO_APP_SRC_SUBDIR_BASENAME: BINFO_APP_SRC_TOPDIR_BASENAME: torch_migraphx BINFO_APP_SRC_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/torch_migraphx BINFO_APP_SRC_CLONE_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/torch_migraphx BINFO_APP_BUILD_DIR: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_05_torch_migraphx HIP_PATH: /opt/rocm_sdk_612 INSTALL_DIR: /opt/rocm_sdk_612 HIP_PLATFORM: amd TASK_RESULT_FILE_INSTALL: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_05_torch_migraphx/.result_install --------------------------- LIST_BINFO_FILE_FULLNAME[87]: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_06_pytorch_bitsandbytes.binfo APP_INFO_FULL_NAME: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_06_pytorch_bitsandbytes.binfo --------------------------- [87] BINFO_APP_NAME: bitsandbytes BINFO FILE: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_06_pytorch_bitsandbytes.binfo BINFO_APP_SRC_SUBDIR_BASENAME: BINFO_APP_SRC_TOPDIR_BASENAME: bitsandbytes BINFO_APP_SRC_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/bitsandbytes BINFO_APP_SRC_CLONE_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/bitsandbytes BINFO_APP_BUILD_DIR: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_06_pytorch_bitsandbytes HIP_PATH: /opt/rocm_sdk_612 INSTALL_DIR: /opt/rocm_sdk_612 HIP_PLATFORM: amd TASK_RESULT_FILE_INSTALL: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_06_pytorch_bitsandbytes/.result_install --------------------------- LIST_BINFO_FILE_FULLNAME[88]: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_07_triton.binfo APP_INFO_FULL_NAME: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_07_triton.binfo --------------------------- [88] BINFO_APP_NAME: triton BINFO FILE: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/039_07_triton.binfo BINFO_APP_SRC_SUBDIR_BASENAME: BINFO_APP_SRC_TOPDIR_BASENAME: triton BINFO_APP_SRC_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/triton BINFO_APP_SRC_CLONE_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/triton BINFO_APP_BUILD_DIR: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_07_triton HIP_PATH: /opt/rocm_sdk_612 INSTALL_DIR: /opt/rocm_sdk_612 HIP_PLATFORM: amd TASK_RESULT_FILE_INSTALL: /home/saidp/Downloads/rocm_sdk_builder/builddir/039_07_triton/.result_install --------------------------- LIST_BINFO_FILE_FULLNAME[89]: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/040_01_onnxruntime_rocm_training.binfo APP_INFO_FULL_NAME: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/040_01_onnxruntime_rocm_training.binfo --------------------------- [89] BINFO_APP_NAME: onnxruntime BINFO FILE: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/040_01_onnxruntime_rocm_training.binfo BINFO_APP_SRC_SUBDIR_BASENAME: BINFO_APP_SRC_TOPDIR_BASENAME: onnxruntime BINFO_APP_SRC_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/onnxruntime BINFO_APP_SRC_CLONE_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/onnxruntime BINFO_APP_BUILD_DIR: /home/saidp/Downloads/rocm_sdk_builder/builddir/040_01_onnxruntime_rocm_training HIP_PATH: /opt/rocm_sdk_612 INSTALL_DIR: /opt/rocm_sdk_612 HIP_PLATFORM: amd TASK_RESULT_FILE_INSTALL: /home/saidp/Downloads/rocm_sdk_builder/builddir/040_01_onnxruntime_rocm_training/.result_install --------------------------- LIST_BINFO_FILE_FULLNAME[90]: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/040_02_onnxruntime_deepspeed.binfo APP_INFO_FULL_NAME: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/040_02_onnxruntime_deepspeed.binfo --------------------------- [90] BINFO_APP_NAME: DeepSpeed BINFO FILE: /home/saidp/Downloads/rocm_sdk_builder/binfo/core/040_02_onnxruntime_deepspeed.binfo BINFO_APP_SRC_SUBDIR_BASENAME: BINFO_APP_SRC_TOPDIR_BASENAME: DeepSpeed BINFO_APP_SRC_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/DeepSpeed BINFO_APP_SRC_CLONE_DIR: /home/saidp/Downloads/rocm_sdk_builder/src_projects/DeepSpeed BINFO_APP_BUILD_DIR: /home/saidp/Downloads/rocm_sdk_builder/builddir/040_02_onnxruntime_deepspeed HIP_PATH: /opt/rocm_sdk_612 INSTALL_DIR: /opt/rocm_sdk_612 HIP_PLATFORM: amd TASK_RESULT_FILE_INSTALL: /home/saidp/Downloads/rocm_sdk_builder/builddir/040_02_onnxruntime_deepspeed/.result_install --------------------------- ROCM SDK build and install ready You can use following commands to check your GPU source /opt/rocm_sdk_612/bin/env_rocm.sh rocminfo ```

It was built without any errors. then, I faced the same symbol error again.

source /opt/rocm_sdk_612/bin/env_rocm.sh
llama-server -m /opt/rocm_sdk_models/Qwen2.5-7B-Instruct-Q8_0/Qwen2.5-7B-Instruct-Q8_0.gguf -c 2048 -ngl 99 --metrics -sm none
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon Graphics, compute capability 9.0, VMM: no
  Device 1: AMD Radeon Graphics, compute capability 9.0, VMM: no
build: 3901 (49f4671b) with cc (Ubuntu 13.2.0-23ubuntu4) 13.2.0 for x86_64-linux-gnu
system info: n_threads = 16, n_threads_batch = 16, total_threads = 32

system_info: n_threads = 16 (n_threads_batch = 16) / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | 

main: HTTP server is listening, hostname: 127.0.0.1, port: 8080, http threads: 31
main: loading model
llama_model_loader: loaded meta data with 38 key-value pairs and 339 tensors from /opt/rocm_sdk_models/Qwen2.5-7B-Instruct-Q8_0/Qwen2.5-7B-Instruct-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 7B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Qwen2.5
llama_model_loader: - kv   5:                         general.size_label str              = 7B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                       general.license.link str              = https://huggingface.co/Qwen/Qwen2.5-7...
llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
llama_model_loader: - kv   9:                  general.base_model.0.name str              = Qwen2.5 7B
llama_model_loader: - kv  10:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  11:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen2.5-7B
llama_model_loader: - kv  12:                               general.tags arr[str,2]       = ["chat", "text-generation"]
llama_model_loader: - kv  13:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv  14:                          qwen2.block_count u32              = 28
llama_model_loader: - kv  15:                       qwen2.context_length u32              = 32768
llama_model_loader: - kv  16:                     qwen2.embedding_length u32              = 3584
llama_model_loader: - kv  17:                  qwen2.feed_forward_length u32              = 18944
llama_model_loader: - kv  18:                 qwen2.attention.head_count u32              = 28
llama_model_loader: - kv  19:              qwen2.attention.head_count_kv u32              = 4
llama_model_loader: - kv  20:                       qwen2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  21:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  22:                          general.file_type u32              = 7
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  30:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  33:               general.quantization_version u32              = 2
llama_model_loader: - kv  34:                      quantize.imatrix.file str              = /models_out/Qwen2.5-7B-Instruct-GGUF/...
llama_model_loader: - kv  35:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt
llama_model_loader: - kv  36:             quantize.imatrix.entries_count i32              = 196
llama_model_loader: - kv  37:              quantize.imatrix.chunks_count i32              = 128
llama_model_loader: - type  f32:  141 tensors
llama_model_loader: - type q8_0:  198 tensors
llm_load_vocab: special tokens cache size = 22
llm_load_vocab: token to piece cache size = 0.9310 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = qwen2
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 152064
llm_load_print_meta: n_merges         = 151387
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 3584
llm_load_print_meta: n_layer          = 28
llm_load_print_meta: n_head           = 28
llm_load_print_meta: n_head_kv        = 4
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 7
llm_load_print_meta: n_embd_k_gqa     = 512
llm_load_print_meta: n_embd_v_gqa     = 512
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 18944
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q8_0
llm_load_print_meta: model params     = 7.62 B
llm_load_print_meta: model size       = 7.54 GiB (8.50 BPW) 
llm_load_print_meta: general.name     = Qwen2.5 7B Instruct
llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOS token        = 151645 '<|im_end|>'
llm_load_print_meta: PAD token        = 151643 '<|endoftext|>'
llm_load_print_meta: LF token         = 148848 'ÄĬ'
llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
llm_load_print_meta: EOG token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOG token        = 151645 '<|im_end|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: ggml ctx size =    0.30 MiB
llm_load_tensors: offloading 28 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 29/29 layers to GPU
llm_load_tensors:      ROCm0 buffer size =  7165.44 MiB
llm_load_tensors:        CPU buffer size =   552.23 MiB
.......................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      ROCm0 KV buffer size =   112.00 MiB
llama_new_context_with_model: KV self size  =  112.00 MiB, K (f16):   56.00 MiB, V (f16):   56.00 MiB
llama_new_context_with_model:  ROCm_Host  output buffer size =     1.16 MiB
llama_new_context_with_model:      ROCm0 compute buffer size =   304.00 MiB
llama_new_context_with_model:  ROCm_Host compute buffer size =    11.01 MiB
llama_new_context_with_model: graph nodes  = 986
llama_new_context_with_model: graph splits = 2
llama_init_from_gpt_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
:0:/home/saidp/Downloads/rocm_sdk_builder/src_projects/clr/hipamd/src/hip_global.cpp:114 : 1173855977 us: [pid:16546 tid:0x7e1f3d1ecc40] Cannot find Symbol with name: Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1
Aborted (core dumped)

Here is the full log with AMD_COMGR_SAVE_TEMPS=1 AMD_COMGR_REDIRECT_LOGS=stdout AMD_COMGR_EMIT_VERBOSE_LOGS=1 ROCM_SDK_PRINTOUT_DEBUG_MESSAGES=1 enabled. sdk_error_output.txt

Said-Akbar commented 1 day ago

@lamikr , I see AMD MI50 (also gfx906) costs around $140 on eBay with shipping. Let me know if you are open to the idea of supporting gfx906. I am willing to ship one of those to you. Or else, if you are in the Bay Area, I can lend you one MI50. This way it will be easier for you to debug and fix issues. Thanks!

lamikr commented 1 day ago

@Said-Akbar Thank you for the suggestion, it would be great if I could loan one of your gfx906 for a while for testing. I live on the bay area but I travel also quite often also to San Francisco if that's easier for you. Are you able to send me a private message to gmail or linked-in?

I just bought one gfx1010 from ebay to better test on rdna1 cards so I would like to hold a while before purchasing the gfx906.

Said-Akbar commented 17 hours ago

sure, let me send you a linkedin message.

lamikr / rocm_sdk_builder

gfx906 (AMD MI60) is failing on run_and_save_benchmarks.sh and llama.cpp #180