Checking AMX used in BF16: how to enable verbose?

[zsym-sjtu]

Describe the issue

I am testing AMX's performance in BF16 inference. It turns out that under different settings of DNNL_MAX_CPU_ISA (AVX512_CORE_AMX AVX512_CORE_BF16 AVX512_CORE_VNNI), the inference time with IPEX is the same. I wonder whether AMX is used or whether my code is correct. Such DNNL_MAX_CPU_ISA way is effective when using oneDNN without IPEX.

So does IPEX use DNNL_MAX_CPU_ISA to determine the ISA used? How can I enable or disable AMX in IPEX?

Then I learned that, as in IPEX Fusions for BF16, IPEX doesn't use oneDNN in BF16. For this reason I can't access the oneDNN operations by setting ONEDNN_VERBOSE.

When I try to use with ipex.verbose(ipex.verbose.VERBOSE_ON):, there is error:

Traceback (most recent call last):
  File "bench-ipex.py", line 125, in <module>
    with ipex.verbose(ipex.verbose.VERBOSE_ON):
AttributeError: type object 'verbose' has no attribute 'VERBOSE_ON'

Following is my code

def benchmark():
    if (sys.argv[1] == 'opt'):
        model = AutoModelForCausalLM.from_pretrained(model_path).eval()
        dtype = torch.bfloat16 # or torch.float ?
        model = ipex.llm.optimize(model, dtype=dtype)

    elif (sys.argv[1] == 'base'):
        model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16).eval()
    else:
        print("arg 1 must be base/opt")
        return

    tokenizer = AutoTokenizer.from_pretrained(model_path)
    tokenizer.pad_token = tokenizer.eos_token

    inputs = tokenizer(input_text, return_tensors="pt", padding=True, truncation=True, max_length=1024)
    attention_mask = inputs['attention_mask']

    for i in range(warmup):
        output = model.generate(
            inputs["input_ids"],
            attention_mask=attention_mask,
            max_new_tokens=128,
            num_return_sequences=1,
            pad_token_id=tokenizer.pad_token_id
            )
        print("Warmup " + str(i))

    tick = time.time()

    for i in range(iterations):
        t0 = time.time()
        output = model.generate(
            inputs["input_ids"],
            attention_mask=attention_mask,
            max_new_tokens=128,
            num_return_sequences=1,
            pad_token_id=tokenizer.pad_token_id
            )
        #print("Timing iteration " + str(i))
        print("Timing iteration " + str(i) + ": " + str(time.time() - t0))

    tock = time.time()

    generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
    print(generated_text)

    print((tock - tick) / iterations)

My ipex version is 2.4.0+cpu.

Many thanks!

[ZailiWang]

Hi, would you tell your CPU model? Can be acquired by lscpu command.

[ZailiWang]

Traceback (most recent call last): File "bench-ipex.py", line 125, in with ipex.verbose(ipex.verbose.VERBOSE_ON): AttributeError: type object 'verbose' has no attribute 'VERBOSE_ON'

This is a bug, we'll fix it soon, thanks.

[zsym-sjtu]

Hi, would you tell your CPU model? Can be acquired by lscpu command.

lscpu

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 192 On-line CPU(s) list: 0-191 Thread(s) per core: 2 Core(s) per socket: 48 Socket(s): 2 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 143 Model name: Intel(R) Xeon(R) Platinum 8475B Stepping: 8 CPU MHz: 3200.000 CPU max MHz: 3800.0000 CPU min MHz: 800.0000 BogoMIPS: 5400.00 Virtualization: VT-x L1d cache: 48K L1i cache: 32K L2 cache: 2048K L3 cache: 99840K NUMA node0 CPU(s): 0-191 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm uintr md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities

[ZailiWang]

Ah... The reason that setting DNNL_MAX_CPU_ISA does not take effect is that, as you found out, oneDNN backend is not used. So AMX should be utilized no matter what you set for DNNL_MAX_CPU_ISA (as it only affects oneDNN) on your 4th gen Xeon. Unfortunately the utilized backend does not provide verbose log functionality to check the utilization of AMX instructions.

[zsym-sjtu]

Then is there any method provided by IPEX or libxsmm to determine whether to use AMX (or AVX-512 etc.) ? Thanks!

[ZailiWang]

Unfortunately libxsmm automatically match the best supported instruction sets, so we cannot disable AMX on your server...

[zsym-sjtu]

Thanks for answering!