Compilation error on llama 7 B with batch size 8

[dacorvo]

When trying to compile llama 7B with:

• batch_size: 8,
• n_positions: 2048,
• tp_degree: 24,
• auto_cast_type: f16.

I get these errors (multiple instances):

2023-Nov-20 08:13:58.361948  4241:4538  ERROR  NMGR:dlr_kelf_load                           Failed to load mlaop                                                                                                                    
2023-Nov-20 08:13:58.361956  4241:4538  ERROR  NMGR:load_kelf_graphs                        Failed to load KELF kelf-0.json                                                                                                         
2023-Nov-20 08:13:58.677772  4241:4551  ERROR  NEFF:json_parse_load_elements                Unable to parse: sg00/Activation.json - 1                                                                                               
2023-Nov-20 08:13:58.677837  4241:4551  ERROR  NEFF:json_parse_load_elements                File sg00/Activation.json size (4375834152) exceeds json parser maximum (4294967295)                                                    
2023-Nov-20 08:13:58.677857  4241:4551  ERROR  NEFF:construct_kbin                          Failed to load subgraph sg00/def.json                                                                                                   
2023-Nov-20 08:13:58.679344  4241:4551  ERROR  NEFF:kelf_load                               Failed to load subgraph 0                                                                                                               

2023-Nov-20 08:13:58.679362  4241:4551  ERROR  NMGR:dlr_kelf_load                           Failed to load mlaop                                                                                                                    
2023-Nov-20 08:13:58.679371  4241:4551  ERROR  NMGR:load_kelf_graphs                        Failed to load KELF kelf-0.json                                                                                                         
2023-Nov-20 08:13:59.027739  4241:4538  ERROR  NMGR:kmgr_load_nn_post_metrics               Failed to load NN: /tmp/ubuntu/neuroncc_compile_workdir/d7686f21-27f8-48c8-8ad6-241f83f4e865/model.MODULE_ddc3bb0a8f815a1d05f6+8737852b.
neff, err: 2                                             
2023-Nov-20 08:13:59.084440  4241:4551  ERROR  NMGR:kmgr_load_nn_post_metrics               Failed to load NN: /tmp/ubuntu/neuroncc_compile_workdir/d7686f21-27f8-48c8-8ad6-241f83f4e865/model.MODULE_ddc3bb0a8f815a1d05f6+8737852b.
neff, err: 2                                                                                                                                                                                                                        
2023-Nov-20 08:13:59.107413  4241:4547  ERROR  NEFF:json_parse_load_elements                Unable to parse: sg00/Activation.json - 1

I am using optimum-neuron example script with the following command:

python examples/text-generation/generation.py export meta-llama/Llama-2-7b-chat-hf --batch_size 8 --sequence_length 2048 --num_cores 24 --auto_casODULE_ddc3bb0a8f815a1d05f6+8737852b.t_type fp16

[aws-mvaria]

Thank you, we are taking a look and will get back to you shortly.

[aws-donkrets]

Hi dacorvo - we have been able to reproduce your issue and are working on a solution. The model compiled successfully; the issue is with loading the model onto the device for execution.

[dacorvo]

Same issue with Zephyr: https://huggingface.co/aws-neuron/optimum-neuron-cache/discussions/3

[dacorvo]

Now closed.