Open michaelfeil opened 7 months ago
We have seen similar behavior with togethercomputer/LLaMA-2-7B-32K
You can also replicate the example above with this code:
#num_cores should be changed based on the instance. inf2.24xlarge has 6 neuron processors (they have two cores each) so 12 total
#larger models will need more cores. You can make your model smaller by changing fp16 to f8. Some models may requre num_cores to be a power of 2
compiler_args = {"num_cores": 2, "auto_cast_type": 'fp16'}
input_shapes = {"batch_size": 1, "sequence_length": 2048}
model_to_test = "deepseek-ai/deepseek-coder-6.7b-instruct"
model = NeuronModelForCausalLM.from_pretrained(model_to_test, export=True, **compiler_args, **input_shapes)
from optimum.neuron import pipeline
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_to_test)
p = pipeline('text-generation', model, tokenizer)
p("My favorite place on earth is", max_new_tokens=64, do_sample=True, top_k=50)
Output:
Setting 'pad_token_id' to 'eos_token_id':32021 for open-end generation.
2024-Jan-30 15:04:44.0384 58491:62420 [0] nccl_net_ofi_init:1415 CCOM WARN NET/OFI aws-ofi-nccl initialization failed
2024-Jan-30 15:04:44.0384 58491:62420 [0] init.cc:137 CCOM WARN OFI plugin initNet() failed is EFA enabled?
[{'generated_text': 'My favorite place on earth is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is iis is is is is is is is is isis is is is is'}]
Could this be the effect of Rope Scaling?
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!
I met the same issue, is there any updated?
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!
I am using the newest AMI image from yesterday, with optimum-neuronx 0.0.17 (https://aws.amazon.com/marketplace/pp/prodview-gr3e6yiscria2) I have not tried using another image yet.
I am trying to evaluate
AutoModelForCausalLM.from_pretrained("deepseek-ai/deepseek-coder-6.7b-instruct")
while using torch.cpu I get the following output, for the neuron equivalent, I get\n\n\n\n
* 512.Reproduction script:
Output: