search

pytorch / executorch

On-device AI across mobile, embedded and edge for PyTorch
https://pytorch.org/executorch/
Other
1.31k stars 206 forks source link

Executorch reports a bug for pages and pages: [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed. #3515

Open mikekgfb opened 1 week ago

mikekgfb commented 1 week ago

https://github.com/pytorch/torchchat/actions/runs/8955682937/job/24596656941?pr=680

Is this just an internal message we should suppress because it makes me worried as a user that the program has a bug, or is this a real bug that needs to be fixed and not just printed to the screen.

Executorch produces too much debug output, and it degrades user confidence and experience. If this is rational development debug output, let's provide a way to suppress at least the most frequent items (the overriding of daya pointer item has been ongoing for a while...

cc: @orionr @iseeyuan @byjlw 

  + python3 torchchat.py generate stories15M --device cpu --pte-path stories15M.pte --prompt 'Hello my name is'
  [program.cpp:130] InternalConsistency verification requested but not available
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  Warning: checkpoint path ignored because an exported DSO or PTE path specified
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  Warning: checkpoint path ignored because an exported DSO or PTE path specified
  Using device=cpu Apple M1 (Virtual)
  Loading model...
  Time to load model: 0.03 seconds
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  Hello my name is Will. Will is a very clever pig. He has been living in a great outfit and ever since.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  Everyday he likes to take a walk and enjoy the sun. He walks over the hills and he walks very slow.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  One day he saw something shiny on the ground. He wanted to get a closer look and decided to rest. He slowly and carefully stepped onto the ground and looked around. He kept walking and kept walking until he was tired.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  He lay down on the ground and soon fell. He was very worried and he was still very tired. He couldn't move and he knew he couldn't move.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  [method.cpp:939] Overriding output data pointer allocated by memory plan is not allowed.
  He stayed there and never told anyone this happen again. He was so weak and he could not move and he never remembered it. And that's how Will had been terrified of this. Once upon a time, there was a little girl named Lily. She had a pet rat named Jerry.
  Max Sequence Length Reached. Ending Conversation.
  ==========
  + exit 0
  + echo 'tests complete'
  tests complete
  *******************************************
  + echo '*******************************************'
  + echo ::endgroup::
JacobSzwejbka commented 1 week ago

I think the logs from ET method.cpp are correct, the problem is a missing api to check if tensors are already memory planned. We can add that to method_meta probably.