Output 0 of ViewBackward0 is a view

[clechristophe]

Hi,

I am trying to reproduce the experiment by finetuning a Llama2 model and I have the following error when doing the forward pass to get the embeddings:

"Output 0 of ViewBackward0 is a view and its base or another view of its base has been modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one."

Did you have the same error ? I think it is a problem with fsdp but I am using the same wrapping policy as you did

[neelsjain]

Usually, you would get that error if the embedding is not shared properly. You might want to add a print(model) to double check that the embedding is in fact wrapped individually as a "_fsdp_wrapped_module" object. The other thing you might want to check is the transformers version and the pytorch version. Pytorch constantly updates the FSDP.

[clechristophe]

Thank you ! You were right, the wrapping policy wasn't good. Now I am able to do the forward through the embedding but I have an error when I go through the whole model model(**batch):

Non-root FSDP instance's _is_root should not have been set yet or should have been set to False

Have you ever seen this error ? I am using a more recent version of pyTorch for other reasons and was wondering if that might be the cause

[neelsjain]

I have not seen this error before, but it could be related to the wrapping policy again