Request for a quick start guidance

[jiahaolu97]

Hi, first thank you for bringing to the community such an interesting and insightful paper. While I want to play with the code, I found it really difficult to start, due to too many scripts and lack of document. Could you please provide one subsection in your RAEDME to guide us easily start training for local LLM models? I am sure by doing this, the repo will become even more popular. Many thanks in advance!

[jxmorris12]

Yeah sure! Thanks for asking. You're trying to train your own inversion model? For what embedding model? I'll add an example.

[jiahaolu97]

Yes! I am trying to train my own inversion model for models from Pythia library (https://github.com/EleutherAI/pythia). I think their architectures are very similar to GPT-3 (according to their paper) Thanks John, your paper is super interesting and enjoyable to read.

[jxmorris12]

No problem! What embeddings do you want to use though? Pythia is a decoder-only model right, so it's not obvious to me how to use it to create sentence embeddings.

[jiahaolu97]

Is it possible to invert any intermediate layer's embedding? For example, the embedding after the third decoder block.

[jxmorris12]

Sure, but transformers operate with one layer per token. So you could take the mean of all the token embeddings and that would sort of give you what you want. My code should actually do this pretty easily...

[kyriemao]

Hi Morris, grear work! I have the same kind request for the training Readme. I want to train my own inversion model based on ANCE embeddings (https://huggingface.co/castorini/ance-msmarco-passage).

[jiahaolu97]

That's awesome! Looking forward to your update 😸

[jxmorris12]

Thanks everyone. I'm fixing a few things to make training more straightforward and will give an update soon.

[jxmorris12]

Hi @RoyLJH -- I added a section to the README with training instructions. Let me know if you have any suggestions for how to improve it.

[jiahaolu97]

Thank you @jxmorris12 !