search

westlake-repl / SaProt

[ICLR'24 spotlight] Saprot: Protein Language Model with Structural Alphabet
MIT License
323 stars 31 forks source link

Training SaProt with a smaller ESM model #45

Open BSharmi opened 3 months ago

BSharmi commented 3 months ago

Hi there!

Awesome work! Do you have a script that I can use to train SaProt with a smaller ESM model? I think I can use the config https://github.com/westlake-repl/SaProt/blob/main/config/pretrain/saprot.yaml but from the name https://github.com/westlake-repl/SaProt/blob/main/config/pretrain/saprot.yaml#L14 it looks like the model is already trained?

Would appreciate any pointers!

Thank you!

LTEnjoy commented 3 months ago

Hi!

You can definitely use the config to train a smaller SaProt. The line 14 doesn't mean the model is trained. It is just the name of the training log. You actually train it from scratch.

If you want to train a smaller version, you have to first create a folder containing the configuration of your model like SaProt_650M_AF2 or SaProt_35M_AF2, except that you do not have to inlclude pytorch_model.bin. Then you only need to change the config_path to your folder path so you can train your own model. image

Hope this could resolve your problem:)

Binyun-Z commented 1 week ago

I have the same question, if I train Saport on a small-scale dataset, how should I design my configuration file?

LTEnjoy commented 1 week ago

I have the same question, if I train Saport on a small-scale dataset, how should I design my configuration file?

Hi. Do you mean by training on a smaller dataset or training a smaller model?