search

KimMeen / Time-LLM

[ICLR 2024] Official implementation of " 🦙 Time-LLM: Time Series Forecasting by Reprogramming Large Language Models"
https://arxiv.org/abs/2310.01728
Apache License 2.0
1.29k stars 221 forks source link

Question about text prototype in reprogramming #73

Closed ztb-35 closed 4 months ago

ztb-35 commented 5 months ago

Hi, there. Thanks for publishing your code. I'm interested in your patch reprogramming. But I didn't find your text prototype in your code TimeLLM.py image This is only the linear projection. I'm not sure if I misunderstand the code. Thanks for your reply.

celestialxevermore commented 5 months ago

It seems like you've caught on quite well. The authors mentioned in the paper, "A simple solution is to maintain a small collection of text prototypes by linearly probing E, denoted as E'." The part you mentioned about the mapping_layer likely corresponds to this.

ztb-35 commented 5 months ago

I think they import the whole model word-embedding weight and then get the text prototype by this linear layer("self.mapping_layer").

kwuking commented 4 months ago

I think they import the whole model word-embedding weight and then get the text prototype by this linear layer("self.mapping_layer").

Yes, your understanding is correct. We have also provided a detailed description in the "Patch Reprogramming" section of our paper, which you can refer to.