ChatGPT baseline experiments.

[MM-IR]

Hi Jiannan, would you please release some of your code/prompt for ChatGPT baselines? I am planning to reuse some in my projects.

However, from your work, if I understand correctly, it seems that for any LM evaluation, "you implement the same in-context exemplars" shown in your appendix? Are my understandings correct?

If so, I can simply follow the instructions provided in your appendix to reproduce some experiments, thanks in advance for your help!

Thanks!

[MM-IR]

When running with ChatGPT API, have you faced with the issues of "Lost connection due to overload with other requests"? May I ask you how do you address it? Thanks!

[szxiangjn]

Hi,

Besides in-context exemplars, we also add some additional instructions. Specifically, in the prompt we will describe the task first and then ask ChatGPT to output the answer directly. I may release them soon after I clean up the code.

Regarding the API overload issue, you may want to reduce your API calling frequency.

[MM-IR]

Thanks for your patience! Let me close this!

So is that mean "you have not met such issue even on so many datasets through your experiments?"

That is surprising to me, as many blogs talk about highly relevant things..

[szxiangjn]

I do have encountered the issue before. A quick solution could be catching the error, suspending the program (like use time.sleep), and retrying it again.

[MM-IR]

Thanks!