search

seonghyeonye / TAPP

[AAAI 2024] Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following
https://arxiv.org/abs/2302.14691
MIT License
79 stars 2 forks source link

How much does your experiment cost（per subtask/bash）? #2

Closed 2018211801 closed 1 year ago

2018211801 commented 1 year ago

I just wanted to reproduce your run_ICIL part of the experiment, and it turned out that it cost 28¥ before the experiment was finished!!!Is there a way to save it?

2018211801 commented 1 year ago

I noticed that your test dataset has 11810 examples, how many examples can I put at least?

hbin0701 commented 1 year ago

Please be aware that replicating our experiment may involve certain costs, considering the usage of the davinci model and the large number of examples (11810).

To answer your first question, please note that the predicted results will automatically be saved in the predicted_examples.jsonl file within the output directory. If you run the experiment again with the same output path, the model will skip the samples whose responses have already been generated and will only append the results for the remaining samples.

Regarding your second question, you have the option to modify the list of tasks in the data/splits/default/test_tasks.txt file to include only a single task, which will then result in 100 instances in total. If you wish to further minimize the number of instances, you can adjust the max_num_instances_per_eval_task parameter in your run_ICIL.sh file to the desired number.

Now, To provide you with a rough estimate of the cost, since we are forcing a 2048 max input length, and the cost is $0.02 per 1K token, running a single task with 100 instances will result in $4. In a similar manner, fully replicating our results (11810) will result in $400~$500.

2018211801 commented 1 year ago

Thank you very much for your patient answer. I was too careless. I have decided to try the open source model GPT-J 6B first.