search

JetBrains-Research / coqpilot

VSCode extension that is designed to help automate writing of Coq proofs.
GNU Lesser General Public License v2.1
47 stars 2 forks source link

Any insight into API cost? #10

Closed adrianleh closed 9 months ago

adrianleh commented 9 months ago

Hi all,

very interested in trying this project. Do you have any insight into the API cost that I should expect to incur? Ofc, this will depend on a number of factors but I think having some general use-cases with different cost estimates might be helpful for other users, so any insights are appreciated.

K-dizzled commented 9 months ago

Hi @adrianleh! You are right, it mostly depends on the number of tries for a particular goal and on the model you have chosen. While testing, we have experienced very low API costs when using gpt-3.5, and relatively big for gpt-4. For a fixed number of 30 tries per each hole I can give you the following numbers: for gpt-3.5 I have paid no more than 3$ per month while using and testing the plugin quite a lot; for gpt-4 numbers were around 8 times as big.

adrianleh commented 9 months ago

Thanks! Is is definitely useful! Happy to close the issue for now