search

OpenBioLink / ThoughtSource

A central, open resource for data and tools related to chain-of-thought reasoning in large language models. Developed @ Samwald research group: https://samwald.info/
MIT License
863 stars 69 forks source link

save generated cots in case of error #102

Closed KonstantinHebenstreit closed 1 year ago

KonstantinHebenstreit commented 1 year ago

The generate function saves the generated cots, when it finished it for all examples. (see script of data-loader, which calls the function) If any error happens, all the generated cots from before are not saved and are therefor lost.

KonstantinHebenstreit commented 1 year ago

Much better now with loop for API Errors in "generate". But will still throw the error after five tries.

KonstantinHebenstreit commented 1 year ago

Generate is called per split of every dataset now. So the given split is lost but if generate is called a Collection of multiple datasets with multiple splits, everything up to the split where an Error occurs multiple times is saved.

The loop inside generate which catches API Errors is done five times (per item). But the Error is raised after that, since we do not want it to loop endlessly.