Fix bottleneck in GPU performance (pipeline batching)

rbroc / echo

A Scalable and Explainable Approach to Discriminating Between Human and Artificially Generated Text

2 stars 1 forks source link

Closed MinaAlmasi closed 1 year ago

MinaAlmasi commented 1 year ago

Fixed pipeline batching for all models, reducing inference using GPUs drastically (going from 30 hours to 1-2 hours).
In the midst of streamlining prompts.
Introduced a logging system which saves completions to logfiles to save progress if script needs to be terminated before finishing the inference.