deepinfra / deepinfra-node

Official TypeScript wrapper for DeepInfra Inference API
https://deepinfra.com/
MIT License
8 stars 0 forks source link

Accept client parameters during model initialization #18

Closed ichernev closed 4 months ago

ichernev commented 5 months ago

At the moment, the DeepInfraClient has readonly fields about the retry params:

export class DeepInfraClient {
  private axiosClient: AxiosInstance;
  private readonly maxRetries: number = MAX_RETRIES;
  private readonly initialBackoff: number = INITIAL_BACKOFF;
  private readonly subsequentBackoff: number = SUBSEQUENT_BACKOFF;
  ...
}

Ideally these will be adjustable from the user. So maybe create a config interface with all relevant fields and pass it to the Model/Client constructors.

interface ClientConfig {
  maxRetries?: number;
  initialBackoff?: number;
  subsequantBackoff?: number;
}

DEFAULT_CONFIG: ClientConfig = {...};

Then the Model class (assuming #17 goes through):

const model  = new TextGeneration(MODEL_NAME, TOKEN, {maxRetries: 1});

i.e optional ClientConfig 3rd argument to overwrite the defaults.

On a related note backoff is normally defined with min, max and coef -- i.e starts from min, then min * coef, then min * coef * coef, etc, but never more than max.