mistralai / client-js

JS Client library for Mistral AI platform
Apache License 2.0
185 stars 46 forks source link

Add support for aborting (preferably via AbortSignal) #5

Closed sublimator closed 6 months ago

wordware-ai commented 11 months ago

Available here: https://www.npmjs.com/package/mistral-edge Just pass an abort signal into the config parameter

e.g.

const abortController = new AbortController();
const tokenStream = streamMistralChat(
    [{ role: "user", content: "Write a haiku about cheese" }],
    {
      model: "mistral-medium",
      temperature: 0.7,
    },
    {
      signal: abortController.signal
    }
);

// Do what you like with the stream and abort it if needed
sneko commented 8 months ago

In addition to using explicitly the signal property, it would be great if the library was really dealing with the timeout property. Currently it has no impact and it defaults to 5 minutes of timeout...

I had to patch with patch-package to work around thanks to a signal:

    const url = `${this.endpoint}/${path}`;
    const options = {
      method: method,
      ...
      timeout: this.timeout * 1000, // Not working
      signal: AbortSignal.timeout(this.timeout * 1000), // This makes the timeout working
    };

@Bam4d it would be great if this repo could have more love from the core team. (cf. https://github.com/mistralai/client-js/graphs/contributors)

EDIT: note that in development environment the client forces using node-fetch that returns an AbortError, whereas once compiled for production into a web app (server-side), it will be a DOMException. So watch out 😄