Usability issues with the inline completer

jupyterlite / jupyterlab-codestral

AI code completions and chat for JupyterLab, Notebook 7 and JupyterLite, powered by MistralAI ✨

http://jupyterlite.github.io/jupyterlab-codestral/

BSD 3-Clause "New" or "Revised" License

6 stars 2 forks source link

Usability issues with the inline completer #16

Open jtpio opened 2 weeks ago

jtpio commented 2 weeks ago

When trying https://github.com/jupyterlite/jupyterlab-codestral/pull/15 locally with MistralAI, the inline completer seems to be "off" most of the time, making it difficult to use:

I believe this was already noticed in https://github.com/jupyterlite/jupyterlab-codestral/pull/8 and may be the same issue.

It's not clear yet whether it's because the Mistral API is slow to respond, because of its 1 req/s limit or how the inline completer is currently set up.

brichet commented 2 weeks ago

I think it's a mix of all of these.

the Mistral API seems to take around 2s to respond on my hand (and sometime the first requests never ends).
currently we trigger the API as soon as a change occurs, and the following changes during the next second are not taken into account.

Maybe we could make it more reliable with a better use of the inline completer and the throttler. I'm thinking about:

not triggering the API on each change. We could wait a small timeout (200ms) to expect the user is not typing anymore.
after having trigger the API, keeping track of the following changes would allow us to cancel the next suggestion (which is probably no longer valid) and trigger another as soon as possible.

jtpio commented 2 weeks ago

the Mistral API seems to take around 2s to respond on my hand (and sometime the first requests never ends).

2s sounds like a lot. Wondering if we can already see some improvements with other providers like in https://github.com/jupyterlite/jupyterlab-codestral/pull/17 for example. If that's the case maybe there isn't much we can do about the Mistral provider.

brichet commented 2 weeks ago

the Mistral API seems to take around 2s to respond on my hand (and sometime the first requests never ends).

2s sounds like a lot. Wondering if we can already see some improvements with other providers like in #17 for example. If that's the case maybe there isn't much we can do about the Mistral provider.

As far as I know, Groq does not provide a completion model out of the box (not with langchain at least).

jtpio commented 2 weeks ago

OK, we could try with OpenAI then.