It can be annoying if the user is editing a markup cell with a lot of exposition with no clear commands to be run but Foyle keeps providing suggestions which aren't relevant. We'd like the model to decide when the text doesn't suggest a command to be run. To do that well we need to build an eval dataset for that.
One signal we could look for in our logs is sessions with
Long durations
Large numbers of completions generated without any being accepted
We could also look at consecutive sessions to potentially see if a user didn't enter a code cell after the current markup cell.
We could eventually link sessions to the previous and next session. The end and start session events are sent in a single request so the backend could link them.
It can be annoying if the user is editing a markup cell with a lot of exposition with no clear commands to be run but Foyle keeps providing suggestions which aren't relevant. We'd like the model to decide when the text doesn't suggest a command to be run. To do that well we need to build an eval dataset for that.
One signal we could look for in our logs is sessions with
We could also look at consecutive sessions to potentially see if a user didn't enter a code cell after the current markup cell. We could eventually link sessions to the previous and next session. The end and start session events are sent in a single request so the backend could link them.