search

ContextLab / chatify

Add ipython magic commands to Jupyter notebooks that provide LLM-driven enhancements
MIT License
22 stars 10 forks source link

Chatify asks for user response #47

Closed jrudoler closed 1 year ago

jrudoler commented 1 year ago

Describe the bug Chatify politely asks for the user to clarify what it can help debug, but as far as I can tell it is not possible to interact with the LLM in a back-and-forth chat (the widget has no text entry / chat box, at least).

To Reproduce Steps to reproduce the behavior:

  1. Go to this tutorial notebook
  2. Scroll down Section 1: Decoding - Investigating model and evaluating performance. Expand the third cell in this section, which should start with # @markdown Execute this cell to train the network and then defines the DeepNetReLU class.
  3. Add some syntactical bug to the cell - e.g. delete a colon in the first line of the training loop. Or don't add any bugs - the behavior is still not expected.
  4. Add the %%explain magic to the top of the cell and run it - then from the dropdown in the Robo-tutor tab, select "get help with debugging" and hit "Submit request"

Expected behavior I expected the Robo-tutor to give me a concrete response, either pointing out a problematic line of code or claiming that the code is error-free.

Screenshots

Screenshot 2023-08-09 at 2 06 35 PM

Desktop (please complete the following information):

HemuManju commented 1 year ago

@jrudoler, thank you for feedback 🙂! Did you try re-running the %%explain magic function?

I followed the steps to reproduce the error, but couldn't reproduce it. Here are a few screenshot of responses: without error with colon error First response is without any error in the code and the second picture is removing a colon in the for loop under train functions.

Please note that responses from LLMs are not always accurate.

jrudoler commented 1 year ago

Yep, it's still producing the same error for me (after starting from scratch multiple times):

Screenshot 2023-08-10 at 2 42 36 PM

I often get some variation of "What's the issue?" or "What seems to be the problem?".

By repeatedly clicking "submit request" I can get it to actually produce a response catching the bug sometimes:

Screenshot 2023-08-10 at 2 47 45 PM

My sample size is small, but the error came up probably 4/5 times.

HemuManju commented 1 year ago

So the response discrepancy seems to stem from the GPT side, which we cannot directly control. Nonetheless, we will keep this in mind while updating the prompts in future release. I will close this for now.