Persistent Python session / Jupyter Notebook integration

[vtenfys]

Duplicates

• [X] I have searched the existing issues

Summary 💡

I'd like to use Auto-GPT with a Jupyter Notebook. My proposal is that Auto-GPT should be able to:

• run the notebook server
• access notebooks via the server
• create, remove, modify and delete cells
• run code within cells

As it stands, Auto-GPT is able to run Python code, but the execution environment does not persist between scripts.

See Jupyter Server REST API: https://jupyter-server.readthedocs.io/en/latest/developers/rest-api.html

Alternatively, it should have the ability to start a persistent Python session, allowing variable reuse across scripts.

See https://github.com/Significant-Gravitas/Auto-GPT/issues/5132#issuecomment-1684356928 for a proof-of-concept

Examples 🌈

ChatGPT Plugin for Noteable: https://noteable.io/chatgpt-plugin-for-notebook/

ChatGPT Code Interpreter: https://openai.com/blog/chatgpt-plugins#code-interpreter

Open Interpreter - alternative to ChatGPT CI: https://github.com/KillianLucas/open-interpreter

Motivation 🔦

I would like Auto-GPT to work with Python code and retain variables between execution, as well as being able to perform analysis and display visualisations etc. This would be useful for data science and similar fields.

Related issues

Semi-related: #3536

[vtenfys]

Hacky proof-of-concept solution without requiring any codebase changes:

• Start Auto-GPT, then before entering the prompt:
• Connect to the container, install and start a notebook server
• Prompt Auto-GPT to connect to the notebook server, and:
  • create a notebook
  • start a kernel
  • insert and run some cells

I've had some success using this method, although understandably it's a bit awkward

[fredrik-hansen]

May I suggest looking at https://jupyter-ai.readthedocs.io/en/latest/users/index.html#installation ?

[pplonski]

I'm interested in such integration as well. I'm looking into something more than Jupyter-AI. I would like to see notebook that is deeply integrated with LLM agent:

• I can prompt LLM agent and it will generate code as cells and execute cells
• LLM agent outputs will be saved in the notebook
• I can reuse created notebook later without accessing LLM agent, for example schedule notebooks periodically as reports
• I would like to have more widgets (not only chat) to interact with LLM agent, for example widgets to give LLM agent a feedback, or widgets with parameters for prompt template

[ntindle]

This seems really useful. Any ideas for implementation? Maybe @Swiftyos has ideas this in the forge?

[Git-Discord]

bentlybro: just a test comment ignore this

• Posted By bentlybro Using The Git-Discord Bot From The AutoGPT Discord

[Swiftyos]

I tend to add a jupyter lab to my projects. Like so:

@main.command()
def notebook():
    """
    Creates a jupyter lab notebook at localhost:8888
    """
    import subprocess
    subprocess.run(
        ['jupyter', 'lab', '--port=8888', '--no-browser',
            '--ip=0.0.0.0', '--allow-root'],
        stdout=subprocess.PIPE
    )

This could be a good way of opening up experimentation more easily. I'll look into adding it

[github-actions[bot]]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions[bot]]

This issue was closed automatically because it has been stale for 10 days with no activity.

[Pwuts]

Unstale, this is a great idea!

[Pwuts]

It may be valuable to combine this with #6253. This combination would make the experience of using the feature very similar to ChatGPT with the integrated code interpreter. For example, then the LLM could generate output like this:

Since you requested to hash the string "Hash this!", I will calculate the 256-bit SHA2 hash:

<python>
import hashlib

hash = sha256("Hash this!").hexdigest()
hash
</python>

Then AutoGPT executes that code, retrieves the result bec78476925e4cda6b22e91551ce4337264bdc3394c4f8297ad238f67a436d0e, adds it to the prompt, and prompts the LLM again so it can generate its output.

[MKdir98]

How can I start this one? I sent messages to most of the maintainers in discord.

[ntindle]

A good place to start would be making a notebook and extracting the agent loop of the existing code to it. Most things can be imported around that for now

[MKdir98]

A good place to start would be making a notebook and extracting the agent loop of the existing code to it. Most things can be imported around that for now

Thanks for your response. For start of jupyter user should run something like ./run jupyter agent autogpt and after that user will see notebook in 8888?

[ntindle]

That would be great

[MKdir98]

Hey guys. Do you think this is OK? I just run ./run jupyter start autogpt And after that notebook will show in browser and I just do this

[MKdir98]

https://github.com/Significant-Gravitas/AutoGPT/pull/6809

I create this one for review. Could you check it, please? And do we have something for merging or just passing pipelines is enough?

[ntindle]

This is a great start! Left a comment. Lmk if you think that’s too much work for one PR. @Pwuts is the owner of this task but he’s OOO this week so filling in

[MKdir98]

As you said here it's good to have some notebooks for developing. Could you give some examples? One_shot prompting example can be one of them? I think we need a structure for this to be clean. Maybe we can create a notebook for every doc page we have.

[MKdir98]

OK. As we talked in here I misunderstood the issue. I read it again.

Let me explain, and please tell me if I'm wrong. In commands part we have something as execute_python_code. We want to save session of this one is each task. We can do this with Jupyter cells. But It will save on memory, and it will delete variables in end of Jupyter. I searched and find dill to store all variables in a file in each step of execute_python_code. Is it correct? @Pwuts

[Pwuts]

@MKdir98 to clarify:

• Yes, the goal is to persist the effects of execute_python_code executions, so that the agent can achieve results step by step instead of having to execute everything at once.
• The proposal is to use Jupyter to achieve the above, and to also allow including plain text (e.g. the agent's reasoning) in the resulting notebook.
• I'm not familiar with dill. If Jupyter doesn't natively persist the state of notebooks, I would hope and expect that solutions for that are readily available.
  • For now, I think memory-only persistence is fine. We can probably keep the Jupyter notebook active until the agent is terminated, and run the notebook again from start to finish to try and restore the state if needed. It isn't a clean solution, but it should work in most cases.

Further note:

• Jupyter(Hub) needs to be set up for untrusted users, to disallow agents to interfere with each other and with the host environment.

[MKdir98]

Hi again. I created a PR for review. https://github.com/Significant-Gravitas/AutoGPT/pull/7078. And also could you clarify Further note part? @Pwuts

Jupyter(Hub) needs to be set up for untrusted users, to disallow agents to interfere with each other and with the host environment.

[MKdir98]

I think we can save the state of the session in files. I wrote it to add it in the future.

[github-actions[bot]]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions[bot]]

This issue was closed automatically because it has been stale for 10 days with no activity.