[Help Wanted] Add Test Suite Toolchain

[ntindle]

We need to add a test suite toolchain for code stability

• [x] Tests run in Pipeline
• [ ] Integration for installing plugins
• [ ] Integration tests for each plugin

[BaseInfinity]

Hi there I may be interested in helping out but would like to get a little more details on expected outcomes =)

Like are we looking for a suite of Unit + Integration tests?

Does the test-suite have to test the plugins itself or does the suite just have to test that they are loaded correctly?

What about first party plugins? Are those expected to have more in depth tests as well?

Thanks!

[ntindle]

Right now, the test suite should run the existing tests so it can be put into automation.

Ideally each plugin would have their own unit tests and we’d have a comprehensive suite of integration tests but that is a longer term goal

[ntindle]

The expected outcomes for today are: Create a workflow that clones the repo, installs dependencies, runs pytest, reports the results and uploads it to codecov.

Most of that can be referenced from the ci pipeline for the main repo

[BaseInfinity]

Okay great! I didn't realize there were tests in place already. I'm pretty comfortable with GH Actions and ive setup these types of workflows before at my day job so I'll take a peek at the ci pipeline for the main repo and see if this is something I can commit to =)

[BaseInfinity]

ive been super pre-occupied catching up with the tech and playing with all the plugins, I should have a chance to look over this on the weekend

[h4ck3rk3y]

Hey @ntindle I created https://github.com/Significant-Gravitas/Auto-GPT-Plugins/pull/123 that adds the CI from the main repo here.

I can see the GH actions run fine on my fork https://github.com/h4ck3rk3y/Auto-GPT-Plugins/pull/3 but I think it hides them for me in the PR as I am not admin.

Would love to help with the integration testing piece! What are you thinking with that? What about spinning up AutoGPT, loading the plugin giving it some prompts and seeing if it responds as expected? Or is it enough to test that the latest image continues to load with the plugin?

[BaseInfinity]

The power of open source :)!

[BaseInfinity]

@h4ck3rk3y: Would love to help with the integration testing piece! What are you thinking with that? What about spinning up AutoGPT, loading the plugin giving it some prompts and seeing if it responds as expected? Or is it enough to test that the latest image continues to load with the plugin?

I was thinking similar lines. Spin up autogpt with plugins and an assert there are no errors on start

Then maybe start adding basic "smoke" tests that runs an example command from each plugin to make sure the plug-in isn't completely borked?

Over time I suspect the more intricate unit/integration test cases come from the authors of the plugins themselves?

[h4ck3rk3y]

Hey @BaseInfinity @ntindle to keep the conversation going -

Here's an example of a nightly run that checks whether the plugin loads as expected; it tests it for three different plugins (but all of them have the same zip)

https://app.circleci.com/pipelines/github/kurtosis-tech/autogpt-package/72/workflows/eac40ed0-17a5-4849-9bf4-1df6258f05b4

Here's the code for it https://github.com/kurtosis-tech/autogpt-package/blob/main/.circleci/config.yml#L41-L69

Would you be open to a similar GH action on this repo?

[ntindle]

Yes, in GHA and we need a time to talk about the install method. Really would love cross (all) platform support though. Is it on the roadmap?

[h4ck3rk3y]

You were very prompt to respond!

Hey @ntindle Kurtosis runs on Linux, MacOS and Windows(via WSL)! Does that work for you? Or do you want native Windows support? We have it on the roadmap but haven't prioritized it yet as we haven't seen much demand for a Windows native client.

Is Windows support a blocker for a GitHub Action that does the integration test?

Here are the installation instructions.

[ntindle]

It's a blocker for being a recommend install method for the whole project but not for actions. Sorry for being unclear

[h4ck3rk3y]

That makes sense; I'll explore how quickly we can get to a native Windows client! In the meantime I'll try to send you a GHA PR to spin up & test plugin installations.

[victorcolombo]

Hey @ntindle! We started to publish a native Windows binary that does not require WSL! We updated the documentation here https://docs.kurtosis.com/next/install/

[BaseInfinity]

I haven't used Kurtosis myself but I'm wondering what the advantage of using something like that versus using a prebaked image for the OS we want: https://github.com/actions/runner-images

I'm always a fan of using as many tools that GH provides when choosing to work with GHA for compatibility/stability.

I'm also open to Kurtosis, like with any other tool I just want to make sure I understand what the advantages of it are =)

[h4ck3rk3y]

@BaseInfinity You can run Kurtosis on top of the pre baked images like in this PR https://github.com/Significant-Gravitas/Auto-GPT-Plugins/pull/131

Think of Kurtosis as a powerful alternative to Python + Bash + Docker Compose.

If you have Kurtosis installed on your system you can have AutoGPT running against the latest stable image published by AutoGPT with a lot of plugins supported out of the box

This snippet here runs AutoGPT with the Twitter Plugin and then gives you the terminal you are accustomed too;

kurtosis run github.com/kurtosis-tech/autogpt-package  --enclave autogpt '{"OPENAI_API_KEY": "test", "MEMORY_BACKEND": "local", "ALLOWLISTED_PLUGINS": "AutoGPTTwitter"}'
kurtosis service shell autogpt autogpt --exec "python -m autogpt"

This snippet here runs AutoGPT with the Email Plugin & Redis (spinning up an instance for you) and then gives you the terminal you are accustomed too;

kurtosis run github.com/kurtosis-tech/autogpt-package  --enclave autogpt '{"OPENAI_API_KEY": "test", "MEMORY_BACKEND": "redis", "ALLOWLISTED_PLUGINS": "AutoGPTEmailPlugin"}'
kurtosis service shell autogpt autogpt --exec "python -m autogpt"

This snippet here runs AutoGPT with the AutoGPTApiTools plugin & Weaviate (spinning up an instance for you) and then gives you the terminal you are accustomed too;

kurtosis run github.com/kurtosis-tech/autogpt-package  --enclave autogpt '{"OPENAI_API_KEY": "test", "MEMORY_BACKEND": "weaviate", "ALLOWLISTED_PLUGINS": "AutoGPTApiTools"}'
kurtosis service shell autogpt autogpt --exec "python -m autogpt"

Note you didn't have to download any plugin, Kurtosis did it for you. Note you didn't have to spin up Redis or Weaviate up yourself, Kurtosis did it for you! You can use remote PineCone as well or remote Milvus!

[h4ck3rk3y]

@ntindle As Victor said we are available as a native windows executable now too!

1. Can I add Kurtosis as a way of spinning up Auto-GPT to the README of the main repo significant-gravitas/Auto-GPT?
2. Should I add the test-plugin-installation.yml to the plugin template repo?

We're working on supporting a local(inside Docker) version of Milvus; we support all first party plugins but are adding support for more third party plugins; trying to get to parity with the README on this repository.

Also; I am more than happy transferring the autogpt-package to the Significant-Gravitas organization; or adding you as a maintainer on the current autogpt-package in case you want to approve PRs etc! Whatever works for you really.

[BaseInfinity]

Kurtosis sounds neat but seems like it's very new tech with little less than 50 stars on Github....I'm worried about its lifespan and installing something that very few people know about

I'm open minded but it's not my call, maybe @ntindle can tag someone from the main GH Repo who's familiar with their current CI implementation and see if this is something they want to adopt?

Not trying to be negative, it's just my reaction anytime I see new tech and choosing it as a foundation....been bit way too many times.

Also I think it's fair to point out that you guys work for Kurtosis so you guys have a bias lol, much different than two passionate developers pushing a tech you enjoy =)

[ntindle]

I'm pretty familiar with the CI of the main repo as I did the base of a large portion of it. Will look at this further later but we have a huge issue with install problems and a small team that will fix problems we have directly is almost always better than a big one who won't

[BaseInfinity]

I'm pretty familiar with the CI of the main repo as I did the base of a large portion of it. Will look at this further later but we have a huge issue with install problems and a small team that will fix problems we have directly is almost always better than a big one who won't

Very good point, love that thinking

Just wanted to play devil's advocate myself, I'm sure whatever we adopt will be fine for our needs, i've just gotten very paranoid over the years lol

To Kurtosis' credit, they are open source

[ntindle]

@h4ck3rk3y lets wait till 0.4.0 comes out as install process might change significantly

[h4ck3rk3y]

Love the discussion here @BaseInfinity! We made our code available on GitHub primarily to reduce anxiety that you have; we want our code to be available if we are long gone so our users can keep using us.

Sounds good @ntindle I'll be on the look for the new version 👍

[BaseInfinity]

Love the discussion here @BaseInfinity! We made our code available on GitHub primarily to reduce anxiety that you have; we want our code to be available if we are long gone so our users can keep using us.

Sounds good @ntindle I'll be on the look for the new version 👍

haha I'm glad this is an inviting discussion, it warms my heart when there's more than 1 person passionate about testing in a repo =)

I'll spend some time looking at the Kurtosis docs and maybe implement it in a simple project of mine to learn about it more.

Thanks for not getting offended lol. If anything you two have proved @ntindle point of being swift and more connected than a bigger box name would be so im on board (not that anyone needed me to be, im new here lol)

Looking forward to seeing this built out and helping out where i can

[h4ck3rk3y]

Thanks for the kind words @BaseInfinity! I hope you enjoy the tool; would love to learn how you felt using it.

[h4ck3rk3y]

@ntindle Hey! Just checking in - Do we know by when 0.4.0 will be out?!

An update from our side - last week someone on my team added arg validation to autogpt-package; so it validates that all plugin args are set before the package is run! Optionally you can skip validation if you want :-)

[ntindle]

No

[h4ck3rk3y]

Hey! We have added support for 0.4.0 here https://github.com/kurtosis-tech/autogpt-package

If someone for some reason wants to run against 0.3.1 they just have to

kurtosis run github.com/kurtosis-tech/autogpt-package@0.3.1

Plugins etc still work! Let us know if you have any next steps for us :)