refactor: clear separation of public and internal packages

[lifeizhou-ap]

Glossary

• Library: A distributed Python package (e.g., goose-ai).
• Package: A package within a Python codebase.

Current State

goose-ai serves as:

• A command-line application.
• A plugin host that discovers and loads plugins.
• A library that plugins depend on, providing base classes for creating toolkits, profiles, and notifiers.

Plugins depend on goose-ai during compile time to extend base classes such as Toolkit. However, at runtime, goose-ai dynamically loads the plugin’s functions or classes, creating an implicit dependency on the plugin.

Problem

Currently, goose-ai does not clearly distinguish between publicly accessible classes/functions and those intended for internal use. This lack of separation can lead to unintended use by plugin developers, potentially causing circular dependency issues.

Example:

• Module A in goose-ai has two functions: load_plugin and goose_ai_util.
• Module B in a plugin has a function render_toolkit, which references goose_ai_util.

When load_plugin in Module A calls render_toolkit to load an additional toolkit, render_toolkit in turn calls goose_ai_util in Module A. This can trigger a circular dependency problem.

Solution

To resolve this issue, restructure the goose-ai codebase to clearly delineate:

• Packages that are publicly accessible to library users (e.g., plugins).
• Packages that are used internally to implement goose-ai as a command-line application and plugin host.

Benefits

Maintainability

Encapsulating internal packages and implementation details makes it easier to refactor or modify goose-ai without affecting the users of the library. This ensures backward compatibility and reduces the risk of breaking changes.

Clear Interface to Prevent Misuse

Plugins will have a well-defined interface for accessing goose-ai functions and classes. This reduces the likelihood of plugin developers inadvertently using internal functions or classes, helping to prevent circular dependency problems.

How

In the src/goose directory, create three distinct packages:

1. _internal:

   • This package is for internal use only and is responsible for:
     • Building the command-line application.
     • Discovering and loading plugins.

2. pluginbase: (open to a better name)

   • This package is publicly accessible and contains:
     • Classes for plugins to extend.
     • Utilities to be shared with plugins.

3. config:

   • This package is publicly accessible and contains configuration constants, such as:
     • Session file location and file extensions.
     • Profile configuration file location.

_internal can reference the pluginbase and config. But pluginbase and config cannot reference _internal, we may set some check to prevent this.

By organizing the codebase in this manner, goose-ai can offer a clear and stable API to its users while safeguarding against issues arising from the unintentional misuse of internal components.

Folder structures

[baxen]

Hey @lifeizhou-ap! Absolutely agree with your categorization of the three things this package is providing, and we should clarify those throughout the code.

However I do consider the majority of the functionality to be "public", and so would avoid the refactor of pushing so much of the content into an _internal package. I'd instead at this point go function by function, or possibly in the future consider putting the leading underscore on a few selected files.

Some examples of why I think these are more public than they might seem:

• build_exchange is intended to be used in other contexts such as developing an editor extension
• Session (from the CLI) is exposed to make it possible to implement extensions to the CLI, such as a tutorial mode
• Developer toolkit could be inherited from to add new functionality in downstream plugins

So i'd suggest we revisit this to highlight some of the private methods, especially any place you're running into circular imports!