Open jlewi opened 1 year ago
Here's what I suspect is going on.
It looks like withDefaultExamples instantiates the examples with PreviousContext via the use of reflection.
This is then passed to WithExamples to return an option function which binds to the examples variable. It looks like that by the time the function returned by WithExamples is actually invoked the actual PreviousContext data might have been freed. So I suspect somehow the use of reflection is allocating memory in a way that causes it to be prematurely GC'd.
Thanks for filing this.
After fixing #1 with f8640d38b98ac98c79a2b3e4ff54764702489f0c I'm able to run go test ./...
with success.
Would you mind pulling and trying again?
Seems like its still failing for me (output below). I'm on darwin/arm and using go 1.21. Are you similar?
go test ./...
? github.com/google/go-react/examples/app-editor [no test files]
? github.com/google/go-react/examples/prompt [no test files]
? github.com/google/go-react/pkg/llms/testing [no test files]
? github.com/google/go-react/pkg/llms/vertex [no test files]
? github.com/google/go-react/pkg/parsers/testing [no test files]
? github.com/google/go-react/pkg/predictors/testing [no test files]
? github.com/google/go-react/pkg/prompters/testing [no test files]
? github.com/google/go-react/pkg/tools [no test files]
? github.com/google/go-react/pkg/tools/userinput [no test files]
--- FAIL: TestDefaultPrompt (0.00s)
--- FAIL: TestDefaultPrompt/default_examples (0.00s)
default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
--- FAIL: TestDefaultPrompt/default_preamble (0.00s)
default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
--- FAIL: TestDefaultPrompt/custom_rules (0.00s)
default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
--- FAIL: TestDefaultPrompt/default_rules (0.00s)
default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
--- FAIL: TestDefaultPrompt/custom_preamble (0.00s)
default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
FAIL
FAIL github.com/google/go-react/pkg/agents 0.239s
ok github.com/google/go-react/pkg/chains (cached)
ok github.com/google/go-react/pkg/chains/confirmation (cached)
ok github.com/google/go-react/pkg/llms (cached)
ok github.com/google/go-react/pkg/parsers (cached)
ok github.com/google/go-react/pkg/predictors (cached)
ok github.com/google/go-react/pkg/prompters (cached)
FAIL
Here's my go version
go version
go version go1.21.0 darwin/arm64
Here's the commit I'm testing
git rev-parse HEAD
f8640d38b98ac98c79a2b3e4ff54764702489f0c
I was able to reproduce this by upgrading to go 1.21 (from 1.20)
Hydrating the default prompt chokes on hydrating the default examples.
This can be reproduced by running the tests
Note the tests fail to compile on main because of #1 ; I have fixed that on my fork https://github.com/jlewi/go-react/commit/53beeb7fdaacbe78aeb32e111cfd6705f06436e3
The problematic part of the template appears to be this line https://github.com/google/go-react/blob/643efbe8af3c57eb15d6c05ab1515c8eb170bc3d/pkg/agents/default_prompt.go#L58
Which is invoking ToJSON on the PreviousContext in the PromptDataExample I think the ThoughtIteration data populated in one or more examples might be corrupt.
When I inspect the data in my debugger i see that for the second example (Examples[1]) the previous context appears to be corrupt. For example, my debugger shows that observation is "unreadable could not resolve interface type".
I suspect this has something to do with the use of reflection here to instantiate things https://github.com/google/go-react/blob/643efbe8af3c57eb15d6c05ab1515c8eb170bc3d/pkg/agents/default_prompt.go#L95