search

google / go-react

Apache License 2.0
31 stars 3 forks source link

Hydrating default prompt chokes on the default examples #2

Open jlewi opened 1 year ago

jlewi commented 1 year ago

Hydrating the default prompt chokes on hydrating the default examples.

This can be reproduced by running the tests

go test ./...
?       github.com/google/go-react/examples/app-editor  [no test files]
?       github.com/google/go-react/examples/prompt  [no test files]
?       github.com/google/go-react/pkg/llms/testing [no test files]
?       github.com/google/go-react/pkg/llms/vertex  [no test files]
?       github.com/google/go-react/pkg/parsers/testing  [no test files]
?       github.com/google/go-react/pkg/predictors/testing   [no test files]
?       github.com/google/go-react/pkg/prompters/testing    [no test files]
?       github.com/google/go-react/pkg/tools    [no test files]
?       github.com/google/go-react/pkg/tools/userinput  [no test files]
--- FAIL: TestDefaultPrompt (0.00s)
    --- FAIL: TestDefaultPrompt/default_examples (0.00s)
        default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
    --- FAIL: TestDefaultPrompt/default_rules (0.00s)
        default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
    --- FAIL: TestDefaultPrompt/default_preamble (0.00s)
        default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
    --- FAIL: TestDefaultPrompt/custom_rules (0.00s)
        default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
    --- FAIL: TestDefaultPrompt/custom_preamble (0.00s)
        default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
FAIL
FAIL    github.com/google/go-react/pkg/agents   0.112s
ok      github.com/google/go-react/pkg/chains   (cached)
ok      github.com/google/go-react/pkg/chains/confirmation  (cached)
ok      github.com/google/go-react/pkg/llms (cached)
ok      github.com/google/go-react/pkg/parsers  (cached)
ok      github.com/google/go-react/pkg/predictors   (cached)
ok      github.com/google/go-react/pkg/prompters    (cached)
FAIL

Note the tests fail to compile on main because of #1 ; I have fixed that on my fork https://github.com/jlewi/go-react/commit/53beeb7fdaacbe78aeb32e111cfd6705f06436e3

The problematic part of the template appears to be this line https://github.com/google/go-react/blob/643efbe8af3c57eb15d6c05ab1515c8eb170bc3d/pkg/agents/default_prompt.go#L58

Which is invoking ToJSON on the PreviousContext in the PromptDataExample I think the ThoughtIteration data populated in one or more examples might be corrupt.

When I inspect the data in my debugger i see that for the second example (Examples[1]) the previous context appears to be corrupt. For example, my debugger shows that observation is "unreadable could not resolve interface type".

I suspect this has something to do with the use of reflection here to instantiate things https://github.com/google/go-react/blob/643efbe8af3c57eb15d6c05ab1515c8eb170bc3d/pkg/agents/default_prompt.go#L95

jlewi commented 1 year ago

Here's what I suspect is going on.

It looks like withDefaultExamples instantiates the examples with PreviousContext via the use of reflection.

This is then passed to WithExamples to return an option function which binds to the examples variable. It looks like that by the time the function returned by WithExamples is actually invoked the actual PreviousContext data might have been freed. So I suspect somehow the use of reflection is allocating memory in a way that causes it to be prematurely GC'd.

poy commented 1 year ago

Thanks for filing this.

After fixing #1 with f8640d38b98ac98c79a2b3e4ff54764702489f0c I'm able to run go test ./... with success.

Would you mind pulling and trying again?

jlewi commented 1 year ago

Seems like its still failing for me (output below). I'm on darwin/arm and using go 1.21. Are you similar?

go test ./...
?       github.com/google/go-react/examples/app-editor  [no test files]
?       github.com/google/go-react/examples/prompt  [no test files]
?       github.com/google/go-react/pkg/llms/testing [no test files]
?       github.com/google/go-react/pkg/llms/vertex  [no test files]
?       github.com/google/go-react/pkg/parsers/testing  [no test files]
?       github.com/google/go-react/pkg/predictors/testing   [no test files]
?       github.com/google/go-react/pkg/prompters/testing    [no test files]
?       github.com/google/go-react/pkg/tools    [no test files]
?       github.com/google/go-react/pkg/tools/userinput  [no test files]
--- FAIL: TestDefaultPrompt (0.00s)
    --- FAIL: TestDefaultPrompt/default_examples (0.00s)
        default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
    --- FAIL: TestDefaultPrompt/default_preamble (0.00s)
        default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
    --- FAIL: TestDefaultPrompt/custom_rules (0.00s)
        default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
    --- FAIL: TestDefaultPrompt/default_rules (0.00s)
        default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
    --- FAIL: TestDefaultPrompt/custom_preamble (0.00s)
        default_prompt_test.go:116: failed to hydrate prompt: template: prompt:32:38: executing "prompt" at <ToJSON .>: error calling ToJSON: runtime error: invalid memory address or nil pointer dereference
FAIL
FAIL    github.com/google/go-react/pkg/agents   0.239s
ok      github.com/google/go-react/pkg/chains   (cached)
ok      github.com/google/go-react/pkg/chains/confirmation  (cached)
ok      github.com/google/go-react/pkg/llms (cached)
ok      github.com/google/go-react/pkg/parsers  (cached)
ok      github.com/google/go-react/pkg/predictors   (cached)
ok      github.com/google/go-react/pkg/prompters    (cached)
FAIL

Here's my go version

go version
go version go1.21.0 darwin/arm64

Here's the commit I'm testing

git rev-parse HEAD 
f8640d38b98ac98c79a2b3e4ff54764702489f0c
poy commented 1 year ago

I was able to reproduce this by upgrading to go 1.21 (from 1.20)