Scene loads, but no vectors created and other issues

[frankcarey]

Following the steps on the README and getting pinecone, clerk, and convex all setup for the first time. Characters seem to move around, but things seem to be partially broken. I'm expecting to be able to see conversations happening between the characters at least, but I only see their intro text. There was a message about "sign in to view messages" and after I do that, the "sign in" message disappears, but I don't see any conversations still.

First thing I noticed was an authentication error in the JS, but I DO see my google account photo in the upper right hand corner, so not clear if that's actually an issue. I turned off all shields in brave and repeated the login process, but still get the same thing.

There was an issue with free pinecone I guess (noticed the closed PR related to the error). I un-commented the lines from convex/lib/pinecone.ts with the diff linked in the ticket, which resolved the immediate errors when running npm run dev it seemed, but I'm still getting errors here and there of a similar (cryptic) sort.

I also notice that pinecone has zero vectors, so I'm guessing this is related.

Looking at the embeddings in convex, they were added earlier it seems, but not recently..

Same thing for memories..

Any ideas?

[frankcarey]

Ah, Looks like I used the wrong name for my pinecone index name.. (Used the key name, not the index name). I'm not getting that error anymore, but still not seeing anything in pinecone... just these messages.

[frankcarey]

I'm trying npx convex run init:reset but getting an error when I try that.

$ npx convex run init:reset
npm WARN config global `--global`, `--local` are deprecated. Use `--location=global` instead.
- Preparing Convex functions...

√ 22:24:56 Convex functions ready! (4.43s)
✖ Failed to run function "init:reset":
Error: Uncaught Error: Error: Not implemented: get body for Response: Try using .text(), .json(), blob(), or .arrayBuffer() instead. Consider calling an action defined in Node.js instead (https://docs.convex.dev/functions/actions).

    at async handler (../convex/init.ts:103:2)

[frankcarey]

OK - I misread this commit. Looks like these lines were meant to be commented out and my mistake with the pinecone index name just happened to cause a similar error. I undid those changes and now this succeeds.

npx convex run init:reset --no-push

.. and I'm seeing 17 vectors now in pinecone (note to others, the UI can take a second to load and show the correct amount on pinecone. Until then it shows 0)

For some reason after this, it took almost 10 minutes more before it started showing conversations, but it is doing that now finally. I see pinecone upserts and queries finally happening. Also convex logs are showing non-zero numbers. Hope this helps someone else if they need to debug as well.

[ianmacartney]

Sorry this was so much trouble! Pinecone used to support namespaces on starter plans, which is what I was developing with, but it seems they've made those paid-only for newer indexes. the init:reset will add a new world and kick off the pinecone inserts. As for the 10 minute delay: my guess is something failed hard when processing a batch of agents, and they got stuck in a "thinking" state. Each agent will only be processed in one batch at a time, but there's a cron job that runs that checks if any agent has been thinking for >10min and resets them. There is a try/except in the action that processes the batch and tries to do cleanup by just having them walk away, but if there was another exception when trying to walk, then it could get stuck. If you find a root cause it'd be great to know what to fix.

[frankcarey]

Sorry this was so much trouble!

No worries! Thanks to the team for creating this and I certainly didn't help matters by getting the Pinecone setup wrong.

A couple suggestions to help folks like me contribute and debug, but feel free to close this ticket as well:

• Having a high-level flow/component diagram of what things are doing would be helpful. Obviously we can read the code, but I notice there are even some principals you posted on the discord already that could be codified into an FAQ or something. I get things are moving fast and this might be premature, but it would certainly help grok the architecture, especially the flow of a batch of agents, etc.
• In lieu of that, the readme has one-word summaries of the third party services being used, but expanding on that a little bit would also help to understand when things go wrong where to focus on.
• Finally a related topic is error reporting. Seems the serverless functions on Convex either don't respond with a helpful error messages (in the Convex logs) or the functions themselves are not catching errors from like Pinecone and passing them through.

Thanks again for your work here, excited to see projects like this advance and I'll look for more ways to contribute!