Issue #386: Create Docker Compose - front & back end

[jahabeebs]

relates to #386

This PR removes the local file references from the examples Dockerfile (it still runs successfully on its own though) and creates a draft Docker Compose that spins up both the new eidolon-ui2 front end and eidolon-server. I have some questions on how to move forward with it

Instructions to run

1) Add .env to root of repo w/OPENAI_API_KEY set 2) run docker-compose up from the eidolon root directory

Questions

Both the front end and back end are building and running fine, but when I interact with the running front end it seems to not be communicating with the back end correctly. Some server calls are successful, like this one when interacting with the chatbot:

eidolon-server_1  | INFO - Request: GET http://eidolon-server:8080/processes?skip=0&limit=100
eidolon-server_1  | INFO - Request: GET http://eidolon-server:8080/openapi.json
eidolon-server_1  | INFO - Response: 200

but others are not, like this chatbot one:

eidolon-server_1  | INFO - Request: POST http://eidolon-server:8080/processes
eidolon-server_1  | INFO - Agent conversational_agent does not exist
eidolon-server_1  | INFO - Response: 404

Is my problem that I should be running the usage-server as well as the eidolon-server? Should I not be running eidolon-server from the examples Dockerfile? Do I need to add each example individually to the Dockerfile to get each of them to work? In this case I'll probably just start with one and then start adding end-to-end tests.

I also noticed that the examples Dockerfile has command CMD ["resources"] but when I added it to the Docker compose like this it always failed to recognize resources and the server crashed immediately, so I removed it. This was the failing part of the docker-compose.yml that I removed:

  eidolon-server:
    build:
      context: ./examples
      dockerfile: Dockerfile
    environment:
      - PYTHONUNBUFFERED=1
      - OPENAI_API_KEY=${OPENAI_API_KEY}
    expose:
      - "8080"
    ports:
      - "8080:8080"
    volumes:
      - ./examples/resources:/app/resources
    command: ["eidolon-server", "resources"]

[jahabeebs]

I can't add you as reviewers so I'll mention you here 😁 @dbrewster @LukeLalor

[jahabeebs]

I can't add you as reviewers so I'll mention you here 😁 @dbrewster @LukeLalor

Update: I talked with Dave about getting the chatbot example running first so I made changes to the docker-compose.yml to use that config for eidolon-server. Now I'm not getting a 404 error when trying to create a new chat but I am getting a 401...specifically this:

eidolon-eidolon-server-1  | INFO - Request: POST http://eidolon-server:8080/processes
eidolon-eidolon-server-1  | INFO - Auth Denied: Authorization header missing
eidolon-eidolon-server-1  | INFO - Response: 401

When I log in I just use any email it redirects me back to the homepage and shows I'm signed in, and I get a 200 response when we get the session data, so I don't know why it doesn't want to run processes. Should I be running something in the docker-compose.yml that isn't already running? I noticed there are some mentions of "processes" in the sdk Dockerfile and eidolon-server is the entry point, should I be incorporating that Docker file into my docker-compose maybe?

[LukeLalor]

The problem is that the dockerfile built enables authorization when building the docker images.

COPY demo_resources/ resources/

we should probably build the images without that line for now and think about how to configure this dynamically better

[jahabeebs]

The problem is that the dockerfile built enables authorization when building the docker images.

COPY demo_resources/ resources/

we should probably build the images without that line for now and think about how to configure this dynamically better

@LukeLalor Thanks for the advice, I excluded the auth yaml from the server Dockerfile and it wasn't giving me the 401s anymore but it was giving me errors because I didn't have a MongoDB configuration. I configured MongoDB in the docker-compose and I was able to get it working. I also added a .env.example so that the docker-compose.yml example will work when running locally

My only worry is that I may have broken the Readme.md instructions for running the conversation chatbot on its own? When I start a virtual env and follow the instructions it doesn't like me input and gives me a 422. Maybe it's unrelated though. When I change the curl to be in JSON like this I don't get a 422 anymore but the input is blank:

JSON request (not 422 but blank response)

curl -X 'POST' "http://localhost:8080/processes/$PROCESS_ID/agent/conversational_agent/actions/converse" -H "Content-Type: application/json" -d '{"input":"What kind of tools can I build with LLM agents?"}'

Unsuccessful example request in chatbot Readme

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
eidolon	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 29, 2024 5:40am

[jahabeebs]

@LukeLalor @dbrewster

I made the final edits to our chatbot e2e test and the GitHub Action is running successfully. Here's the latest test run on my eidolon fork:

https://github.com/jahabeebs/eidolon/actions/runs/9280656466/job/25535456261

This is a video of what it looks like locally when you run pnpm run playwright:debug

https://github.com/eidolon-ai/eidolon/assets/47253537/951c224a-4160-4541-ae0c-b05914b3af3a

Some configuration things you may or may not want me to change: 1) I have the e2e.yml running on pushes to main and PRs, not sure if this is too much. The e2e test is very heavy and takes around 5-6 minutes to run. This configuration is the same as the test.yml configuration

on:
  push:
    branches: [main]
  pull_request:

2) I've configured playwright to upload a screenshot if a failure occurs to webui/apps/eidolon-ui2/tests/screenshots. This way you can see if it's a real failure or if the chatbot is just hallucinating. I can remove this logic if you'd like

In order for e2e.yml to run successfully in the eidolon repo the following secrets need to be set in GitHub:

      AUTH_TRUST_HOST: ${{ secrets.AUTH_TRUST_HOST }}
      EIDOLON_SERVER: ${{ secrets.EIDOLON_SERVER }}
      NEXTAUTH_SECRET: ${{ secrets.NEXTAUTH_SECRET }}
      NEXT_PUBLIC_API_URL: ${{ secrets.NEXT_PUBLIC_API_URL }}
      NEXT_PUBLIC_DEBUG: ${{ secrets.NEXT_PUBLIC_DEBUG }}
      NEXT_PUBLIC_LOG_LEVEL: ${{ secrets.NEXT_PUBLIC_LOG_LEVEL }}
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

[jahabeebs]

@LukeLalor I believe I've addressed all the changes, here's the successful actions run with the changes I made based on your comments. Thanks!

https://github.com/jahabeebs/eidolon/actions/runs/9312680168/job/25633823756?pr=1

[LukeLalor]

👀

[jahabeebs]

I say ship it. I created a new branch integration_branch so we can make sure the tests pass over there.

@jahabeebs, let me know if you are unable to and I can force merge it.

@LukeLalor I tried but don't have permissions, could you force merge?

[LukeLalor]

@jahabeebs done, but the tests are failing over on the integration branch