Issue #401: Cleanup for React packages for publishing

jahabeebs commented 1 month ago

@LukeLalor @dbrewster

closes #401

Description

Adds tsup to bundle eidolon-components and eidolon-client
Bumps turbo to 2.0
eslint-config was not working at all, I had to follow the instructions in the turbo docs to get this working and now it properly lints every package.
From eidolon-ui2 I did have to remove the "extends": "@repo/typescript-config/react-library.json" because it was causing issues and I adapted the tsconfig from the starter repo in the next.js examples repo (https://github.com/vercel/vercel/blob/main/examples/nextjs/tsconfig.json)
A ton of tsconfig and tsconfig lint changes for the published packages and eidolon-ui2. I can try to trim them down if you think there are too many
Adds outputs to turbo.json in webui and rename from pipeline to tasks ("pipeline" is not part of the turbo schema)
Separated the server and client components of eidolon-components and creates separate paths for them because they were causing server/client violations in eidolon-ui2
Added changeset workflow file (to .github/workflows) and cli dependency (to webui), not sure the best way to test this

I need help to:

Install the changeset bot to the eidolon repo: https://github.com/apps/changeset-bot see docs here: https://github.com/changesets/changesets/blob/main/docs/automating-changesets.md
Best way to test the full changeset workflow? I don't want to try to publish and steal the npm namespaces

jahabeebs commented 3 weeks ago

@LukeLalor @dbrewster I think it's just about finished, I've updated the description with as much detail as I could about my changes (unfortunately I couldn't find a way to change any less files) but we can chat about it on Thursday

jahabeebs commented 2 weeks ago

resolved merge conflicts

jahabeebs commented 1 week ago

Remaining issues:

Test webui:

[Polly] [adapter:fetch] Using the fetch adapter in Node has been deprecated. Please use the node-http adapter instead. PASS test/server/process-server-handler.test.ts

Playwright tests: "fetch failed" when front end tries to access server--no api requests succeeding even though the server is healthy

logs:

2024-06-27 01:50:31 INFO:     Started server process [1]
2024-06-27 01:50:31 INFO:     Waiting for application startup.
2024-06-27 01:50:31 /usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:394: UserWarning: <built-in function callable> is not a Python type (it may be an instance of an object), Pydantic will allow any object with no validation since we cannot even enforce that the input is an instance of the given type. To get rid of this error wrap the type with `pydantic.SkipValidation`.
2024-06-27 01:50:31   warn(
2024-06-27 01:50:33 --- Logging error ---
2024-06-27 01:50:33 Traceback (most recent call last):
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/kernel.py", line 71, in get_resources
2024-06-27 01:50:33     ret[k] = tu[0].promote(kind)
2024-06-27 01:50:33              ^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/resources/resources_base.py", line 32, in promote
2024-06-27 01:50:33     return clazz.model_validate(self.model_dump())
2024-06-27 01:50:33            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/pydantic/main.py", line 551, in model_validate
2024-06-27 01:50:33     return cls.__pydantic_validator__.validate_python(
2024-06-27 01:50:33            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/reference_model.py", line 160, in _validate
2024-06-27 01:50:33 INFO - Overriding builtin resource 'Reference.ToolCallLLMWrapper'
2024-06-27 01:50:33 INFO - Building machine 'DEFAULT'
2024-06-27 01:50:33     spec.model_validate(self.model_extra or {})
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/pydantic/main.py", line 551, in model_validate
2024-06-27 01:50:33     return cls.__pydantic_validator__.validate_python(
2024-06-27 01:50:33            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/builtins/logic_units/web_search.py", line 75, in <lambda>
2024-06-27 01:50:33     default_factory=lambda: os.environ["CSE_ID"],
2024-06-27 01:50:33                             ~~~~~~~~~~^^^^^^^^^^
2024-06-27 01:50:33   File "<frozen os>", line 679, in __getitem__
2024-06-27 01:50:33 KeyError: 'CSE_ID'
2024-06-27 01:50:33 
2024-06-27 01:50:33 During handling of the above exception, another exception occurred:
2024-06-27 01:50:33 
2024-06-27 01:50:33 Traceback (most recent call last):
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/logging/__init__.py", line 1110, in emit
2024-06-27 01:50:33     msg = self.format(record)
2024-06-27 01:50:33           ^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/logging/__init__.py", line 953, in format
2024-06-27 01:50:33     return fmt.format(record)
2024-06-27 01:50:33            ^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_client/util/logger.py", line 15, in format
2024-06-27 01:50:33     return super().format(record)
2024-06-27 01:50:33            ^^^^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/logging/__init__.py", line 687, in format
2024-06-27 01:50:33     record.message = record.getMessage()
2024-06-27 01:50:33                      ^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/logging/__init__.py", line 377, in getMessage
2024-06-27 01:50:33     msg = msg % self.args
2024-06-27 01:50:33           ~~~~^~~~~~~~~~~
2024-06-27 01:50:33 TypeError: not all arguments converted during string formatting
2024-06-27 01:50:33 Call stack:
2024-06-27 01:50:33   File "/usr/local/bin/eidolon-server", line 8, in <module>
2024-06-27 01:50:33     sys.exit(main())
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/bin/agent_http_server.py", line 242, in main
2024-06-27 01:50:33     uvicorn.run(
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 577, in run
2024-06-27 01:50:33     server.run()
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 65, in run
2024-06-27 01:50:33     return asyncio.run(self.serve(sockets=sockets))
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
2024-06-27 01:50:33     return runner.run(main)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
2024-06-27 01:50:33     return self._loop.run_until_complete(task)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/uvicorn/lifespan/on.py", line 86, in main
2024-06-27 01:50:33     await app(scope, self.receive, self.send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
2024-06-27 01:50:33     return await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
2024-06-27 01:50:33     await super().__call__(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
2024-06-27 01:50:33     await self.middleware_stack(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 151, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 571, in __call__
2024-06-27 01:50:33     return await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 103, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 103, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 77, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 103, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 103, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 51, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
2024-06-27 01:50:33     await self.middleware_stack(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 765, in app
2024-06-27 01:50:33     await self.lifespan(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 732, in lifespan
2024-06-27 01:50:33     async with self.lifespan_context(app) as maybe_state:
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
2024-06-27 01:50:33     return await anext(self.gen)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/bin/agent_http_server.py", line 173, in start_os
2024-06-27 01:50:33     machine: AgentMachine = machine_spec.instantiate()
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/reference_model.py", line 195, in instantiate
2024-06-27 01:50:33     return self._get_reference_class()(*args, **kwargs)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/agent_machine.py", line 60, in __init__
2024-06-27 01:50:33     for name, r in AgentOSKernel.get_resources(AgentResource).items():
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/kernel.py", line 73, in get_resources
2024-06-27 01:50:33     register_resource_promote_error(k, tu[0].kind, kind, e)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/resource_load_error_handler.py", line 31, in register_resource_promote_error
2024-06-27 01:50:33     logger.exception(f"Error promoting resource {name} of kind {resourceKind} to {kind}", e)
2024-06-27 01:50:33 Message: "Error promoting resource search_agent of kind Agent to <class 'eidolon_ai_sdk.system.resources.agent_resource.AgentResource'>"
2024-06-27 01:50:33 Arguments: (KeyError('CSE_ID'),)
2024-06-27 01:50:33 INFO - Starting agent 'conversational_agent'
2024-06-27 01:50:33 INFO - Starting agent 'scraping_agent'
2024-06-27 01:50:33 INFO - Starting agent 'research_agent'
2024-06-27 01:50:33 INFO - Starting agent 'speech_agent'
2024-06-27 01:50:33 INFO - Server Started in 2.34s

dbrewster commented 1 week ago

That stack trace is normal as one of the services, the web research service, can't be started, but it shouldn't affect anything. It looks like the server never saw the request from the UI. Is the environment variable / config file correct for the UI? I think you ran into this when you were building the docker compose command line

jahabeebs commented 1 week ago

That stack trace is normal as one of the services, the web research service, can't be started, but it shouldn't affect anything. It looks like the server never saw the request from the UI. Is the environment variable / config file correct for the UI? I think you ran into this when you were building the docker compose command line

That stack trace is normal as one of the services, the web research service, can't be started, but it shouldn't affect anything. It looks like the server never saw the request from the UI. Is the environment variable / config file correct for the UI? I think you ran into this when you were building the docker compose command line

I fixed the webui component test that was failing (had to change an import)

The only other issue is the front end/back end communication thing:

When I use docker-compose.yml locally the front end is able to make calls to the back end and can fetch the chatbot processes, but as soon as I try to start a new chat or access a new chat there's a front-end error that I can't see because of minification (the server stays healthy, it's just a client-side error). I tried setting NODE_ENV=development everywhere I could to be able to see the actual error but I haven't been able to get past that minification for some reason

When I run the front end with the pnpm run dev command (rather than build) + a docker back end the chatbot works perfectly and the front end doesn't crash

So for some reason the built version of the front end is crashing but not the development version...even in React strict mode

jahabeebs commented 1 week ago

@dbrewster Unfortunately I still haven't figured out the source of this minified JS error that I get after building everything. Even when I pass in NODE_ENV=development to docker-compose and the webui dockerfile it still minifies on build. Here's a video of what happens (can trigger the crash also by pressing add chat). When running the front end in dev mode rather than built it works just fine.

Minified react error 130: https://react.dev/errors/130

A possible cause of the issues is me splitting up the eidolon-components package into client and server dists. Technically we can't have class components as server components (like AgentHandler, FileHandler, FilesHandler, see here for details: https://github.com/vercel/next.js/issues/47232 and here: https://github.com/palantir/blueprint/issues/6263), however I tried 1) refactoring them as functional components (ended up with same minified error) and 2) making them client components (eidolon-ui2 wouldn't even build, get an error "TypeError: o.TX is not a constructor" when building the routes

https://github.com/eidolon-ai/eidolon/assets/47253537/c1d3dab9-890a-4b2f-95c1-87604aab708a

eidolon-ai / eidolon

Issue #401: Cleanup for React packages for publishing #457

Description

I need help to: