search

microsoft / FLAML

A fast library for AutoML and tuning. Join our Discord: https://discord.gg/Cppx2vSPVP.
https://microsoft.github.io/FLAML/
MIT License
3.89k stars 506 forks source link

Failure cases for autogen agents #1111

Open sonichi opened 1 year ago

sonichi commented 1 year ago
  1. extract all telescopes that are up for auction on https://astromart.com/ (using a wrong url; using wrong assumption of the html structure)
  2. Find $a + b + c$, given that $x+y \neq -1$ and \begin{align} ax + by + c & = x + 7,\ a + bx + cy & = 2x + 6y,\ ay + b + cx & = 4x + y. \end{align}. (using numpy; need the hint: use sympy)
  3. What date is today? Which big tech stock has the largest year-to-date gain this year? How much is the gain? (using outdated ticker FB; need the hint: FB -> META)
qingyun-wu commented 1 year ago

Is it possible to summarize the observed reasons for the failures?

sonichi commented 1 year ago

Is it possible to summarize the observed reasons for the failures?

One can run these examples and observe the chat. Then, one can try adding hints to make it work. It'll be nice to reflect from that and make it smarter in future. Would you like to try this with the learning agent?