Incorrect Block Parsing in Baml Response from LLM

ekinsdrow commented 1 month ago

Common description

I have encountered an issue where Baml parses the response from the LLM incorrectly. Specifically, it seems that one block type is being incorrectly converted into another block type in the response.

Issue

The column_list block from the LLM response is incorrectly converted to a breadcrumb block in the Baml response. The configuration shows that these blocks should remain distinct, but for some reason, Baml is changing one type of block to another.

Expected Behavior

Baml should parse the LLM response correctly, preserving the block types and structure as provided by the LLM response. The column_list block should not be converted into a breadcrumb.

Part of my configuration

class Page{
    object string @description("Always page")
    icon Icon @description("Icon of page")
    children (Bookmark | Breadcrumb | BulletedListItem | Callout | Code | ColumnList | Divider | Embed | Equation | File | Heading1 | Heading2 | Heading3 | ImageFile | NumberedListItem | Paragraph | PDF | Quote | Table | TableOfContents | ToDo | Toggle | Video)[] @description("Some blocks of the page")
}

// ------------ Breadcrumb ----------------
class Breadcrumb {
  type string @description("Always breadcrumb")
  breadcrumb map<string, string> @description("Always empty map")
}

/// ------------ Column List ----------------
class ColumnList {
  type string @description("Always column_list")
  column_list ColumnListBody @description("Column list block with columns")
}

class ColumnListBody {
  children Column[] @description("The columns in the column list. Max length is 5")
}

class Column {
  type string @description("Always column")
  column ColumnBody @description("Always empty map for columns")
}

class ColumnBody {
  children (Bookmark | Breadcrumb | BulletedListItem | Callout | Code | Divider | Embed | Equation | File | Heading1 | Heading2 | Heading3 | ImageFile | NumberedListItem | Paragraph | PDF | Quote | Table | TableOfContents | ToDo | Toggle | Video)[] @description("Content of the column. Can contain any block type. Min length is 1")
}

/// ------------ Other blocks ----------------

LLM response

{
  "object": "page",
  "icon": {
    "emoji": "📚"
  },
  "children": [
    {
      "type": "column_list",
      "column_list": {
        "children": [
          {
            "type": "column",
            "column": {
              "children": [
                {
                  "type": "heading_3",
                  "heading_3": {
                    "rich_text": [
                      {
                        "type": "text",
                        "text": {
                          "content": "The Lord of the Rings"
                        }
                      }
                    ],
                    "is_toggleable": false
                  }
                },
                {
                  "type": "paragraph",
                  "paragraph": {
                    "rich_text": [
                      {
                        "type": "text",
                        "text": {
                          "content": "J.R.R. Tolkien"
                        }
                      }
                    ]
                  }
                },
                {
                  "type": "to_do",
                  "to_do": {
                    "rich_text": [
                      {
                        "type": "text",
                        "text": {
                          "content": "Read again"
                        }
                      }
                    ],
                    "checked": false
                  }
                }
              ]
            }
          }
        ]
      }
    }
  ]
}

Baml parsing reponse

{
  "object": "page",
  "icon": {
    "emoji": "📚"
  },
  "children": [
    {
      "type": "column_list",
      "breadcrumb": {}
    }
  ]
}

sxlijin commented 1 month ago

Yikes! Thanks for letting us know, will look into this immediately.

As an aside: we have work planned to add literals into the language (which I see you're sorely missing here), but as a workaround, you might be able to use singly-valued enums for your types right now, ex:

enum BreadcrumbType {
  Breadcrumb
}

class Breadcrumb {
  type BreadcrumbType
  ...
}

(This unfortunately collides with the other issue you filed about the generated code for enum constants.)

sxlijin commented 1 month ago

Can you share the type definitions for Heading3, Paragraph, RichText, and ToDo?

After a bit of digging, we've found a number of scenarios that parse correctly and some that do not parse, but it gets hard to say how they should parse without digging into specifics, and I would prefer that whatever fix we ship for this be based on your actual scenario, not our own synthetic tests.

My best guess right now is that there's a parse error in column_list.children that's rolling up into the union parsing decision for the type of Page.children[0], because per the testing I'm doing, that's the minimum required to trigger this, but I'd like to be sure that that's the case before doing it.

ekinsdrow commented 1 month ago

Hey! Thanks for your answer. I will take a look in singly-valued enums, thanks!

Here is defenitions of types which you asked -

/// ------------ Heading 3 ----------------
class Heading3 {
  type string @description("Always heading_3")
  heading_3 HeadingBody @description("Heading 3 block with rich text")
}

class HeadingBody {
  rich_text RichText[] @description("The rich text content of the heading")
  is_toggleable bool @description("Whether the heading is toggleable")
}

/// ------------ Paragraph ----------------
class Paragraph {
  type string @description("Always paragraph")
  paragraph ParagraphBody @description("Paragraph block with rich text")
}

class ParagraphBody {
  rich_text RichText[] @description("The rich text displayed in the paragraph block")
  children (Bookmark)[] @description("Optional nested child blocks for paragraph")
}

/// ------------ RichText ----------------
class RichText {
  type string @description("Always text")
  text RichTextText @description("The content of the rich text element")
}

/// ------------ To Do ----------------
class ToDo {
  type string @description("Always to_do")
  to_do ToDoBody @description("To-do block with checkbox and content")
}

class ToDoBody {
  rich_text RichText[] @description("The text content of the to-do item")
  checked bool? @description("Whether the to-do item is checked")
  children (Paragraph | Callout | ImageFile)[] @description("Optional nested child blocks for to-do")
}

ekinsdrow commented 1 month ago

Now, it's start working very strange, now almost all my children became breadcrumb -

{
  "object": "page",
  "icon": {
    "emoji": "📚"
  },
  "children": [
    {
      "type": "heading_1",
      "breadcrumb": {}
    },
    {
      "type": "paragraph",
      "breadcrumb": {}
    },
    {
      "type": "divider",
      "divider": {}
    },
    {
      "type": "heading_2",
      "breadcrumb": {}
    },
    {
      "type": "table",
      "breadcrumb": {}
    },
    {
      "type": "divider",
      "divider": {}
    },
    {
      "type": "heading_2",
      "breadcrumb": {}
    },
    {
      "type": "bulleted_list",
      "breadcrumb": {}
    },
    {
      "type": "divider",
      "divider": {}
    },
    {
      "type": "heading_2",
      "breadcrumb": {}
    },
    {
      "type": "paragraph",
      "breadcrumb": {}
    },
    {
      "type": "bulleted_list",
      "breadcrumb": {}
    },
    {
      "type": "divider",
      "divider": {}
    },
    {
      "type": "heading_2",
      "breadcrumb": {}
    },
    {
      "type": "bulleted_list",
      "breadcrumb": {}
    },
    {
      "type": "divider",
      "divider": {}
    },
    {
      "type": "heading_2",
      "breadcrumb": {}
    },
    {
      "type": "paragraph",
      "breadcrumb": {}
    },
    {
      "type": "quote",
      "breadcrumb": {}
    },
    {
      "type": "quote",
      "breadcrumb": {}
    }
  ]
}

sxlijin commented 1 month ago

OK, so I think the right approach for you is to use singly-valued enums like so (it definitely seems to work when testing); I've filed #949 for us to look into the other issues coming up here, but using singly-valued enums as a hack for literals will, I think, do the trick for you:

/// ------------ Heading 3 ----------------
class Heading3 {
  type Heading3Type
  heading_3 HeadingBody @description("Heading 3 block with rich text")
}

class HeadingBody {
  rich_text RichText[] @description("The rich text content of the heading")
  is_toggleable bool @description("Whether the heading is toggleable")
}

/// ------------ Paragraph ----------------
class Paragraph {
  type ParagraphType
  paragraph ParagraphBody @description("Paragraph block with rich text")
}

class ParagraphBody {
  rich_text RichText[] @description("The rich text displayed in the paragraph block")
  children (Bookmark)[] @description("Optional nested child blocks for paragraph")
}

/// ------------ RichText ----------------
class RichText {
  type RichTextType
  text RichTextText @description("The content of the rich text element")
}

/// ------------ To Do ----------------
class ToDo {
  type ToDoType
  to_do ToDoBody @description("To-do block with checkbox and content")
}

class ToDoBody {
  rich_text RichText[] @description("The text content of the to-do item")
  checked bool? @description("Whether the to-do item is checked")
  children (Paragraph | Callout | ImageFile)[] @description("Optional nested child blocks for to-do")
}

// ---
class Page {
  type PageType
  object string @description("Always page")
  icon Icon @description("Icon of page")
  children (Bookmark | Breadcrumb | BulletedListItem | Callout | Code | ColumnList | Divider | Embed | Equation | File | Heading1 | Heading2 | Heading3 | ImageFile | NumberedListItem | Paragraph | PDF | Quote | Table | TableOfContents | ToDo | Toggle | Video)[] @description("Some blocks of the page")
}

enum BookmarkType {
  Bookmark @alias("bookmark")
}

enum BreadcrumbType {
  Breadcrumb @alias("breadcrumb")
}

enum BulletedListItemType {
  BulletedListItem @alias("bulleted_list_item")
}

enum CalloutType {
  Callout @alias("callout")
}

enum CodeType {
  Code @alias("code")
}

enum ColumnListType {
  ColumnList @alias("column_list")
}

enum DividerType {
  Divider @alias("divider")
}

enum EmbedType {
  Embed @alias("embed")
}

enum EquationType {
  Equation @alias("equation")
}

enum FileType {
  File @alias("file")
}

enum Heading1Type {
  Heading1 @alias("heading1")
}

enum Heading2Type {
  Heading2 @alias("heading2")
}

enum Heading3Type {
  Heading3 @alias("heading3")
}

enum ImageFileType {
  ImageFile @alias("image_file")
}

enum NumberedListItemType {
  NumberedListItem @alias("numbered_list_item")
}

enum ParagraphType {
  Paragraph @alias("paragraph")
}

enum PDFType {
  PDF @alias("pdf")
}

enum QuoteType {
  Quote @alias("quote")
}

enum RichTextType {
  RichText @alias("rich_text")
}

enum TableType {
  Table @alias("table")
}

enum TableOfContentsType {
  TableOfContents @alias("table_of_contents")
}

enum ToDoType {
  ToDo @alias("to_do")
}

enum ToggleType {
  Toggle @alias("toggle")
}

enum VideoType {
  Video @alias("video")
}

hellovai commented 1 month ago

@ekinsdrow I think this issue is fixed in 0.56+.

I added your specific unit test into our repo, and it appears to work well.

https://github.com/BoundaryML/baml/pull/980

BoundaryML / baml