dosubot[bot]
commented
5 months ago To address the issue with ConfluenceReader not handling multiline table cells correctly, you can modify the process_page method to better process HTML tables into markdown. This involves using BeautifulSoup to parse the HTML content, identify table elements, and convert them into a markdown format that preserves multiline cells.
Here's a concise approach to achieve this:
- Parse the HTML Content: Use BeautifulSoup to parse the page content and find all table elements.
- Convert Tables to Markdown: Implement a method,
_html_table_to_markdown, that converts HTML tables to markdown. This method should handle multiline cells by replacing newline characters with<br>tags to maintain the multiline appearance in markdown. - Replace HTML Tables with Markdown: In the
process_pagemethod, replace the original table HTML with the markdown version generated by your new method.
Here is a simplified example of how you might implement these changes:
from bs4 import BeautifulSoup
def process_page(self, page, include_attachments, text_maker):
html_content = page["body"]["export_view"]["value"]
soup = BeautifulSoup(html_content, 'html.parser')
for table in soup.find_all("table"):
markdown_table = self._html_table_to_markdown(table)
table.replace_with(markdown_table)
text = text_maker.handle(str(soup)) + "".join(attachment_texts)
return Document(text=text, doc_id=page["id"], extra_info={...})
def _html_table_to_markdown(self, table):
headers = [th.get_text(strip=True) for th in table.find_all('th')]
rows = table.find_all('tr')
markdown = '| ' + ' | '.join(headers) + ' |\n' + '| ' + ' | '.join(['---'] * len(headers)) + ' |\n'
for row in rows:
cells = row.find_all(['td', 'th'])
cell_texts = [cell.get_text(separator="\n", strip=True).replace('\n', '<br>') for cell in cells]
markdown += '| ' + ' | '.join(cell_texts) + ' |\n'
return markdownThis code snippet introduces a method to convert HTML tables to markdown, ensuring that multiline cells are correctly formatted. Adjust the process_page method to use this new functionality, which should resolve the issue with invalid markdown for tables with multiline cells.
Let me know if you need further assistance or have any questions regarding this solution.
#### Details
**Sources** - [llama-index-integrations/readers/llama-index-readers-confluence/llama_index/readers/confluence/base.py](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/readers/llama-index-readers-confluence/llama_index/readers/confluence/base.py)**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
To continue the conversation, mention @dosu.
glenacota
Bug Description
The ConfluenceReader doesn't produce valid markdown when loading Confluence pages with tables having cells with multiple lines.
For example, given a Confluence page with the following table
of items
* in cells
the ConfluenceReader will create a document having the following markdown as content
which is then not rendered as expected
| single line
Note that the LangChain ConfluenceLoader handles multiline table cells correctly - below is the document content generated for the same table used as a reference for the example above
Version
Version: 0.10.23
Steps to Reproduce
An example of a table + expected valid markdown + currently obtained invalid markdown is provided in the description of this bug issue.
Relevant Logs/Tracbacks
No response