I am processing some financial texts that contain tables with MarkdownTextSplitter. It works very well but perhaps too well for my needs.
As you could see from logs below I am getting very small chunks that contain some hierarchy information but only single data row from markdown table per chunk. This is significantly diminishing the quality of results...
If I send larger part of table or entire table to LLM (via TextSplitter) I get more context and model can sum up rows, subtract and give more intelligent answers in general.
Single row isn't very informative on its own. With just a few random rows selected form entire table (and not all rows on their own match similarity search criteria as they do contain very little information) I don't get good answers placed in larger context.
So my question: is there a way to stop splitting at table not row level?
2024/06/28 05:31:16 ---chunk---
2024/06/28 05:31:16 # DODATKOWE INFORMACJE I OBJAŚNIENIA
## Warszawa, 24 marca 2020 r.
| Treść | 2018 | 2019 |
| --- | --- | --- |
| Pracownicy na stanowiskach nierobotniczych (etaty) | 1 554 | 2 343 |
2024/06/28 05:31:16 ---chunk---
2024/06/28 05:31:16 # DODATKOWE INFORMACJE I OBJAŚNIENIA
## Warszawa, 24 marca 2020 r.
**_3)_** **_Przeciętnym w roku obrotowym zatrudnieniu, z podziałem na grupy zawodowe;_**
| Treść | 2018 | 2019 |
| --- | --- | --- |
| Pracownicy na stanowiskach robotniczych i pokrewnych (etaty) | 16 | 15 |
2024/06/28 05:31:16 ---chunk---
2024/06/28 05:31:16 # DODATKOWE INFORMACJE I OBJAŚNIENIA
## Warszawa, 24 marca 2020 r.
| Treść | 2018 | 2019 |
| --- | --- | --- |
| Zatrudnienie wg stanu na dzień bilansowy w osobach | 1 575 | 2 363 |
2024/06/28 05:31:16 ---chunk---
2024/06/28 05:31:16 # DODATKOWE INFORMACJE I OBJAŚNIENIA
## Warszawa, 24 marca 2020 r.
| Treść | 2018 | 2019 |
| --- | --- | --- |
| w tym kobiety | 821 | 1 368 |
2024/06/28 05:31:16 ---chunk---
2024/06/28 05:31:16 # DODATKOWE INFORMACJE I OBJAŚNIENIA
## Warszawa, 24 marca 2020 r.
| Treść | 2018 | 2019 |
| --- | --- | --- |
| Ogółem przeciętne zatrudnienie (etaty) | 1 415 | 1 834 |
2024/06/28 05:31:16 ---chunk---
2024/06/28 05:31:16 # DODATKOWE INFORMACJE I OBJAŚNIENIA
## Warszawa, 24 marca 2020 r.
| Lp. | Treść | 01.01.-31.12.2018 r. | 01.01.-31.12.2019 r. |
| --- | --- | --- | --- |
| 8 | Wynagrodzenie z tytułu funkcji płatnika: | 3 270 687,00 | 3 515 863,00 |
2024/06/28 05:31:16 ---chunk---
2024/06/28 05:31:16 # DODATKOWE INFORMACJE I OBJAŚNIENIA
## Warszawa, 24 marca 2020 r.
| Wyszczególnienie | Stan na 31.12.2018 r. | Zwiększenia | Wykorzystanie | Rozwiązanie | Stan na 31.12.2019 r. |
| --- | --- | --- | --- | --- | --- |
| - niewykorzystane urlopy | 8 918 615,86 | 2 057 949,16 | 0,00 | 0,00 | 10 976 565,02 |
2024/06/28 05:31:16 ---chunk---
2024/06/28 05:31:16 # DODATKOWE INFORMACJE I OBJAŚNIENIA
## Warszawa, 24 marca 2020 r.
| Wyszczególnienie | Stan na 31.12.2018 r. | Zwiększenia | Wykorzystanie | Rozwiązanie | Stan na 31.12.2019 r. |
| --- | --- | --- | --- | --- | --- |
| - regulaminowe wygrane | 11 417 995,55 | 11 057 134,55 | 0,00 | 10 773 580,72 | 11 701 549,38 |
I am processing some financial texts that contain tables with MarkdownTextSplitter. It works very well but perhaps too well for my needs.
As you could see from logs below I am getting very small chunks that contain some hierarchy information but only single data row from markdown table per chunk. This is significantly diminishing the quality of results...
If I send larger part of table or entire table to LLM (via TextSplitter) I get more context and model can sum up rows, subtract and give more intelligent answers in general.
Single row isn't very informative on its own. With just a few random rows selected form entire table (and not all rows on their own match similarity search criteria as they do contain very little information) I don't get good answers placed in larger context.
So my question: is there a way to stop splitting at table not row level?