Allow to disable skipping empty rows/columns at the beginning of the worksheet

[wikiped]

There is a similar issue, which is closed without, as it appears, any fix (with v0.9.1 being actual).

Expected behaviour is outlined in examples of python-calamine for an option skip_empty_area:

from python_calamine import CalamineWorkbook

workbook = CalamineWorkbook.from_path("file.xlsx").get_sheet_by_name("Sheet1").to_python(skip_empty_area=False)
# [
# [",  ",  ",  ",  ",  ",  "],
# ["1",  "2",  "3",  "4",  "5",  "6",  "7"],
# ["1",  "2",  "3",  "4",  "5",  "6",  "7"],
# ["1",  "2",  "3",  "4",  "5",  "6",  "7"],
# ]

This automatic behavior is kind of surprising when dealing with files with empty parts at the beginning, as the whole rows calculations get confusing. i.e. for header_row you have to count rows as they appear (i.e. including empty rows), but for skip_rows you have to count as if there are no empty rows, which is clearly not the most user-friendly approach.

Please consider adding an option/parameter to disable default behavor. Thank you!

[PrettyWood]

header_row is the row chosen for the header so of course all rows are taken into account. Once this is set, skip_rows is just the number of rows you want to skip for pagination for example. If you needed to add the header row every time, I reckon this would make the API harder to use.

That being said we could add a parameter similar to skip_empty_area. Just to know what is the use case?

[wikiped]

what is the use case?

In an ETL pipeline incoming Excel files use arbitrary layout (inluding multi-rows headers and/or empty rows between header and the data). The task is to process the files using predefined specs for first_header_row, last_header_row, first_data_row - set by the user once. The user is only dealing with visible Excel row numbering and is not aware of the internals of how the actual processing happens. So if the user sees that the header starts at row 2 and ends at row 2 and data starts at row 4 - the resulting dataframe should start from row 4. currently it does not. Because skip_rows=1 (based on previous settings) is making dataframe to start from row 5.

[PrettyWood]

This requires changes on calamine side. I will try to open a PR soon. Since 0.11 has some nice improvements, we don't want to postpone it even more for that

[severinh]

I'm afraid I'm facing the same issue.

Our systems are currently using python-calamine with .to_python(skip_empty_area=False).

What's the use case: In one of our systems, users specified a row index in their Excel sheet that is of special interest. But we cannot just start reading from that row (with header_row). We need the data of the entire sheet. And that sheet may contain empty rows at the top. By silently skipping empty rows at the top of the sheet (and not knowing how many were skipped), the sheet we would get out of fastexcel would not line up with the row configuration from the user, and we could not locate that special row anymore. I don't like the system for its fragility - but for now we're stuck with it. :)

Hence, for moving to fastexcel, we would need the equivalent of that skip_empty_area=False option in python-calamine.

[severinh]

@PrettyWood I see that you've been working to fix this by adding a header_row option to Calamine: https://github.com/tafia/calamine/pull/453. Thanks again. I'd like to understand: How will that be integrated with the fastexcel API? Will that then be tied to the header_row option of fastexcel? If yes, I worry that this improvement to calamine will not actually solve the problem we have with calamine/fastexcel. In short, using fastexcel's header_row with something other than None is not an option for us.

Concretely, we have the following need:

Suppose you have the following Excel sheet:

We need to convert that Excel file to a Polars DataFrame exactly how it is in Excel, where each row of the Excel is a row in the DataFrame (4 rows), and the DataFrame columns are any placeholder such as __UNNAMED_$X:

__UNNAMED_0	__UNNAMED_1
null	null
null	null
a	a
b	b

When attempting to switch from python-calamine to fastexcel, I tried to do that using excel_reader.load_sheet_by_name(name, header_row=None). But that does not currently work because the two empty rows get lost.

Once https://github.com/tafia/calamine/pull/453 is in and integrated, will it be possible to get the above DataFrame from FastExcel? And if so how? If I understand correctly, using excel_reader.load_sheet_by_name(name, header_row=0) will not be an option because the first null row would then be missing (since turned into DataFrame header).

In summary, I just need a way to get the rows of an Excel exactly as they are, without dropping any empty rows, and without turning any row into the DataFrame header.

[PrettyWood]

@severinh The API between fastexcel and calamine is a bit different. On fastexcel we already have header_row = None to say we don't want any header but at the moment you would get

header_row = None

┌──────────────┬──────────────┐
│ __UNNAMED__0 ┆ __UNNAMED__1 │
│ ---          ┆ ---          │
│ str          ┆ f64          │
╞══════════════╪══════════════╡
│ a            ┆ 0.0          │
│ b            ┆ 1.0          │
└──────────────┴──────────────┘

header_row unset

┌─────┬─────┐
│ a   ┆ 0   │
│ --- ┆ --- │
│ str ┆ f64 │
╞═════╪═════╡
│ b   ┆ 1.0 │
└─────┴─────┘

So the parameter I plan to modify on fastexcel side is skip_rows.

        :param skip_rows: Specifies how many rows should be skipped after the `header_row`.
                          Any rows before the `header_row` are automatically skipped.
                          If `header_row` is `None`, it skips the number of rows from the
                          start of the sheet.

None will mean "auto"

---

So to answer your question you'll be able to do that with load_sheet(..., header_row=None, skip_rows=0) Hope it's clear and I'm not mistaken 🙏

[severinh]

@PrettyWood Thanks for taking the time to explain the plan for fastexcel. I think what you describe should work.

Looks like in the meantime also your improvement to calamine was released as part of 0.26. Congratulations! :)