ToucanToco / fastexcel

A Python wrapper around calamine
http://fastexcel.toucantoco.dev/
MIT License
82 stars 4 forks source link

Allow to disable skipping empty rows/columns at the beginning of the worksheet #209

Open wikiped opened 3 months ago

wikiped commented 3 months ago

There is a similar issue, which is closed without, as it appears, any fix (with v0.9.1 being actual).

Expected behaviour is outlined in examples of python-calamine for an option skip_empty_area:

from python_calamine import CalamineWorkbook

workbook = CalamineWorkbook.from_path("file.xlsx").get_sheet_by_name("Sheet1").to_python(skip_empty_area=False)
# [
# [",  ",  ",  ",  ",  ",  "],
# ["1",  "2",  "3",  "4",  "5",  "6",  "7"],
# ["1",  "2",  "3",  "4",  "5",  "6",  "7"],
# ["1",  "2",  "3",  "4",  "5",  "6",  "7"],
# ]

This automatic behavior is kind of surprising when dealing with files with empty parts at the beginning, as the whole rows calculations get confusing. i.e. for header_row you have to count rows as they appear (i.e. including empty rows), but for skip_rows you have to count as if there are no empty rows, which is clearly not the most user-friendly approach.

Please consider adding an option/parameter to disable default behavor. Thank you!

PrettyWood commented 3 months ago

header_row is the row chosen for the header so of course all rows are taken into account. Once this is set, skip_rows is just the number of rows you want to skip for pagination for example. If you needed to add the header row every time, I reckon this would make the API harder to use.

That being said we could add a parameter similar to skip_empty_area. Just to know what is the use case?

wikiped commented 3 months ago

what is the use case?

In an ETL pipeline incoming Excel files use arbitrary layout (inluding multi-rows headers and/or empty rows between header and the data). The task is to process the files using predefined specs for first_header_row, last_header_row, first_data_row - set by the user once. The user is only dealing with visible Excel row numbering and is not aware of the internals of how the actual processing happens. So if the user sees that the header starts at row 2 and ends at row 2 and data starts at row 4 - the resulting dataframe should start from row 4. currently it does not. Because skip_rows=1 (based on previous settings) is making dataframe to start from row 5.