BUG: read_csv with lines starting `whitespace #`: engine="python" skips them, engine="c" doesn't

[denis-bz]

Pandas version checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of pandas.

• [X] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
from io import StringIO
print( f"{pd.__version__ = }" )

csv = """
col1;col2;col3
1;4.4;99
    #2;4.5;200
3;4.7;65
""" 

for engine in ["c", "python"]:
    print( f"\n-- {engine = }" )
    df = pd.read_csv( StringIO( csv ), sep=";", comment="#", engine=engine,
            skip_blank_lines=True,  # default
            )
    print( df )

Issue Description

pd.version = '2.1.2'

-- engine = 'c' col1 col2 col3 0 1 4.4 99 1 \t NaN NaN 2 3 4.7 65

-- engine = 'python' col1 col2 col3 0 1 4.4 99 1 3 4.7 65

Expected Behavior

with skip_blank_lines=True, all engines should skip lines starting with whitespace commentchar

Installed Versions

INSTALLED VERSIONS ------------------ commit : a60ad39b4a9febdea9a59d602dad44b1538b0ea5 python : 3.10.0.final.0 python-bits : 64 OS : Darwin OS-release : 19.6.0 Version : Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 machine : x86_64 processor : i386 byteorder : little LC_ALL : None LANG : None LOCALE : en_GB.UTF-8 pandas : 2.1.2 numpy : 1.26.1 pytz : 2023.3.post1 dateutil : 2.8.2 setuptools : 57.4.0 pip : 23.3.1 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : 4.9.3 html5lib : 1.1 pymysql : None psycopg2 : None jinja2 : 3.1.2 IPython : 8.18.0.dev pandas_datareader : None bs4 : 4.12.2 bottleneck : None dataframe-api-compat: None fastparquet : None fsspec : 2023.6.0 gcsfs : None matplotlib : 3.8.1 numba : None numexpr : None odfpy : None openpyxl : 3.1.2 pandas_gbq : None pyarrow : None pyreadstat : None pyxlsb : None s3fs : None scipy : 1.11.3 sqlalchemy : None tables : None tabulate : None xarray : 2023.10.1 xlrd : None zstandard : None tzdata : 2023.3 qtpy : None pyqt5 : None

[denis-bz]

messier than I thought, try sep=r"\s+" too related: https://github.com/pandas-dev/pandas/issues/36069

[denis-bz]

Fwiw, a tiny testcase generator for this --

# 24 x 4 csv, permutations of A , \t #
# to test https://github.com/pandas-dev/pandas/issues/55916

from itertools import permutations
from io import StringIO 
import pandas as pd

print( f"{pd.__version__ = }" )  # 2.1.3

rows = ["A,B"] + ["".join( p ) for p in permutations( "A,\t#" )]
csvstr = "\n".join( rows )

for engine in ["c", "python"]:
  # for sep, for comment
    print( f"\n-- {engine = }" )
    df = pd.read_csv( StringIO( csvstr ), sep=",", comment="#", engine=engine,
            skip_blank_lines=True,  # default
            )
    print( df )