Open galipremsagar opened 3 years ago
This is an issue in pandas as well. Our code is conformant with pandas behavior. https://github.com/rapidsai/cudf/blob/60d25ea04bf018905836310a18133806f780cb22/cpp/src/io/csv/reader_impl.cu#L583-L587
In [18]: pdf = pd.DataFrame({'a':pd.Series([1, 2, 3, None], dtype='Int64')})
In [19]: pdf.dtypes
Out[19]:
a Int64
dtype: object
In [20]: pdf.to_csv('temp.csv')
In [21]: cat temp.csv
,a
0,1
1,2
2,3
3,
In [22]: pd.read_csv('temp.csv').dtypes
Out[22]:
Unnamed: 0 int64
a float64
dtype: object
I guess the question is with nullable dtypes support introduced in pandas aswell do we still want to mimic pandas behavior?
I guess the question is with nullable dtypes support introduced in pandas aswell do we still want to mimic pandas behavior?
Agreed. This sounds like a bug in Pandas at this point and we shouldn't replicate the behavior. @galipremsagar can you open an issue on Pandas for this?
Agreed. This sounds like a bug in Pandas at this point and we shouldn't replicate the behavior. @galipremsagar can you open an issue on Pandas for this?
Linking for reference: https://github.com/pandas-dev/pandas/issues/36712
From the Pandas issue: Pandas will not change this behavior until they start using nullable types by default. We can still go ahead with the change.
This issue has been marked rotten due to no recent activity in the past 90d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.
Looks like we didn't go ahead with the change :D
FWIW, Pandas still does not default to nullable types, and there's a PR to add an option to use nullable types in read_csv
https://github.com/pandas-dev/pandas/pull/48776.
Describe the bug We seem to be reading in an integer column with nulls as a floating column. This is similar to pandas behavior but since we have support for nullable dtypes in all dtypes, should we even type-cast to float?
Steps/Code to reproduce bug
Expected behavior Expected behavior is to not type-cast to float and only cast to the inferred dtype.
Environment overview (please complete the following information)
Environment details Please run and paste the output of the
cudf/print_env.sh
script here, to gather any other relevant environment detailsClick here to see environment details
Additional context Surfaced while running fuzz tests: #6001