Open galipremsagar opened 3 years ago
cc: @kaatish @vuule
If the values overflow
int
&uint
types, they should be inferred asstring
column.
I don't think we want this behavior. Pandas infers as object
which are Python integers as opposed to strings, no?
Pandas infers as
object
which are Python integers as opposed to strings, no?
Not actually, pandas appears to be storing them as strings:
>>> import pandas as pd
>>> df = pd.read_csv('a.csv')
>>> df['3']
0
1 17758512297797920768
2
3 3374168267804635136
4
...
365
366
367
368 14301866441110444032
369
Name: 3, Length: 370, dtype: object
>>> df['3'][1]
'17758512297797920768'
>>> type(df['3'][1])
<class 'str'>
TIL... Either way, I would defer to what makes sense from a general C++ perspective instead of forcing Pandas semantics.
In this case, cudf CSV reader is inferring the column type to be float column. This will lead to loss of data partially.
IMO this is a good reason to use strings here.
This issue has been marked stale due to no recent activity in the past 30d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be marked rotten if there is no activity in the next 60d.
This issue has been labeled inactive-90d
due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.
Describe the bug A csv files contains values greater than max(int64) and some null values. In this case, cudf CSV reader is inferring the column type to be
float
column. This will lead to loss of data partially.Steps/Code to reproduce bug
a.csv
: a.csv.zip(Rename this toa.csv
)Expected behavior If the values overflow
int
and fit intouint
, they should be inferred asuint
. If the values overflowint
&uint
types, they should be inferred asstring
column.Environment overview (please complete the following information)
Environment details Please run and paste the output of the
cudf/print_env.sh
script here, to gather any other relevant environment detailsClick here to see environment details
Additional context Surfaced in fuzz testing: #6001