Dataset property: columns types

[alessiamarcolini]

• numerical_cols [2b]
• categorical_cols [2b]
• boolean_cols [2b]
• string_cols [2b]
• mixed_cols [2b]

[lorenz-gorini]

• compare all vs np.all

• previously consider the pandas dtype and analyze only columns with object dtype

• use pd.select_dtypes to select columns with a certain dtype recognized by pandas (it may use generic dtypes like number

• use pd.to_numeric or pd.to_timedelta or pd.to_datetime when fixing errors (before using some manually selected dictionaries)

pd.api.types.infer_dtype()

We may use this pandas function to find the column types. It uses multiple layers for recognition:

1. dtypes of the Series (if they are pandas special types only. See https://pandas.pydata.org/pandas-docs/stable/user_guide/basics.html#dtypes),
2. dtype of the numpy array
3. If 2. returned a np.dtype = object, then further checks are performed by looking at the type of each sample. Possible types are: DateTime, TimeDelta, Integer,...

What we need to implement

• Combine some returned strings into one specific column type (floating, integer, decimal, complex -> numeric) -> use a map/dictionary
• Identify categorical columns (that are automatically recognized by pandas only if the dtype was already manually specified)

Categorical Data

• CategoricalDType

  cat_type = pandas.CategoricalDType(categories=['a','b'], ordered=True)
  s = pd.Series(list('babc')).astype(cat_type)  # also `CategoricalDtype(list('abcd'))` can replace cat_type directly