support tabular data with csv files

zzsi commented 4 years ago

Given a folder of csv or csv.gz files with a shared schema, output:

Table stats: row count, column count, column names with inferred types. With an inferred type, also infer whether it is a numerical or categorical variable. For numerical variable, output min, max, mean, median. For categorical, output number of levels, and frequencies of levels.
Percentage of missing (including nan, null, "") values for each column. Rank them by percentage missing.
Number of duplicate records.

No ml model training is involved.

@mwharton3 does this look like what you want for tabular data?

zzsi commented 4 years ago

I can work on it, though I don't have permission to assign issues to myself

mwharton3 commented 4 years ago

I just gave everyone write access, you should be able to work on it now.

And yes, totally. I wonder if we just extend functionality of a pandas data frame? I’ve found the df.describe() method to be really useful in particular (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html).

@jerryschirmer was talking about something like this yesterday, I wonder if he might have any other ideas to overcome pandas limitations.

mwharton3 commented 4 years ago

Include save feature as well

kungfuai / kaishi