Open dylanzxc opened 4 years ago
Good job @dylanzxc . This is a good starting point. I would like to hear your thoughts on which insights should we implement for dataprep and are there any other insights that we should include?
To answer the first question, you can put each insight into the following categories. Below I show an example of Category outliers. In this way, you can identify which insights are more useful and common to us.
To answer the second question, you can ask yourself what you are looking for when seeing, e.g., a histogram. You can check i) dispersion (whether the data is concentrated around the mean), ii) skewness, ii) following a certain distribution. Then, you may add insights like high dispersion, high skewness, fit to a normal distribution, etc.
@jnwang Thank you, Professor. You gave me a very good outline to consider this problem. Based on the existing plots we have in dataprep, I think we can include the following insights.
Categorical Data Insights:
Category Outliers Add the insight with bar chart in plot(df), plot(df,x)
Majority/Attribution(share >=50): Outstanding 1 Outstanding 2 Outstanding last
Add these insights with bar chart and pie chart in plot(df), plot(df,x)
Numerical Data Insights:
Add these insights with histogram in plot(df), plot(df,x)
A related question yield is, are we gonna count outstanding on each bin or on every single value. If it’s on each bin then it's easy to implement, which is very similar to the categorical data, however, if it’s based on every single value then we need a hypothesis test algorithm(existing paper to learn).
Datetime Data Insights:
Correlation Insights:
Plot_missing insights
For question 2 are there any other insights that we should include?
I think besides adding more insights, we can discover more dimensions of the datasets in other words add more groupby() and show the above insights on the new dimensions or cross dimensions. For example, for categorical data, we can group by each category and for datatime, we can group by month, week, or more specific intervals to yield insights. I think this kind of approaches are very common and useful in business analysis area such as user habit analysis.
My plan for this task is, starting with easier insights using existing graphs and computation results, doing research at the same time for more ideas. One of the potential difficulty is, to generate some of the above insights(eg. time-series insights), we need a reliable and rigorous algorithm from statistics' point of view such as the following screenshot. This could take some time.
I'd like to hear your advice :) @dovahcrow @jinglinpeng
Good job @dylanzxc . In a high level, there are two places to show insights:
For each fig. in current function(plot
, plot_correlation
and plot_missing
), we show the insight associated with that fig.
We have a place to show the ranked top-k insights of the whole dataset. Still, the insight is associated with the fig.
I think currently the first part is clear. For the second part, it is unclear 1) what new insights we want to add. 2) whether we need a separate function such as insight(df)
or we can directly put it into plot(df, insight = True)
. I agree with your plan. Let us work on the first part as a starting point.
Hi, I have summarized the insights from Power BI and papers that we can probably include in dataprep. I think we can have a discussion on this.
The following two links are very helpful and there are examples to most of the insights I mentioned above. https://www.microsoft.com/en-us/research/uploads/prod/2016/12/Insight-Types-Specification.pdf https://docs.microsoft.com/en-us/power-bi/consumer/end-user-insight-types @jnwang @jinglinpeng @dovahcrow @brandonlockhart @eutialia @Sanjana12111994 @Waterpine