julian-ramos
commented
9 years ago Actually before the second step, organize the data in the next manner, this is just for graphical purposes, put all the morning then all the afternoon and then all the evening. This should not affect at all the clustering part.
Once you are done with all this there is another task that will be interesting to do. Can you go back to the session data and compute a transition matrix for each category. This means that you count every time you observe the transition from one app to the other. We do this for every device_id. Finally we obtain a transition matrix notice it may not be symmetric. Once we obtain that then the challenge again is trying to find whether there is a pattern in the transition probabilities. For all this task we should be using the merged categorization instead of the full one.
royxue
For future reference:
First, since what we got is so sparse first step is to simply merge categories. For instance we do not need to have every single game category in fact maybe we can just merge all the games into one category. So lets do it this way. Each one of us is going to group categories(with constraint of grouping only sparse categories which means a sparse category could join a non-sparse, however a non-sparse should not be considered for merge)
Second, we want to run a clustering algorithm on this data set. For that we need to find out the right number of clusters so we need to do the next: Run k-means for different sizes 2 to say a maximum of 20. Get for every clustering the silhouette score. Thus, we will get a trajectory so we will be able to see when can we expect to reach the right number of clusters. Also, we want to compute the gap statistic. Use the implementation in R which already does everything for you meaning you don't have to write the code for kmeans.
Third, check on the number of clusters we obtain. If reasonable then we can proceed to actually look at the centroids we obtain for that number of clusters and maybe even run different clustering algorithms.