Create a clustering algorithm to cluster based on the feedback properties (run in SageMaker)

[toddysm]

Cluster the data that is exported in #91

The properties are:

• Sentiment (positive, negative, neutral)
• Question/No question
• Emotion

[ZenSorcere]

Initial Estimates

• Planning (issue writeup, initial rough time estimations): 30 min
• Research (Clustering algorithms, Sagemaker methods/syntax/how-tos, previous clustering issues in repo, nltk and other javascript libraries): 8 hours
• Implementation and Testing (code writing/optimizing, getting needed imports/libraries to work, functional clustering algorithm adjustments, error handling): 6 hours
• Documentation (writeup code explanation and how-to for unfamiliar user): 1.5 hours
• Deployment (PR Write-up): 1.5 hours TOTAL: 17.5 hours

[ZenSorcere]

Clustering algorithm to cluster based on Floop extracted conversation metadata

Pre-Requisites:

• All required imports are included in the sagemaker notebook.

Implementation:

• Implemented via Sagemaker notebook: iss127-mpg2-clustering-of-fedback-attr
• Script accesses the S3 bucket and the enriched-conv-1k-sagemaker.json file. Metadata elements for all json entries are pulled an added to an array of dictionaries, which are converted to a pandas dataset.
• An elbow curve algorithm is used to determine optimal number of clusters (which turns out to be 3).
• Using KModes clustering (one of the unsupervised Machine Learning algorithms that is used to cluster categorical variables), we can calculate the dissimilarities between observations and assign them to the closest cluster, as determined by selecting random observations as leaders/cluster definitions. New modes are defined for each cluster, and the process repeats until no re-assignments are required.
• A "Cluster" column is added to the dataframe, and the determined cluster number is assigned for each metadata set.
• The final dataframe output in Sagemaker looks like the below image:
• A bar graph is shown to give visual indication of the cluster sizings:

[ZenSorcere]

Activities Documentation

Date	Activity	Time Spent
3/5/22	Planning	15 minutes
	Research	3 hours
	Implementation/ Testing	1.5 hours
	Task Management	15 minutes
3/06/22	Implementation/ Testing	2 hours
	Research	1 hour
3/07/22	Research	45 minutes
	Implementation/ Testing	15 minutes
3/09/22	Documentation	30 minutes
3/10/22	Documentation	15 minutes
	Discussion	15 minutes
3/11/22	Research	1 hour
	Implementation/Testing	1 hour
	Deployment	1 hour

Time Totals

Activity	Time Spent
Planning	.25 hour
Research	5 hours
Implementation/Testing	4.75 hours
Task Management	.25 hour
Deployment	1 hour
Documentation	.75 hour
Discussion	.25 hour
TOTAL TIME:	12.25 HOURS