Final Exam

[darkestfloyd]

Only doubts, no cheating!

[darkestfloyd]

chi2 for feature selection

[DeekshaD]

I asked for clustering

[darkestfloyd]

cant use dbscan

[Omairss]

I'm trying autoencoder

[DeekshaD]

I used gmm. 0.63 hm

[Omairss]

You used GMM for feature selection?

[DeekshaD]

gmm for clustering, pca for reduction

[darkestfloyd]

shift incorrect cluster labels to majority cluster. got 6% increase

[Omairss]

I'm getting 10% for weighted average

[darkestfloyd]

lol, ill commit my notebook

[darkestfloyd]

just make sure you dont copy anything across!

[DeekshaD]

How the fuck to read this text file? DO i parse it?

[Omairss]

ng_df = pd.DataFrame(columns = ['id', 'topic', 'text'])

ng_list = []

for line in ng: temp_dict = {}

try: 
    #print(line.split(',')[1])
    #print(line.split(',')[0].split("[")[1])
    print(line.split(',')[2].split("]")[0])

    temp_dict['topic'] = line.split(',')[1]
    temp_dict['id']  = line.split(',')[0].split("[")[1]
    temp_dict['text'] = (' ').join(line.split(',')[2:])

except Exception as e: print (line)

ng_list.append(temp_dict)

ng_df = pd.DataFrame(ng_list)

[DeekshaD]

What is the shape of your countvect transformed matrix?

[darkestfloyd]

(18846, 1000)

[DeekshaD]

I'm not getiing it, what di you pass for the vectorizer?

[DeekshaD]

Arent there only 4k lines in sample text?

[darkestfloyd]

vectorize on full data, then pick only the 4k rows

[Omairss]

Want me to post the code?

[DeekshaD]

Are you done?

[darkestfloyd]

ill push both my notebooks in 10 mins, anything that has an empty comment on top of the block, use that.