swansonk14 / p_tqdm

Parallel processing with progress bars
MIT License
457 stars 44 forks source link

Progress bar not progressing while running function #14

Closed VinodKumar9576 closed 4 years ago

VinodKumar9576 commented 4 years ago

Progress bar not progressing while running the function, but directly showing full completed bar after 100%.

VinodKumar9576 commented 4 years ago

image You can see here, program is running but progress bar still showing 0.

VinodKumar9576 commented 4 years ago

` def txt2vec_q1 (df): vecs1 = [] for qstn1 in list(df): doc1 = nlp(qstn1) #word2vec conversion using mean_vec1 = np.zeros([len(doc1),300])

making array of zeros of size of question

for word1 in doc1:
  vec1 = word1.vector #getting the vector of each word in question
  #vec1 = vec1.reshape(len(vec1),1)
  try:
    idf = tfidf_vect[str(word1)]
  except:
    idf = 0

  mean_vec1 +=vec1*idf
mean_vec1 = mean_vec1.mean(axis=0)
vecs1.append(mean_vec1)

return vecs1

from p_tqdm import p_map q1_txt2vec = p_map(txt2vec_q1, qstns_train['question1'])

q1_txt2vec_arr = np.array(q1_text2vec) qstns_train['q1_feats_m'] = list(q1_txt2vec_arr.reshape(q1_txt2vec_arr.shape[1],q1_txt2vec_arr.shape[2]))`

VinodKumar9576 commented 4 years ago

Fyi my df qstns_train['question1'] of shape 409456 X 1

swansonk14 commented 4 years ago

Hi @VinodKumar9576 , thank you for pointing that out! I previously had code in p_tqdm that required the iterables to be lists and treated other objects (like pandas DataFrames) as single element lists. I've now fixed this so that all iterables, including pandas objects, are processed correctly: https://github.com/swansonk14/p_tqdm/releases/tag/v_1.3.3. Please install p_tqdm version 1.3.3 to get the fix.

VinodKumar9576 commented 4 years ago

Thank you

VinodKumar9576 commented 4 years ago

Hi Still dataframe input is being converted to list. See the example simple program.

image

Input contains string in each dataframe row. Instead of printing the whole sentence, each letter is printed.

VinodKumar9576 commented 4 years ago

Hi @swansonk14,

can you please check the above, still not solved for Dataframe inputs.