read_datum updates

[zkokaja]

• put in_glove option inside align_with loop so that it's not a default.
  • verify that in_gpt2 only includes words and not tokens
• change "correct" and "incorrect" mods to use rank instead of just top1 (may be pending pickling updates)

[zkokaja]

@hvgazula for second item

[hvgazula]

second item addressed in 3bd2c384

[zkokaja]

This issue is blocked until pickling includes the rank in pickles: issue

[hvgazula]

https://github.com/hassonlab/247-pickling/issues/56 Resolved and pushed to main.

[zkokaja]

@VeritasJoker now you can perform this change in encoding. @miahong please link your code where you do top-5 prediction

[miahong]

@zkokaja @hvgazula
https://github.com/miahong/247-pickling/blob/forharsha/scripts/tfsemb_main.py#:~:text=%23%20Added%20for%20top5,)%2C%205)%5D

[miahong]

@zkokaja @hvgazula https://github.com/miahong/247-pickling/blob/forharsha/scripts/tfsemb_main.py#:~:text=top5_predicted_words%20%3D%20%5B%5D,(top5_predicted)

[miahong]

@zkokaja @hvgazula https://github.com/miahong/247-pickling/blob/forharsha/scripts/tfsemb_main.py#:~:text=%23%20top%2D1%20probabilities,true_y_probability%2C%20entropy

[miahong]

@zkokaja @hvgazula https://github.com/miahong/247-pickling/blob/forharsha/scripts/tfsemb_main.py#:~:text=y_true%20%3D%20np,%5D%20%3D%20final_top5_prob

[miahong]

@zkokaja @hvgazula https://github.com/miahong/247-pickling/blob/forharsha/scripts/tfsemb_main.py#:~:text=tokenizer.eos_token-,final_embeddings%20%3D%20%5B%5D,df%5B%22entropy%22%5D%20%3D%20entropy,-return%20df%2C

[zkokaja]

second part of this request is done now as well