Question about the batch encoding

Hi, Congrats on leading position of MTEB with a 1.5B model! Here I want to consult about the batch encoding problem. Specifically, I found that LLM-based embedder is sensitive to the encoding batch size. You could verify with:

all_text = [
 "Becky Bloomwood |  scheduled to commence immediately, Luke orders her to sort everything out in her spare time. Taking the advice of her kindly neighbors, Becky uses eBay in order to sell most of the items to make most of the money back to pay Luke back. She also accidentally sells 20 Tiffany clocks (which was meant as a gift for the Arcodas Group). When Luke discovers Becky's mistake from his co-worker who bought the clocks for a cool 20k pounds, he is cross with her careless attitude and they go days without talking to each other. The Bloomwoods come for a visit and ",
 'Becky Bloomwood |  ready to tell Becky the news. They wanted to wait until she settled back home before revealing important news to their daughter. Becky learns that she has an older half-sister named Jessica because her father had a relationship with another woman named Marguerite. Graham explained that they had a relationship before she met another man named Bill Bertram who was an owner of a grocery store in Cumbria and married him. Excited at the prospect of having a sister, Becky thinks that Jessica will be exactly like her, will share a similar passion for shopping and watching films and even helping ',
 "Becky Bloomwood |  cope with Suze no longer being her friend. Becky is disappointed when she initially meets Jess, who is thrifty, environmentally friendly and a bit standoffish. Jess lacks enthusiasm when Becky gives her a gift. After a heated confrontation with her sister, Becky faces marital problems with Luke whom is displeased with the way she's behaving and having found out that she set him up for a meeting with Nathan Temple. While both of them spend a few days fighting with each other, Becky feels insulted when Luke talks highly of Jess in being more helpful than her. She is angry and ",
 "Becky Bloomwood |  tears, Becky tells him off that she's had a lot of her own problems the last few weeks and he never had time for her. Despite showing compassion for her, Luke maintains he needs time away from Becky and hopes while he's away, she can reassess her behavior. After he leaves, Becky reads from a paper which compiles a list of complaints Luke has with her and concludes that her marriage has fallen apart. She tries to call Suze, but she is too busy with Lulu in planning an outing for their children. Next, Becky attempts to call her parents for ",
 "Becky Bloomwood |  only to remember that they're going on a wellness cruise. After a trip to the supermarket fails to distract Becky from her thoughts, she tries to sleep it off in her first night in the loft alone. Soon she has a nightmare about transforming into her nemesis, Alicia, that makes her realize what she has done. Becky heads to Scully in Northern England to reconcile with Jess and learn how to be thrifty. Jess initially refuses to help out after her behavior in London, but Jim offers her a place to stay at his friend's Bed and Breakfast. While in Scully, ",
 "Becky Bloomwood |  befriends Jim's teenaged daughter, Kelly, and she becomes a supporting friend similar to Suze. Learning more from Jim about her sister's past under Bill's harsh upbringing, Becky finally realizes that Jess had been originally right about her being a spoiled brat all her life. After failing to talk to Jess, Kelly suggests that she forgets about her and believes they're nothing alike. Becky almost takes her advice, but Jim convinces her to come with his family up to an environmental meeting. At an environmental meeting, she befriends Robin and other protesters and they like her ideas for more media attention. However, ",
 "Becky Bloomwood |  is against it and wants Becky to leave. Robin insists that she stays because he and the other protesters feel that she had better ideas and the meeting is no place for personal family issues. While talking to Jess alone, Becky discovers that she recently had a test done for a blood clot from an accident she had years earlier. Becky is shocked by this, but then recalls that she was told to get tested by Jess if they were related, something that she pushed on the back of her mind. Jess admitted that after learning more about her mother's flirtatious ",
 'Becky Bloomwood |  from her aunt, she was really seeking a DNA test between her and Graham before finding more potential fathers. Becky comes to the conclusion that since there was no DNA test done along with everyone else assuming, she and Jess may not be related and is heartbroken by this. Before leaving, Becky apologizes to Jess for her earlier behavior and admits she was right. Her behavior compounded more problems than solutions and her marriage to Luke finally fell apart. Jess forgives Becky and wishes her the best of luck in saving her marriage. Back at her room, Becky ponders what to ',
 "Becky Bloomwood |  as she packs her bags for the return trip home. Without Suze to help her cope and her parents being away, Becky must find a way to save her marriage on her own. Robin and the other protesters beg Becky to stay and help them, but she refuses and tells them she must return home. Jim tries to convince her to stay and sincerely admits he is concerned about her well being and having a mental breakdown while she's alone in London. He believes by helping the protesters out in getting more media coverage, it could distract her from her problems ",
 "Becky Bloomwood |  home. Becky appreciates Jim's concerns, but remains resolute in returning home to save her marriage to Luke. Before leaving, Jim shows Becky, Jess's rock collection and tells her the story about everything. This made her resentment not only disappear, but also made Becky feel guilty for the way she hurt Jess by insulting her passion in rock collecting. She also comes to the realization that shopping for unnecessary items can be boring as well, but has never admitted it until now. Becky follows Jess to the annual hiking to apologize again, finally realizing that Jess is exactly like her and share ",
 "Becky Bloomwood |  same level of passion in spite of their different hobbies. Unfortunately, she is not only wearing the incorrect footwear for such a long hike, but also inappropriately dressed as well. Becky doesn't follow the trail and gets injured. Luckily for her, Jess has the mental and moral fortitude to help her. Becky reconciles with Jess when she admits to how she had seen the beauty of her rock collection and the passion she holds for them. Jess tries to deny it, but Becky admits that she hides the passion better. She admits when it comes to shopping, Becky has a harder ",
 "Becky Bloomwood |  hiding it when it comes to clothes and other unnecessary items. It was also there Jess finally tells Becky that she was only half to blame for the marriage falling apart and also reveals Luke's part of the damage because he was so busy trying to win over his new clients that he ignored her needs. Jess then asks about Suze, which Becky admits she had found a new friend and her other friend, Danny being to busy that she hasn't got any friends. She discredits that claim and admits that Becky has her as a friend and sister. Jess finally ",
 'Becky Bloomwood |  to help Becky learn how to be thrifty and loves the silver Tiffany bean necklace she gave her. At the hospital, she manages to reconcile with Luke, who reveals that he was quite angry with her for having set up a meeting with Nathan Temple behind his back and admits she was originally right about Temple. He promises to work out their marriage together. Becky also reconciles with Suze, whom admits while she still likes having Lulu around, she also misses the fun in being best friends with her. At the end, Becky enjoys being thrifty and learns of her pregnancy.',
 "Becky Bloomwood |  In Shopaholic & Baby Becky is going through the joys of expecting her first child, complete with shopping for baby and a new home. However, there is only one problem, she is going to Dr. Braine, a family obstetrician who has seen previous Brandon women's pregnancies and which Becky doesn't like going to. During a shopping trip to Bambinos, she overhears a conversation between other women about Venetia Carter, a celebrity obstetrician to whom she wants to go. After convincing Luke to see her, Becky discovers why he is uncomfortable with changing obstetricians. Venetia is his ex-girlfriend from his university days, but Becky doesn't think much of it ",
 "Becky Bloomwood |  she reassures them that she's seeing someone else. She suspects that Luke is having an affair Venetia, particularly after Venetia herself tells Becky as much. After she persuades Becky that she and Luke are seeing each other and their relationship was like Penelope and Odysseus. She asks Luke if he's having an affair with Venetia. He denies this and admits the truth to Becky about his ex-girlfriend. Realizing the truth and feeling bad for wanting to change obstetricians, she apologizes to Luke for her behavior. They agree it's in the best interest of their baby to return to Dr. Braine. During that time, Becky still has issues with ",
 'Becky Bloomwood |  best friend, Lulu, and discovers a couple of photos that reveals her for the hypocrite she is from a PI she hired in secret to keep an eye on Luke and Venetia. During her baby shower, Becky shows Suze the photos of Lulu\'s own problems with her children and she sees her for who she is. However, Suze refuses Becky\'s request to publish an article exposing Lulu on the British tabloid "The Daily World" and prefers to keep the photos safe so she can look at them in secret. Becky learns about Arcodas Group\'s CEO, Iain Walker, whom is making horrible rumors about Brandon C\'s reputation and threatening ',
 "Becky Bloomwood |  leave. She later receives photos from the same PI of him being in the company with other women, including her current obstetrician. At first he thought he was photographing Luke, but when Becky points Luke out in the background, the PI got angry at his son. The PI's son defends himself by admitting he had actually been following Iain and Venetia for a while because of possible rumors that they're planning something bad for Brandon Communications. Overhearing this, she suspects that Venetia is helping Iain ruin Brandon C's reputation in her latest attempt to get Luke to leave Becky. She shows the photos of Iain and Venetia ",
 'Becky Bloomwood |  to Luke, but refuses to answer him about where she got them from (possibly feeling bad for suspecting Luke of having an affair with Venetia). Near the end, Luke rebukes Venetia for what she did to Becky and tells everyone about her troublesome nature. Back in college, Venetia feign a pregnancy in her attempt to keep them in a relationship together. Though he was relieved that she wasn\'t pregnant, Luke was still infuriated with Venetia\'s behavior and ended their relationship. In a last ditch effort, she tries to convince Luke to leave Becky, calling her a "dim-witted little airhead" who doesn\'t deserve him. Furious with the insult, ',
 "Becky Bloomwood |  refuses and tells Venetia off that he loves his wife and there's nothing she can do to change that. Luke states that Becky may shop at times, but he sees her as a caring and intuitive person, whereas he sees Venetia as a lying and manipulative person. In that moment, Becky informs Luke they lost out on the house they were planning to buy and are effectively homeless. Suze offers them her family's unused summer home in Scotland, but the Bloomwoods insist that Luke and Becky stay in their home in Surrey. In the midst of the arguing and being homeless, Becky's water breaks and she goes ",
 'Becky Bloomwood |  labor. Her furious mother, Jane, rebukes Venetia for her mistreatment towards her daughter and orders her out of their lives. Once she\'s gone, Suze reveals her plans to tell "The Daily World" about Venetia in the hopes of helping her friends and other pregnant women avoid getting her. During that time, a family member calls Dr. Braine in to assist Paula with Becky\'s labor. Through her professional and personal woes, Becky happily ends the book a new mum to daughter Minnie. Luke and Becky temporarily move in with her parents after losing out on the home they hoped to buy, while Luke works to save his company.',
 "Becky Bloomwood |  In Mini Shopaholic, Minnie is now 2 years old. There is a major financial crisis, and after Minnie's christening is a bust, Becky decides to throw a surprise party for Luke's birthday. However, many doubt she can keep the party a secret. Meanwhile, Jess and Tom get married in secret in Chile and Janice is trying to get them to have children. Minnie is going off the rails, and their fifth proposed house purchase has fallen through and, as it turns out, Jane is anxious for them to leave, so she can have her house back. Luke and Becky agree that Luke won't ",
 "Becky Bloomwood |  a new car and, seeing his willingness to sacrifice, Becky agrees to wear every single item in her wardrobe three times before shopping for anything new. Meanwhile, Becky is also trying to persuade Luke to have another baby, but he refuses because of Minnie's increasingly bad behavior. Any old reason is good enough for Becky to want another child; she sees some pom-pom hats and wants them to be known as the children in the pom-pom hats, much as she herself wanted to be known as The Girl in the Denny and George Scarf, or The Girl With the Angel Bag. Becky also ",
 "Becky Bloomwood |  estrangement from her own parents after an argument she has with them. One such argument involve a disagreement that she had with Graham and Jane while discussing Minnie's behaviour. They both suspect that Becky is being selfish in not accepting responsibility for her own actions that lead to Minnie's bad behavior, that of which she denies. Graham and Jane also thinks she's interfering with their lives by inviting herself when they want privacy with each other. After being fed up with Becky's irresponsibility and inability to deal with her problems, she, Graham and Jane have their first serious row and goes weeks without ",
 'Becky Bloomwood |  to each other, even when Jess has to battle Janice for trying to give Tom libido induced teas and other methods to make them have a child. Becky still has issues with Elinor whom wants to see Minnie, but soon she recognizes some similarities between them. She tries to warn Luke that they are behaving similar to his mother in keeping her away from Minnie. He staunchly denies it, admitting that he is furious with Elinor about a comment she made about Annabel before her death and the terrible mistreatment that she gave Becky over the years. She eventually reconciles with Elinor ',
 'Becky Bloomwood |  she insists on helping out with the party, which by now is in shambles. Elinor also finally accepts responsibility for her actions in selfishly abandoning Luke, and in a moment of extreme vulnerability, refers to Annabel as "his real mother". Becky asks Elinor for help in assessing Minnie\'s bad behavior and she agrees. Still worried about Minnie\'s wild behavior, Luke (without consulting Becky) hires Nanny Sue, a nanny with her own TV show, who goes into people\'s homes and they wind up "sobbing on her shoulder and asking, \'Nanny Sue, how can we be better people?\'" When Becky comes home and finds ',
 "Becky Bloomwood |  what he did, she is infuriated with Luke because he didn't consult with her first about it. She admits that she had been talking to Elinor about similar problems with Minnie. This angers Luke for he didn't want his mother to have anything to do with them after what she did in insulting both Becky and Annabel. Before a fight could break out, Nanny Sue calls them to their senses and makes a surprise assessment. She identifies both Luke's fears with having a second child and his issues with his mother, Elinor. Nanny Sue admits that she secretly went undercover to observe ",
 "Becky Bloomwood |  family's behavior at a mall they went to, even before she was called in to assess Minnie's behavior. Becky and Luke are both shocked by this confession since they didn't know that she had gone undercover beforehand. Luke is relieved when Nanny Sue tells him that she knew he had nothing to do with it because he tried hard not to spoil his daughter. She also admits she must confront Becky as a favor to Graham and Jane, revealing they're the ones who called her. She is nervous when Nanny Sue admitted that she had observed Becky's own behavior in a toy ",
 "Becky Bloomwood |  and suspected she was responsible for Minnie's bad behavior. Her buying designer made clothing, fighting with Minnie for an expensive toy pony that she wanted so bad and her refusal to let go of it, proved Nanny Sue's point. Becky defended herself by admitting she had originally refused to buy it for Minnie and angrily tried to remind her about financial trouble she and Luke are both in. While Nanny Sue agreed, she mentions it didn't excuse for the fact that when she caved in, it further supported her suspicion. She makes Becky see that in her spoiling Minnie too much: she ",
 "Becky Bloomwood |  a serious role in her bad behavior. Nanny Sue is the first person to ask Becky if she's considered the possibility she might have a shopping addiction. Becky tries to deny it, claiming she doesn't have an addiction much to Nanny Sue's amusement. However, Minnie's piped up comments of items do nothing to dissuade anyone of that notion and further humiliates her. Feeling betrayed by her own daughter, Becky finally admits that she does and had a hard time learning when to stop. She also realizes her own inability to take responsibility for her actions is what led to her relationship with ",
 'Becky Bloomwood |  parents being strained and how her spoiling contributed to Minnie\'s bad behavior are the real reasons why they\'ve had been banned from previous Christmas Grottos. Ironically, Becky realizes the woman she had been talking to at the mall was Nanny Sue with children of her own and how it was similar to what Elinor and even her own parents had told her that all Minnie really needs is discipline. After she agrees to go to "Shopaholics Boot Camp" and seek help, Becky and Luke can move forward with the idea of expanding their family...if only they can survive the "surprise" party, which ',
 "Becky Bloomwood |  two million people in London have found out about, which Becky did try to keep a secret. Now reformed, Becky visits Surrey to apologize to Graham and Jane for her earlier behavior. She realized they were originally right in their assessment, but was in denial because of wanting to give Luke a surprise birthday party. In finally accepting responsibility for herself, Becky reconciles with her parents. Having enough of Janice's behavior, Becky defends Jess and Tom's decision to adopt their first child by telling her off to leave them alone. She is fed up Janice's actions in trying to force them to ",
 'Becky Bloomwood |  a natural child first and that she should support their decision. The latter is offended with Becky for calling her out by demanding her to support her decision to want a natural child first. However, she refuses to back down and stands firm with Janice, bargaining with her to ease up if she allows Jess and Tom to adopt first before having a natural child. At the end of the book, Becky has become aware that Luke is doing business for Sage Seymour (a Julia Roberts-type movie star) and that he wants to take the whole family to Hollywood for three months.'
]

from sentence_transformers import SentenceTransformer
import json
import torch
device = torch.device("cuda:0")
embedder = SentenceTransformer("infgrad/stella_en_1.5B_v5", trust_remote_code=True).to(device)
# embedder = SentenceTransformer("BAAI/bge-large-en-v1.5").to(device)

## encode with batch size 32
batch_embedding = embedder.encode(all_text,convert_to_tensor=True)

## separately encoded
single_embedding = []
for text in all_text:
    embedding = embedder.encode(text,show_progress_bar=False,convert_to_tensor=True)
    single_embedding.append(embedding)
single_embedding = torch.stack(single_embedding,dim=0)

## compare
for atol in [1,1e-1,1e-2,1e-3,1e-4,1e-5,1e-6]:
    print(atol,torch.allclose(batch_embedding,single_embedding,atol=atol))

And it gives:

1 True
0.1 True
0.01 True
0.001 False
0.0001 False
1e-05 False
1e-06 False

It is even worse when applying {"torch_dtype":torch.float16}.

However, with other bert-like embedder such as BAAI/bge-large-en-v1.5, it works fine:

1 True
0.1 True
0.01 True
0.001 True
0.0001 True
1e-05 True
1e-06 True

So I want to know if you have encountered such problem? Do you think this would have a huge impact on the final performance? Looking forward to your insights!

DunZhang / Stella

Question about the batch encoding #1