Best-of-n Pipeline: `ArrowInvalid: offset overflow while concatenating arrays`

[RylanSchaeffer]

When I try running python run_bon_pipeline.py, I hit the error:

Repo card metadata block was not found. Setting CardData to empty.
Starting BoN pipeline...
^MMap:   0%|          | 0/1000 [00:00<?, ? examples/s]^MMap:   1%|          | 8/1000 [00:00<00:14, 69.47 examples/s]^MMap:   2%|▏         | 18/1000 [00:00<00:12, 78.27 examples/s]^MMap:   3%|▎         | 28/1000 [00:00<00:11, 83.84 examples/s]^MMap:   4%|▍         | 38/1000 [00:00<00:11, 86.36 examples/s]^MMap:   5%|▍         | 48/1000 [00:00<00:11, 85.35 examples/s]^MMap:   6%|▌         | 57/1000 [00:00<00:11, 83.34 examples/s]^MMap:   7%|▋         | 67/1000 [00:00<00:11, 83.48 examples/s]^MMap:   8%|▊         | 76/1000 [00:00<00:11, 82.19 examples/s]^MMap:   9%|▉         | 88/1000 [00:01<00:11, 78.53 examples/s]^MMap:  10%|▉         | 97/1000 [00:01<00:11, 79.95 examples/s]^MMap:  11%|█         | 106/1000 [00:01<00:11, 80.49 examples/s]^MMap:  12%|█▏        | 116/1000 [00:01<00:10, 80.83 examples/s]^MMap:  13%|█▎        | 126/1000 [00:01<00:10, 81.81 examples/s]^MMap:  14%|█▎        | 135/1000 [00:01<00:10, 82.72 examples/s]^MMap:  14%|█▍        | 144/1000 [00:01<00:10, 82.96 examples/s]^MMap:  15%|█▌        | 153/1000 [00:01<00:10, 80.36 examples/s]^MMap:  16%|█▌        | 162/1000 [00:01<00:10, 80.07 examples/s]^MMap:  17%|█▋        | 171/1000 [00:02<00:10, 81.87 examples/s]^MMap:  18%|█▊        | 181/1000 [00:02<00:09, 83.20 examples/s]^MMap:  19%|█▉        | 193/1000 [00:02<00:10, 80.64 examples/s]^MMap:  20%|██        | 203/1000 [00:02<00:11, 69.99 examples/s]^MMap:  21%|██▏       | 213/1000 [00:02<00:10, 73.75 examples/s]^MMap:  22%|██▏       | 223/1000 [00:02<00:09, 78.23 examples/s]^MMap:  23%|██▎       | 232/1000 [00:02<00:09, 77.72 examples/s]^MMap:  24%|██▍       | 242/1000 [00:03<00:09, 79.30 examples/s]^MMap:  25%|██▌       | 251/1000 [00:03<00:09, 79.62 examples/s]^MMap:  26%|██▌       | 260/1000 [00:03<00:09, 79.87 examples/s]^MMap:  27%|██▋       | 270/1000 [00:03<00:08, 81.76 examples/s]^MMap:  28%|██▊       | 279/1000 [00:03<00:08, 81.28 examples/s]^MMap:  29%|██▉       | 289/1000 [00:03<00:08, 82.73 examples/s]^MMap:  30%|██▉       | 299/1000 [00:03<00:08, 82.87 examples/s]^MMap:  31%|███       | 309/1000 [00:03<00:08, 78.12 examples/s]^MMap:  32%|███▏      | 317/1000 [00:03<00:08, 76.57 examples/s]^MMap:  32%|███▎      | 325/1000 [00:04<00:09, 73.99 examples/s]^MMap:  34%|███▎      | 335/1000 [00:04<00:08, 76.72 examples/s]^MMap:  34%|███▍      | 345/1000 [00:04<00:08, 79.50 examples/s]^MMap:  35%|███▌      | 353/1000 [00:04<00:08, 78.71 examples/s]^MMap:  36%|███▌      | 362/1000 [00:04<00:08, 78.30 examples/s]^MMap:  37%|███▋      | 371/1000 [00:04<00:08, 78.25 examples/s]^MMap:  38%|███▊      | 381/1000 [00:04<00:07, 80.05 examples/s]^MMap:  39%|███▉      | 390/1000 [00:04<00:07, 81.32 examples/s]^MMap:  40%|███▉      | 399/1000 [00:04<00:07, 81.84 examples/s]^MMap:  41%|████      | 409/1000 [00:05<00:07, 82.50 examples/s]^MMap:  42%|████▏     | 421/1000 [00:05<00:07, 78.33 examples/s]^MMap:  43%|████▎     | 432/1000 [00:05<00:08, 70.89 examples/s]^MMap:  44%|████▍     | 442/1000 [00:05<00:07, 74.34 examples/s]^MMap:  45%|████▌     | 450/1000 [00:05<00:07, 73.97 examples/s]^MMap:  46%|████▌     | 458/1000 [00:05<00:07, 74.02 examples/s]^MMap:  47%|████▋     | 467/1000 [00:05<00:07, 74.36 examples/s]^MMap:  48%|████▊     | 476/1000 [00:06<00:06, 75.19 examples/s]^MMap:  48%|████▊     | 485/1000 [00:06<00:06, 76.44 examples/s]^MMap:  49%|████▉     | 493/1000 [00:06<00:06, 76.91 examples/s]^MMap:  50%|█████     | 502/1000 [00:06<00:06, 78.01 examples/s]^MMap:  51%|█████     | 511/1000 [00:06<00:06, 78.09 examples/s]^MMap:  52%|█████▏    | 520/1000 [00:06<00:06, 79.59 examples/s]^MMap:  53%|█████▎    | 529/1000 [00:06<00:05, 80.30 examples/s]^MMap:  54%|█████▍    | 538/1000 [00:06<00:05, 80.27 examples/s]^MMap:  55%|█████▍    | 548/1000 [00:06<00:06, 73.84 examples/s]^MMap:  56%|█████▌    | 557/1000 [00:07<00:05, 76.30 examples/s]^MMap:  57%|█████▋    | 567/1000 [00:07<00:05, 79.48 examples/s]^MMap:  58%|█████▊    | 577/1000 [00:07<00:05, 82.61 examples/s]^MMap:  59%|█████▊    | 587/1000 [00:07<00:04, 83.78 examples/s]^MMap:  60%|██████    | 600/1000 [00:07<00:04, 80.64 examples/s]^MMap:  61%|██████    | 610/1000 [00:07<00:04, 81.52 examples/s]^MMap:  62%|██████▏   | 620/1000 [00:07<00:04, 84.73 examples/s]^MMap:  63%|██████▎   | 629/1000 [00:07<00:04, 83.72 examples/s]^MMap:  64%|██████▍   | 638/1000 [00:08<00:04, 83.17 examples/s]^MMap:  65%|██████▍   | 647/1000 [00:08<00:04, 81.88 examples/s]^MMap:  66%|██████▌   | 659/1000 [00:08<00:04, 76.22 examples/s]^MMap:  67%|██████▋   | 668/1000 [00:08<00:04, 77.11 examples/s]^MMap:  68%|██████▊   | 678/1000 [00:08<00:04, 79.19 examples/s]^MMap:  69%|██████▉   | 688/1000 [00:08<00:03, 80.08 examples/s]^MMap:  70%|██████▉   | 697/1000 [00:08<00:03, 79.63 examples/s]^MMap:  71%|███████   | 710/1000 [00:08<00:03, 80.69 examples/s]^MMap:  72%|███████▏  | 719/1000 [00:09<00:03, 80.40 examples/s]^MMap:  73%|███████▎  | 729/1000 [00:09<00:03, 82.36 examples/s]^MMap:  74%|███████▍  | 741/1000 [00:09<00:03, 77.91 examples/s]^MMap:  75%|███████▌  | 750/1000 [00:09<00:03, 78.01 examples/s]^MMap:  76%|███████▌  | 758/1000 [00:09<00:03, 76.68 examples/s]^MMap:  77%|███████▋  | 767/1000 [00:09<00:03, 77.47 examples/s]^MMap:  78%|███████▊  | 777/1000 [00:09<00:03, 72.65 examples/s]^MMap:  79%|███████▊  | 787/1000 [00:09<00:02, 75.78 examples/s]^MMap:  80%|███████▉  | 796/1000 [00:10<00:02, 77.29 examples/s]^MMap:  80%|████████  | 805/1000 [00:10<00:02, 79.25 examples/s]^MMap:  82%|████████▏ | 817/1000 [00:10<00:02, 77.04 examples/s]^MMap:  83%|████████▎ | 826/1000 [00:10<00:02, 78.10 examples/s]^MMap:  84%|████████▎ | 835/1000 [00:10<00:02, 78.14 examples/s]^MMap:  84%|████████▍ | 843/1000 [00:10<00:02, 76.96 examples/s]^MMap:  85%|████████▌ | 852/1000 [00:10<00:01, 78.21 examples/s]^MMap:  86%|████████▌ | 861/1000 [00:10<00:01, 79.11 examples/s]^MMap:  87%|████████▋ | 870/1000 [00:11<00:01, 80.54 examples/s]^MMap:  88%|████████▊ | 879/1000 [00:11<00:01, 79.51 examples/s]^MMap:  89%|████████▉ | 889/1000 [00:11<00:01, 73.09 examples/s]^MMap:  90%|████████▉ | 898/1000 [00:11<00:01, 75.38 examples/s]^MMap:  91%|█████████ | 908/1000 [00:11<00:01, 78.48 examples/s]^MMap:  92%|█████████▏| 917/1000 [00:11<00:01, 79.77 examples/s]^MMap:  93%|█████████▎| 926/1000 [00:11<00:00, 80.22 examples/s]^MMap:  94%|█████████▎| 936/1000 [00:11<00:00, 82.81 examples/s]^MMap:  94%|█████████▍| 945/1000 [00:11<00:00, 81.53 examples/s]^MMap:  96%|█████████▌| 955/1000 [00:12<00:00, 80.72 examples/s]^MMap:  96%|█████████▋| 965/1000 [00:12<00:00, 81.09 examples/s]^MMap:  98%|█████████▊| 975/1000 [00:12<00:00, 83.29 examples/s]^MMap:  98%|█████████▊| 984/1000 [00:12<00:00, 83.13 examples/s]^MMap:  99%|█████████▉| 994/1000 [00:12<00:00, 84.24 examples/s]^MMap: 100%|█████████▉| 999/1000 [00:32<00:00, 84.24 examples/s]^MMap: 100%|█████████▉| 999/1000 [00:32<00:00, 30.71 examples/s]
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /lfs/ampere8/0/rschaef/miniconda3/envs/reward_modeling_env/lib/python3.10/site-packages/datasets │
│ /arrow_dataset.py:3538 in _map_single                                                            │
│                                                                                                  │
│   3535 │   │   │   │   │   │   │   ):                                                            │
│   3536 │   │   │   │   │   │   │   │   writer.write_row(example.to_arrow())                      │
│   3537 │   │   │   │   │   │   │   else:                                                         │
│ ❱ 3538 │   │   │   │   │   │   │   │   writer.write(example)                                     │
│   3539 │   │   │   │   │   │   num_examples_progress_update += 1                                 │
│   3540 │   │   │   │   │   │   if time.time() > _time + config.PBAR_REFRESH_TIME_INTERVAL:       │
│   3541 │   │   │   │   │   │   │   _time = time.time()                                           │
│                                                                                                  │

< lots of text > 

│ /lfs/ampere8/0/rschaef/miniconda3/envs/reward_modeling_env/lib/python3.10/site-packages/datasets │
│ /arrow_writer.py:584 in write_table                                                              │
│                                                                                                  │
│   581 │   │   │   writer_batch_size = self.writer_batch_size                                     │
│   582 │   │   if self.pa_writer is None:                                                         │
│   583 │   │   │   self._build_writer(inferred_schema=pa_table.schema)                            │
│ ❱ 584 │   │   pa_table = pa_table.combine_chunks()                                               │
│   585 │   │   pa_table = table_cast(pa_table, self._schema)                                      │
│   586 │   │   if self.embed_local_files:                                                         │
│   587 │   │   │   pa_table = embed_table_storage(pa_table)                                       │
│                                                                                                  │

< lots of text>

│ │                     essential for the well-being of people and the planet. It is essential   │ │
│ │                     for maintaining ecosystems, regulating climate, and providing clean air  │ │
│ │                     and water. It is also essential for preserving unique species and        │ │
│ │                     habitats. Conservation of biodiversity is key to preserving biological   │ │
│ │                     diversity, which is essential for the evolution of new species and the   │ │
│ │                     survival of existing species.",...,"Conservation of biodiversity is      │ │
│ │                     essential for preserving and restoring the planet's ecosystems, which    │ │
│ │                     are essential for maintaining healthy ecosystems and helping to prevent  │ │
│ │                     the spread of disease. The loss of biodiversity is increasing due to     │ │
│ │                     habitat loss, pollution, and over-harvesting.","The importance of        │ │
│ │                     conservation of biodiversity is recognized worldwide as it plays an      │ │
│ │                     essential role in maintaining the stability of ecosystems and the        │ │
│ │                     balance of ecosystems services. Biodiversity is an important source of   │ │
│ │                     new ideas and innovation in conservation strategies, which can also help │ │
│ │                     to reduce poverty and improve human health. Biodiversity also helps to   │ │
│ │                     maintain existing ecosystems and their services, such as flood           │ │
│ │                     protection, soil conservation, and nutrient cycling. The loss of         │ │
│ │                     biodiversity can cause drastic changes in the ecosystem, leading to      │ │
│ │                     serious consequences such as ecosystem breakdown, economic losses, and   │ │
│ │                     reduced resilience to future events.","Conservation of biodiversity is   │ │
│ │                     essential for maintaining the balance of the planet's ecosystems and     │ │
│ │                     maintaining the health of its species. Conservation efforts are          │ │
│ │                     important in order to maintain the existing biodiversity, while also     │ │
│ │                     preserving the genetic and physical diversity of the species.            │ │
│ │                     Additionally, biodiversity is important for maintaining traditional ways │ │
│ │                     of life, providing food and water, and contributing to the economy.","   │ │
│ │                     Conservation of biodiversity is important for preserving the natural     │ │
│ │                     wealth of the planet. It involves protecting areas, plants, animals, and │ │
│ │                     habitats that are home to many species, as well as managing and          │ │
│ │                     protecting existing ecosystems to ensure the resilience of their habitat │ │
│ │                     and wildlife. Additionally, it encourages the sustainable use of natural │ │
│ │                     resources, such as water and air, and reduces the risk of species        │ │
│ │                     extinction."," Conservation of biodiversity is an important element of   │ │
│ │                     sustainable development because it helps protect habitats, ecosystems    │ │
│ │                     and native species from destruction. It is also essential for            │ │
│ │                     maintaining ecosystem services, such as providing food, water and air,   │ │
│ │                     and acting as a check against climate change. It is an important         │ │
│ │                     principle for sustainable management of resources and can help ensure    │ │
│ │                     the long-term survival of human cultures, too."],["When I was in         │ │
│ │                     college, I was struggling to make decisions and find motivation. I       │ │
│ │                     realized that I needed to take a step back and remind myself that there  │ │
│ │                     was no right answer. It was important to trust my gut instinct and not   │ │
│ │                     let external pressure get in the way of achieving my goals. Taking a     │ │
│ │                     step back allowed me to refocus my energy and regain the focus I needed  │ │
│ │                     to make the changes I wanted to make.","When I was preparing for an      │ │
│ │                     exam, I faced a lot of obstacles, but I was determined to prove that I   │ │
│ │                     was capable of achieving success. I went to study sessions regularly,    │ │
│ │                     studied hard, and tried to focus on the subject matter. Through all of   │ │
│ │                     this, I managed to pass the exam and prove to myself that I could do     │ │
│ │                     anything I put my mind to.","I overcame difficulties when I found a way  │ │
│ │                     to make the most of a difficult situation by thinking outside of the     │ │
│ │                     box. By finding a way to use my skills and knowledge to solve a problem, │ │
│ │                     I was able to come up with a creative solution that addressed the issue. │ │
│ │                     Through this, I was able to focus on the positive and come out on the    │ │
│ │                     other side with newfound confidence.","When I was in college, I had to   │ │
│ │                     take an independent study course on the history of African-American      │ │
│ │                     civil rights movement. I worked hard to make it through the course and   │ │
│ │                     ultimately passed the exam with high marks. I also had to take an online │ │
│ │                     class in a different course and it was a challenge, but I was determined │ │
│ │                     to make it through. Ultimately, I made it through both classes and, as a │ │
│ │                     result, I was awarded the highest grade in the class. It was a testament │ │
│ │                     to my hard work and dedication to finish the course.","When I was a      │ │
│ │                     freshman in college, I was very uncertain about what to do with my life. │ │
│ │                     Despite struggling with making decisions, I found a way to solve my      │ │
│ │                     problem by pursuing a passion I had been hinting at in my mind: art.     │ │
│ │                     After a few months of creating artwork and talking to artists, I felt    │ │
│ │                     more confident in my decision and I decided to pursue a degree in art.   │ │
│ │                     This decision meant more than anything as it allowed me to become an     │ │
│ │                     asset to the art community and to explore new opportunities.",...,"I     │ │
│ │                     overcame difficulties by learning to separate my responsibilities from   │ │
│ │                     my emotions and staying optimistic.","I overcame difficulties by         │ │
│ │                     learning to rely on my intuition and following my gut instinct. This     │ │
│ │                     helped me to gain strength of character, focus on the task at hand, and  │ │
│ │                     cope with failure. By taking the time to think things through, I was     │ │
│ │                     able to work through my doubts and come up with a plan that worked for   │ │
│ │                     me. I also used visualization techniques to help me think about a        │ │
│ │                     solution and stay motivated. Finally, I worked hard and persisted        │ │
│ │                     through the inevitable setbacks to reach my goal and became a better     │ │
│ │                     version of myself in the process.","I overcame my difficulties when I    │ │
│ │                     faced adversity and worked hard. I kept pushing through every obstacle   │ │
│ │                     that came my way and eventually found a solution that worked for me.     │ │
│ │                     Through hard work and perseverance, I overcame the difficulty and was    │ │
│ │                     eventually able to find a way out.","I overcame difficulties by learning │ │
│ │                     to control my emotions and realize the importance of keeping my faith in │ │
│ │                     spite of setbacks. I knew that I was capable of overcoming these         │ │
│ │                     obstacles and reached out for help when I needed it. I started believing │ │
│ │                     in myself again and slowly but surely I started to make progress.","I    │ │
│ │                     overcame difficulties when I decided to give up negative thoughts and    │ │
│ │                     embrace positive beliefs, instead. Positive thoughts are like little oil │ │
│ │                     pumps, constantly pushing out positive energy and encouraging me to move │ │
│ │                     forward. I started exercising, got organized and started celebrating the │ │
│ │                     small things in life. I learned to trust myself and what I could         │ │
│ │                     accomplish, instead of relying on other people. Ultimately, I was able   │ │
│ │                     to face challenges and obstacles head on with confidence and             │ │
│ │                     determination."]]]                                                       │ │
│ │                     gold_scores:                                                             │ │
│ │                     [[[0.83203125,0.6328125,1.28125,0.01153564453125,0.953125,...,0.2285156… │ │
│ │              self = <datasets.arrow_writer.ArrowWriter object at 0x7fcdb0645660>             │ │
│ │ writer_batch_size = 1000                                                                     │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ in pyarrow.lib.Table.combine_chunks:4289                                                         │
│                                                                                                  │
│ in pyarrow.lib.pyarrow_internal_check_status:154                                                 │
│                                                                                                  │
│ in pyarrow.lib.check_status:91                                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ArrowInvalid: offset overflow while concatenating arrays

Is this to be expected? If not, what is the correct solutions?

[RylanSchaeffer]

One solution I found: Explicitly use a batch size for .map():

gold_labelled_generations = gold_labelled_generations.map(_truncate_answers, batched=True, batch_size=10)

[tlc4418]

Hmm interesting, I don't remember having seen this error, but maybe I wasn't using map before. Thank you for sharing, maybe this will help other people as well.