AssertionError: Batch can only be done if at least 4 beams have been generated

[creatorrr]

Sorry for the barrage of bug reports, @asahi417 but ran into another one at inference time. Running model.generate_qa using the t5-base-squad-multitask model sometimes throws this assertion. Could this be a bug in the transformers lib?

Stacktrace:

----> 1 qas = [model.generate_qa(text) for text in texts]

File ~/src/github.com/julep-ai/monorepo/services/nlp-service/.venv/lib/python3.8/site-packages/lmqg/language_model.py:296, in TransformersQG.generate_qa(self, context, batch_size, num_beams, cache_path, answer_model, num_questions, sentence_level)
    275 def generate_qa(self,
    276                 context: str,
    277                 batch_size: int = None,
   (...)
    281                 num_questions: int = 10,
    282                 sentence_level: bool = False):
    283     """ Generate question given context.
    284
    285     @param context: Input text.
   (...)
    294     @return: List of generated sentences.
    295     """
--> 296     list_answer = self.generate_a(
    297         context, batch_size=batch_size, num_beams=num_beams, cache_path=cache_path,
    298         sentence_level=sentence_level, answer_model=answer_model, num_questions=num_questions
    299     )
    300     list_context = [context] * len(list_answer)
    301     logging.info('running model for `qg`')

<<<<<<<<<<<<<<CLIPPED>>>>>>>>>>>>>>>>

File ~/src/github.com/julep-ai/monorepo/services/nlp-service/.venv/lib/python3.8/site-packages/transformers/generation_beam_search.py:226, in BeamSearchScorer.process(self, input_ids, next_scores, next_tokens, next_indices, pad_token_id, eos_token_id)
    225 if self._done[batch_idx]:
--> 226     assert (
    227         len(beam_hyp) >= self.num_beams
    228     ), f"Batch can only be done if at least {self.num_beams} beams have been generated"
    229     assert (
    230         eos_token_id is not None and pad_token_id is not None
    231     ), "generated beams >= num_beams -> eos_token_id and pad_token have to be defined"

AssertionError: Batch can only be done if at least 4 beams have been generated

[asahi417]

I have never observed this error before, so it might be something to do with the transformers library... Would you be able to share the input that causes this error?

[creatorrr]

https://gist.github.com/creatorrr/03dfcf6a2b907a517ec8f58dc9f28be9

[creatorrr]

I think it may be happening for the really small inputs

[asahi417]

Thanks! When you define model object, do you put any options or you just use the default configuration?

[creatorrr]

Just exactly as in the example in the docs: model = TransformersQG(language='en', model='lmqg/t5-large-squad-multitask')

[asahi417]

Ok, I'll investigate it and hopefully add some error handling to the function!

[asahi417]

which version is your transformers?

[asahi417]

from lmqg import TransformersQG
model = TransformersQG(language='en', model='lmqg/t5-large-squad-multitask')
data = [
  "Admissions are open from Play Group to class IX & XI for Academic session 2022-2023 Minimum age for admission to Play Group is 2 years 6 months, class Nursery is 3 years 6 months and to class KG is 4 years 6 months at the beginning of the session (i.e. in April of the year). The School would inform parents about interview and selection process in due course of time. TRANSFER CERTIFICATES Students who are under transfer from schools (including outside U.P.) must have their Transfer Certi cates countersigned by the Inspector of Schools of the state concerned.",
  "For admission to Play Group, Nursery, KG, Level 1, parents are required to submit: A photocopy of the date of birth certi cate of their wards issued by a competent authority at the time of admission Five passport size photographs of the ward and ve each of the parents Requisite admission form For admission to other classes a certi cate from the head of the school where the student is presently studying, will have to be produced at the time of admission. The Original copy of the Birth Certi cate must be produced for veri cation when required. Application for admission should be given in the appropriate form. Children, along with their parents, will be interviewed, if necessary, by the Principal. Aadhar Card of the Ward and Parents",
  "Prospectus and Registration form can be obtained from the school o ce by paying the registration fee and the completed Registration form need to be deposited in the School o ce within a speci c time. The said registration fee shall be non-refundable. *** Special o er in Admission Fee, For more details, Please contact School o ce RULES FOR ADMISSION The application for admission must be given on the appropriate form. The school reserves the right to evaluate the child in any or all subjects. Parents are requested to give an advance notice of at least a month for the withdrawal of their wards. Otherwise, a fee equivalent to the tuition fee of one month shall be charged for issuing the transfer certi cate. Annual Examination are held in March. Cumulative assessment of the whole year is considered for the child’s promotion into the next class. A student who fails twice in the same class will have to be withdrawn from the School. PAYMENT OF FEE The fee is to be paid in the School O ce maximum in twelve installments before the end of the academic session. Fee once paid is not refundable. Payment of fee by Cash / DD / Cheque shall be th accepted up to the 10 of every installment in which payments falls due. Cheques / DD should be drawn in favour of Seth M.R. Jaipuria School, Hardoi – REVENUE A/C, payable at Hardoi. In case a cheque is dishonoured, a ne of 100 will have to be paid along with the fee and bank charges as applicable. After the end of the month, a ne of 50 will be charged per week or part thereof for late payment of fee. If the fee remains unpaid for two consecutive months, the name of the student will be struck o the rolls. The fact that a memo or bill has not been received by a parent does not exempt the student from paying fee nor prevents the school from taking necessary action against the said student for non- payment of fee.",
  "If any student whose name is struck o the rolls seeks re- admission, he/she will be charged a re-admission fee equivalent to the Admission fee of that particular academic session, provided re- admission is permissible and sanctioned by the Principal. If any collection charges are levied, the same would be payable by the parents / guardians. be",
  "ENGLISH Science, exposed latest",
  "that Fee Will be accepted only through Cashless Channels in favour of “Seth M. R. Jaipuria School Hardoi – Revenue A/C”. Till 10th of the Fee Due Month through ERP, Online Cashless Modes, Cheque, School fee Counter and directly in Axis Bank Hardoi A/C. Fee Cheque can be deposited in School accounts along with duly lled fee deposit slip at the SCHOOL PREMISES during working hours of working days, till 10th of the due month. With Late FEE at the School counter: Fee with late ne and other applicable dues as described in fee receipt book and/or student’s dairy will be accepted at the School counter, between 9. AM to 02.PM. Please ll in all the information in the Fee Receipt Slip. On the back of the slip please ll up Cheque Number, Date, Your Bank Name & Branch. Please ll the Name, Class and Admission Number of student(s) and your contact number on the back of the Cheque too. Please use MICR Cheque or Demand Drafts only. Out station Cheque will not be accepted . If fee is deposited by Non MICR or out station Cheque, it may result in non deposit of fee, which will cause inconvenience to parents and imposition of ne. Cheques should be CTS only. In case of holidays on due date of fee collection, the fee will be accepted on the next working day. In case the fee is deposited through cheque, parents have to ensure that the cheque is issued by them towards the fee deposit, is cleared from their bank. In any case school will not be responsible for the loss of cheque in transit, return of cheque or late clearing of the cheque. Please ensure that there are adequate funds in your account to cover the cheque issued by you. Cheque not honoured for any reason can lead to serious re-percussion. Transport Fee will be extra and taking School Transport is optional for parents. Parents can send their child on their own as well. School Fee has to be paid in instalments as per the schedule mentioned. Fee paid is not refundable under any circumstances. Payment gateway charges as applicable. Registration fee and Admission fee once deposited, will not be refundable under any circumstances.",
  "Class Composite Fee (Monthly) Play Group, LKG & UKG 2920 Class I -II 3600 Class III-V 3660 Class 6-8 4150 Class IX-X 4950 With computer With P.E. Class XI-XII (Science Stream) 7310 7090 Class XI-XII (Commerce Stream) 6980 6710 (Commerce) Monthly * Transport Fee will be extra and taking School Transport is optional for parents. Parents can send there child on there own as well. * Payment will be made through Cash/Cheque/Demand Draft in favor of Seth M. R. Jaipuria School, Revenue Account, or Fee can be deposited directly in Axis Bank Acocunt. * School Fees has to be paid in 6 equal monthly installments as per the schedule mentioned above.",
  "* Any payment through IMPS/NEFT/RTGS are not allowed."
]
for i in data:
    model.generate_qa(i)

I tested the above command and I didn't get any error, so I wonder it was due to the transformers issue that has been sorted in recent version.