MemoryError: Unable to allocate 29.3 GiB for an array with shape (2211861,) and data type <U3551

[rohankarande2023]

Getting an error while creating a Character Level Tokenizer for PubMed_200k_RCT_numbers_replaced_with_at_sign NLP project.

Create Character Level Tokenizer:

char_vectorizer=tf.keras.layers.TextVectorization(max_tokens=Num_Char_Tokens,output_sequence_length=char_per_sentence, name='char_vectorizer')

Adapt character vectorizer to training characters

char_vectorizer.adapt(train_chars)