search

mrdbourke / tensorflow-deep-learning

All course materials for the Zero to Mastery Deep Learning with TensorFlow course.
https://dbourke.link/ZTMTFcourse
MIT License
5.05k stars 2.5k forks source link

MemoryError: Unable to allocate 29.3 GiB for an array with shape (2211861,) and data type <U3551 #606

Open rohankarande2023 opened 8 months ago

rohankarande2023 commented 8 months ago

Getting an error while creating a Character Level Tokenizer for PubMed_200k_RCT_numbers_replaced_with_at_sign NLP project.

Create Character Level Tokenizer:

char_vectorizer=tf.keras.layers.TextVectorization(max_tokens=Num_Char_Tokens,output_sequence_length=char_per_sentence, name='char_vectorizer')

Adapt character vectorizer to training characters

char_vectorizer.adapt(train_chars)

memoryError2 memoryError