peleiden / daluke

A Danish-speaking language model with entity-aware self-attention
MIT License
9 stars 0 forks source link

Make data loading faster #131

Open asgerius opened 3 years ago

asgerius commented 3 years ago

Right now it is very slow and takes up a significant chunk of the 24 h limit. Multiprocessing has been attempted, but with disappointing results due to the massive amounts of memory that had to be copied (I think). Instead try optimize the code itself