search

forPelevin / gomoji

Helpful functions to work with emoji in Golang
MIT License
190 stars 28 forks source link

Data size #20

Closed ivanjaros closed 1 year ago

ivanjaros commented 1 year ago

Is there any way the 1.25MB file with emojis. that is loaded into memory, could be made WAY smaller?

forPelevin commented 1 year ago

Hi @ivanjaros. I'm not sure how we can reduce the emoji's variable size. Do you have any ideas about that? And why is 1.25 MB critical for you?

ivanjaros commented 1 year ago

My current use case is that I want to filter out emojis and so having to keep over 1meg of data for it in memory is extreme. So I was wondering if there is a way the dataset can be compressed somehow? If not, no worries, I got recommendation for a solution in uniseg library that won't require entire dataset. I tried to do it manually by defining the unicode ranges but it was not working due to those multiple runes issue that you're using uniseg for.