search

axa-group / nlp.js

An NLP library for building bots, with entity extraction, sentiment analysis, automatic language identify, and so more
MIT License
6.27k stars 620 forks source link

Reduce on-disk model NLP size by compressing it #1165

Open LiamKarlMitchell opened 2 years ago

LiamKarlMitchell commented 2 years ago

Is your feature request related to a problem? Please describe. My model nlp file is growing quite large over time.

Describe the solution you'd like Would be great to compress it.

Describe alternatives you've considered Yeah I can and probably will do this manually with the import and export features.

But could be a good feature to have? Just thought I would open this to ask if anyone has considered a way to do this in the NLPManager? Or perhaps there is a way to do this already built-in that I am not aware of, did look and couldn't seem to find one in docs.

Perhaps an option of compressed defaulting to false.

https://github.com/axa-group/nlp.js/blob/master/docs/v3/nlp-manager.md#importexport-using-json

Apollon77 commented 2 years ago

I would propose to simply detect it by the provided filename. When ends with ".gz" then store it using gz compression. else not. Could be implememented rather easy