infinitered / nsfwjs

NSFW detection on the client-side via TensorFlow.js
https://nsfwjs.com/
MIT License
8.06k stars 540 forks source link

Allowing for pre gzip the 6 model files #93

Closed bongobongo closed 1 year ago

bongobongo commented 5 years ago

The 6 model files are quite large. If the nsfwjs are to be used by many clients, there might be others out there who see the need to pre compress the model files - and just serve the pre compressed files to the client. This way saving bandwidth and CPU usage on the server - as well as the user has quicker access to the web page. Found a solution on how to serve pre compressed files from Apache here: https://gist.github.com/bhollis/2200790 So there obviously is no need to change anything in model.json file to make this work, since it all can be solved in a .htaccess file. Originally this was formed as a question, since I believed there had to be some changes to the model.json file to make this work. But there is no need for that. I'm not an expert in Apache and code in .htaccess - but the information below has been tested on a live server (web hotel), and does work (you just have to get the new lines correct though - they got messed up by some reason when displayed here).

Here is how I did it:

First I compressed the 6 files in the model directory (the files starting with "group1-", using 7-zip. So the original file named group1-shard1of6 ended up with this filename: group1-shard1of6.gz After compressing the 6 files, deleted the 6 corresponding uncompressed files, and added a .htaccess file (in same directory where I have the model files) with the content below (new lines is removed by the editor here, so they have to be fixed if using this code):

Options +SymLinksIfOwnerMatch RewriteEngine On AddEncoding gzip .gz ExpiresActive On ExpiresDefault "access plus 2 months" RewriteCond %{HTTP:Accept-encoding} gzip RewriteCond %{REQUEST_FILENAME} !-f RewriteRule ^(.*)1of6 group1-shard1of6.gz [QSA,L] RewriteRule ^(.*)2of6 group1-shard2of6.gz [QSA,L] RewriteRule ^(.*)3of6 group1-shard3of6.gz [QSA,L] RewriteRule ^(.*)4of6 group1-shard4of6.gz [QSA,L] RewriteRule ^(.*)5of6 group1-shard5of6.gz [QSA,L] RewriteRule ^(.*)6of6 group1-shard6of6.gz [QSA,L] <Files *.gz> ForceType text/javascript </Files>


With and without gzip compression for the first model file: group1-shard1of6 = 4096 KB group1-shard1of6.gz = 2698 KB

Just by compressing the 5 largest files the client will "save" approximately 5 * 1400 KB = 7000 KB in the download process.

kevinvangelder commented 5 years ago

There is a newer version of the model that is substantially smaller while still maintaining 90% accuracy which might be useful for you.

cdanwards commented 1 year ago

@bongobongo I'm going to close this issue for now as it's been a while since a possible solution was provided. We'll gladly reopen this issue if you'd like us to. Thanks!