thi-ng / umbrella

⛱ Broadly scoped ecosystem & mono-repository of 199 TypeScript projects (and ~180 examples) for general purpose, functional, data driven development
https://thi.ng
Apache License 2.0
3.39k stars 150 forks source link

[Mime] But smaller ;) #339

Closed TheThing closed 2 years ago

TheThing commented 2 years ago

So I saw your thi-ng/mime which is an awesome reduced set of the infamous huge mime database and was very happy about it. However I still felt we could do better so I took the "generated.ts" file and uhmm... compressed it and managed to reduce its final size in half :)

Original: 16.3KB Compressed: 5.52KB

I did this for my package Flaska which was inspired by Bottle.py as a single-file module http package. I wanted mime-type support in it and including a big database of 17KB would have cause quite an impact but this way, the impact was kept smaller.

I figured I'd throw this out there in case you're interested in cutting the mime database size in half ;)

Here's the entire code and database in compressed format (not browser compatible so that might be a dealbreaker):

import zlib from 'zlib'

let MimeTypeDbRaw = Buffer.from('G6osAJwFdhv1SrFZSJz4SQJuhCSzWMqRl/mTge/er7L+ds/rnr9Ub3M7IqCSiNJgWSapw9Lny+zVadeap7R13N82CgFnYjgIkgm2/2VqmTYwWDc4sg67Pj7jIle5rE1SBdH2e7+nNdMDotgDEIXBACphiDVYPyB5xgHkVQm7ciQjGR8p96lz4V2kJFSQKMmlc4gBmN/0zrdU7/tjWD0Q2mmt9Tb8xGpRkQRvnbvDSWwSfx89zvTN0i05wS9K7Ob+lplWCNdk9vaWp07SuLGKPwQxo+RdJ/ItamTGttQLQt5dU6ZrQl1FDHxzE5GM+r7HEsC/lZKoKyG3kg9GLt9ojBgVecsLQ0F5Byb5/J5WeMdqkg3h2jw14aRKFpufk1G2jwad7Nx5Tah/zZ0cnNrtbSeFwG5LNlSLL7fKBensC1w/dI1Hwac7PgvTN1pMJg0a5Yfpp4xNrlRSITiAgaVUpJnEU7Zvm30WA//N9ghvLxkkCbTchVDTZFpopID90XE6Txp5El0lhftbgedJkp0qmh8csHXboW2/8xDuzCXQVHtaP83Qqu3nSLNw0rMV8z6Wfp8D0g4YeLssl01oVXeaJIa3Ue6whPe+TSINJi5UHAn8D8GzSC2vFNd8P3i4tZQoE4DKwiJMoTD86Dehg3QsCvN1pBEc1tgtc0QKWDHMI5PoPlAUlJOct7SvP4n6uxmmXuHHzKYhmg+9aE8kfzDldKNo/uOLA0wJP1W+5wqYXU1wzGz5X2JFvxvQEzQx7QQCYLid+iZkluJLCjxT+EV9N9Tn7FIQC8tCi+PqNabXnxWDEwt71Bm4PC5AKhfPiLqE64wcgofL7fJrvMocOqD5lfiZQT/GM4niWT9VGgoaHrghB+r7+5F7zoj+C+HYltV0r89sxPLhz92NMtCT85HkPyCSnCA1NM9QmIgBEnPi2S/Y40D6bwCdbYCbgnjU/gTvBGoVxAGkByQ6RA9oFGgMaDSkM+c0KYOERNUdDsP4YIu8ADe07MuRDsueauwqQYfB93T48+S17V1Bukv0bbh83A/h23sYcbti/w8+nEalfotrZw3rFg8zPvpFRRyXumWUQan+aBnMXedzeIqctTKKkOPQqujrEAoN3FLU+KCLCULPs4far5+QZuIqf1mlJXjPd4N9K8t+sPDbxkv+Ie4n/MGyGaZPRp6+sw5oQD1t7736da+aoHIwJlLCLLQNHFi7baNIwX9tmBf2p0EmZjAm637TU05dlwTd9jftHa/voRlevbivUCYRXh+HVt+e8AkA7lIWRTfEuJznowoU8TLJ+J4qm3spgKocz9NlyzS7SjXGcc0xmvzRjXudSfJcfYvduBlXfqI+DmcF8BqDRYYu7XA33meUxziahcX9v+EVt3vhUqH76+FhnqYRJU1vy0VlKzun4kMISrXwqRL6YVjdyplJdz2RSkckge7GNRgXV/fAPIxfIao50p00/SxOFJh0yy604WmkA+rwcV9cuhkPWdyw2HOu3Xf4ksGsZmLQaaSWqqT7i+oDcO26UpECNpG00q2QhZuqr2Upci0Y0c+ZzK/KmgDyceUXN5XhaDrvm535sQYLl4ZQhCho9J3HRPVHEbG+mGT/IS1o568S6bzL1zo/ueAZAyva53gY926d8oiPe7lQlLlqikzg8gwjpkMSpKozZ9+cs3um4zMSgLzrV6eqaqJzwPBFUYC6Z7o3T6KsHt//unOjS/fSHJ0Wz4vD6MHPEl1Xy9w5tXQ85oRffjz+GV1GkfHLElBjcBN6+VU+hliUhPSif95lsGG/dg1Woq4PklpSvCBJxfDAjxNtEBnGURLLA/9G94W6pK7eg5B7CZx4VTWYt6apHtzNsFvDmtS05ivfwtTmqMP9dcLMYWawsTDQ7ESdzJuM3/DG3L93RBPTujGypqyQRlm1nCNHW4Vkpt/x/VHMV2icfH/iLVJcET7/4UurTfdo9f2Ed7P38kJ8IXZDDW9BUz/S6jap8xvxQXZp4KyrOEyCEqx7dlWV9OA7hUVjg62Je4t8RS6wRrqxdADLiocr560OzeBsicckUTE8CMe+fy9lv38/X7aN1ayrg6wRpFL1LSB9o8dgktadPff1W0xidxPHDoL04Hr/lG36pxg9YKGASgPm6yBgHxec+APX2ILnFMYjmz8XfAQ2TDj0RNJEl20KeO0Q9z7najUpYU5an0G6aAFpckQ1VSlC3CVuX5AGPsA0m2ZfRatJzWuv9K+EIzXmNgsjeEy23NMmzLyKv0BKr09V2cpJUBjEsNRhI5vePo++J0O7YE3sCwYrvXGuh5ocsmGAcUtQeO4hDkVOvUBvanETJxGJY4ouvA66yESH4aZMEKLaj2N4DdV+1nVE5FLEJZN1R9+K0whdm+FU84CuDGPJJrI1g4uqj4Z73tn8KrLNVSbcV9+fCl5PLFtyC4Q2DJfUiyAZ+Eip1ezDJdi0vuZsMd2nWmSG3YdiGP1To8NW6X4ZTP68wHKA5QbIzReekvatgE+xG5Y2qqyjc2H1BxSUtaBFYVbbo26CIhZdhfFuUSIpvB0ubRhnngDi3SyOQBJzNXtAEflj00dkq4NwwE3V4iE2aI6IUcT2Rv//589PLgvOSfQlIwwr/2KJYcrKzO69Dr48g2cgFybaXMx2wvJBYf+gaC03QQv6llM4PT2pvq3fVHnHvG45qMUCreYijd/DVLOakJ5wIh8BikgxQjHCe+Ztk5qWV7BE5miwXXYVEgnt93e5YP5b/c7xly4QKAsGr5OvbKcCkKTAYp3W10gpuCKU01o4Gik6WLdNPvbRh7yWAa61MY3d2DKt1wXeKTJT04uh4a/gaEjg0PjitOvgohWvR1eA/veBnatEz2UhXn9pxqUvMFdshGgFx+S9/kQXDOqb1IYTNQTovBAYoxYKRRemN5r6dWhOV3Z23aLzjkgQ55f2+3koSXsTAubYBK30VyAoIIXx1MILmI5Tqd23gLatAZoWgsu9CyxvxuG/3FqRmNBX9B0QpvMzfc1kI7jJ/wEOc4PtObzvcryJl9NtbkuAUNNa1FiQsMbe5vYHPJtiumkFFF1SQ7sGlgQkBcu73ltctjgXRs0r/x+GxJQ31/GI5SwMYcmJSiJiGkJwH8gA+lPvac0JaaBhw8aQwbZ1EyrxW4kOCBvIfHH008TTHdMdZ2eYZfi2C/bW9cBnUpbTmxpMwXt9HqsVWN154GLQKwo0iJkEikiI9TZWzMcj5K1dvP763ND13LBi4CSVwAl7WEref4PP7hmzI83BtPb60EvoID4yMltjCaj9XZA/Yk6XkTZYBwyobGBrEzEL7S/uqUgdT4UZ1viZa/QaPxgk2bcvKMaHdq6/VDm/GgcGG2y51oNie/fnz/+MYHZT9BagO4ybiOzeoAitNFtTsVvGUztw0T1mBYVwukP71WKn7cbrP7z+B3WoD/WpvtR38uM6UE9HEuirUQbDdM7GCnptmF9HWaWlCwhui9ttnet6j7BbwaeCTKLvVMOCpGM4eAfkgHSQSuLP2xHeD17ONNdfk+bWGNOGY0DDidDQ6jiqhzwqbuSsjYwk0Pa6OZ2zr4sy4a2UJ5wHuau6ck6BUWusjpPaPtAiNMtxg+VutB4VGtqIA1LqJFSWxlq4f3IieqSoxEL6BO+/hAdMuBbKCSn51iwDr/L2MSs42zgw+BAsJ6BPR/R2xOTyJxLqA+HwIbb1TcHcgbCNYUKaxdEIRVtZVu9BlD23QrrSSVoGM4w4tL2BtwqAEhQ7qVlpGuNmFKZHCKCAZLlNCZJ9oR/yxQSZA5/d7nDjsWwV6lFVlhB2HzYYqjRVdltnZ6mr7Ffg0ODclE9fgJDX8bPfhISXsmSylRDjc9Utr68Z0m7Sdli0giLiNUaS0rvq0xrScO3oBvv5MlDpzZ6YycJ5CDTbVZQy5hiO6WY6mPk1ciQZVFxIxKfycIO+BExW0ZqbFLC9gKTKre9AN5+kg4QXBksjbw6J9sZeRUrIz4c1E/+WZHEjbpZy5JWmBwDOwpWQVd5DSAZt8KkfhpVaLDLxQftJNvKtz8ud9OjjiIb5CGcSXph7M6KCeg03w92OC8j9iiTgGDdHdwvpLZ1obFZ1jRk0t328/1jP4Z7dPV0qMZXYsMDEmS5Vl2rXKI3SsJrNHnMTSrriQvpu1qiEOXN7T3NkJ6gynaAuJ0pAtbUOK3R1HYh1vGJ5/CNeJTt6bTIpaM55PzayoYmXMZVzSga3PvYvWERP1wKCMli489rO9gKameA2r6m1X41ikfX+72niZqLwEhL285XKqMaoDX/pK7x0st+PeVCyH+54sIN5RXYDGNyRA6WSjCvmvY5ckati7Vr7LZYWu4xGzWTOgwJyT91KrN31Q5Qtu0/RecP5gpKqvtNQkyZ6dE7VNkN/mqRC64vcC0n8I6VxcZnwI5ULJrthAy9B0YyuNYvAwXX2bNQW9IKVfcLkdKNhfDSOv2KRM1U2u49J7DSih/jSg98idje/hfyrOX+aHeAGn/yU8tlJFPqSaOd7+1xIf2qLcj6afEzrFuecZ/13uRRncxUqLX2dcsRUD//ctoJjWprnSiLTBoaFcSjzOcQhU2meKKUYksycN9wIjRvcDCG1s8pZCeWqmkcVdBc8dPU6b/NIE05r7iji64waJMFJ3eCae4U/SFWSk2cOYi7OxQgdOiIe3xQfQLRf7EkGcwVD9DJwTeDMZar6Ornc5ulEMKHNUyAg3yeTtweeUrfUSDoEG/hJLJ65hcyu4WYOJqc9WgubQNto4ilIlWGEqnVukTJ1Lg76GM0jRhTELYiJnSoJF97jSmMzukoU5t2igQyI0iMonbpCH+tUsf4Q6KQfhMhyadQXMbdjMswGHwjEGU3M0cYB6ZBeSnVe3jebUMCeCcWJgZKJ1/ncTcTtW0zLW5UYBP52TZsMqidaaeTqmaA1LTYtDuez7nuSSNDtpUwIfE5iN/LZiGyJrK/UKyiz+TYdaazCCpShITE7p9cUuVkPTy9t9Z4kY2BsF4/buxfKjGSUPSD6dSxaq7x8WbdyOqGwgUv7hY+3yOIet803hL+8N9AG6lutIKulPdOT24DgOMX2fU85ifyWx0T+/ov89ydlv59d+stBpnHDy92ajDIvWlUTCPuinWIspLdbDOq+QRTFHRSQbNg6I0nM+xEyP8LQwOqjA/b2Cmzbg3ScM0pXAsxIEmvgSWzzVqNY+lleBCXGjpklfuodHm56nuk0fGbneVtdO6Nagtu1ZrVA70ZeyRMCGZ7owhmn3L3eBL0bnavQWv0LXRsjnLvArPy1/v/zi77e+Q9S7/yV3IlE/3+ERT3kIasD9o9VxYtvEUOttURrlqA8tP74eNT5Zr6gpYV2jRfpH1dVjYM2vFXa/G/11aWlIc1zenG1XeUXTOhQvYLWyh8LFJGwlbVWgtVe+lWwPsp6YEEfH5GkxuemAspUVVoQWds1QGxGa40AZzUvhgXIrbiyYrBP7WCdXGbFlmCVvoxS+OrGGbRw+Nm1OO5amlxY2auLGhuL7h8=', 'base64')
let inbetween = JSON.parse(zlib.brotliDecompressSync(MimeTypeDbRaw).toString())

Just wanted to share this trick with you that I used and am using for large static data. Feel free to close this issue.

postspectacular commented 2 years ago

Thanks, @TheThing (oh hey! :) - alas that package really needs to be browser compatible (personally, that's where I'm using it most), so (sadly) will have to pass on this approach. Still, it be good to take another pass over the current list and do some more pruning. There're quite a few pretty exotic MIME types still included, which we IMHO could lose without causing too much upset... will create another task/issue when I get back home

TheThing commented 2 years ago

That's actually really cool and I think I kinda agree. I do like the mime library and if it gets further pruning. Also, when I pulled the generated I had to make adjustment to how js and mjs files were generated as they should be text/javascript according to the latest spec. There's even an issue about this over at mime-db but the maintainer of that library doesn't wanna listen to reason.

I figured since you are pruning you might also wanna keep that in mind.

Here's the issue in question: https://github.com/jshttp/mime-db/issues/194 Also have a great weekend :)

postspectacular commented 2 years ago

Used the wrong issue number in the commit log, but just FYI: https://github.com/thi-ng/umbrella/commit/5a51f7d0e2c508228954b06d031d2405c2bf258a 👍