Publish huge data files/BLOBs on npm?

[mk-pmb]

Hi! My converter for MIDI wavetables (SF2 → MIDI.js format) finally works, so I'd like to share the resulting files with others. NPM would seem the logical place because that way it would be easy to require any of the data files. (JSON directly, original MIDI.js via custom loader.) Problem is, they're huge. The average size of a data file is about 1 MB, the largest is about 2.5 MB, but there are 516 of them (128 instruments + 1 drum set) × 2 audio formats × 2 container formats). Should data files like these be on npm at all? If so, how to package them? If I upload each of them as a separate package, each would be of acceptable size but the packages list for my username becomes useless. If I upload them all as just 1 package, it's not very helpful for people who need just one data format. If I upload one package for each data format, it bloats my packages-by-user list less but still isn't too helpful for people who need just a few instruments, but need each of them in 2 audio formats.

Another issue which is technical at first and a policy issue only later: In case someone decides to make a trimmed-down version (their 20 favorite instruments) and re-uploads them, a tarball seems bad for deduplication, and even with a single package it puts a huge burden on who can afford to maintain it (bandwidth/ traffic/ upload time). Could we invent some better BLOB storage for npm, with an adapter to git-annex? Then it might become a policy to consider using that BLOB storage for publishing huge data files. Of course the choice of storage method should be transparent to people who just install the package the usual way.

[kemitchell]

Were you able to find the answer to your question? If not, I'd suggest you e-mail support@npmjs.com. They can answer questions about use of the public registry.