nodeconf / US-CFP

Call for participation for NodeConf 2015
15 stars 0 forks source link

Compression's natural limits: the joys of re-inventing the wheel #9

Closed serverless-mom closed 9 years ago

serverless-mom commented 9 years ago

I'd like to see a talk that covers the re-invention or re-engineering of something considered "solved" e.g. encryption, table indexing, or in my case file compression. I give this example as the kind of talk I would give on this subject, but I would much rather see it given by someone else who did a similar thing:

Over the course of few weeks, I came up with a great idea for how to compress very large text files.

My basic approach was a binary tree that assigned the shortest possible values to the most common strings, and used a frequency table that was standard across all texts in one language.

In the pursuit I hit pitfalls and took shortcuts that turned out to be ineffective time-sinks, but my final version actually kinda worked!

Now, I also discovered when I was nearly done that I was re-inventing a well known form of compression called Huffman encoding, and existing modules did this better than I could hope to.

However, part of the point was that by building from scratch a system to solve a 'solved' problem (think of other examples in encryption or storage) I gained basic insight into the limitations of each compression method available.

mikeal commented 9 years ago

As you may have seen we had to cancel the speaking event at the Fox Theatre.

You're welcome to join us at Walker Creek Ranch for NodeConf Adventure which is an un-conference with attendee driven worksshops and discussion sessions. If you'd like to adapt this topic or any other idea to that format and you're planning on attending just log an issue in the Adventure repo.