mini_mime vs marcel

[pjmartorell]

Hi, I don't know if this is the right place to post it, but I'm trying to compare mini_mime vs marcel regarding looking up by extension, because I think both gems cover the same space. I was trying to compare the number of extensions registered, the performance and memory consumption of every gem.

	mini_mime	marcel
#extensions ​	File.open(MiniMime::Configuration.ext_db_path).readlines.count => 1196	Marcel::EXTENSIONS.count => 1243

Regarding memory handling, mini_mime has a hash cache of 200 rows and misses are binary-searched from a file while marcel loads all records in a hash in memory. Is not reading from a file less performant than loading everything in memory? Loading everything in memory consumes more memory obviously, but the gain in performance outweighs the memory consumption, in my opinion.

Also I noticed that both DBs in mini_mime contain similar data but is there any reason why are not both DBs merged removing duplicates? I saw that when merging both files the number of rows/extensions is 1210, but I'm not completely sure if it's due to an error removing duplicates:

irb(main)> File.readlines(MiniMime::Configuration.ext_db_path).each do |line|
irb(main)*     s << line.strip
irb(main)> end
irb(main)> File.readlines(MiniMime::Configuration.content_type_db_path).each do |line|
irb(main)*     s << line.strip
irb(main)> end
irb(main)> s.length
=> 1210

[ahorek]

unlike mini_mime, which is just a simple table of extension -> content type, marcel and mime_magic also allow lookup by file signature https://en.wikipedia.org/wiki/List_of_file_signatures (magic numbers). This is considered as a security feature, that's why Rails use it.

https://github.com/mime-types/ruby-mime-types - has a much more complex API, mini_mime uses the same DB, but it's simplified for performance reasons (1 extension = 1 mime type).

btw Rack also has its own DB https://github.com/rack/rack/blob/master/lib/rack/mime.rb#L51

sometimes it's hard to persuade some maintainers to do a change https://github.com/rest-client/rest-client/pull/557 and it would be even harder to do a much more breaking change in marcel just to save a few kb of memory. Yes, it would be nice and I'm 100% pro, but I also don't think it's realistic :)

[pjmartorell]

@ahorek thanks for the reference to https://github.com/rest-client/rest-client/pull/557, is exactly what I wanted to know/understand.

[SamSaffron]

I started discussing @georgeclaghorn

My long term thinking here.

• Move discourse/mini_mime to rails/mini_mime ...
• Merge marcel into mini_mime so mini_mime can also do lookup by file content
• Keep parity with mime-types (which was an underlying goal) ... so if Marcel has 1243 and mime-types only has 1196 we got to upstream missing stuff into mime-types-data
• Keep the perf characteristics of mini-mime (aims to be fastest implementation for cached lookups, reasonable default cache size) . Majority of processes do a very small amount of mime lookups misses are unlikely. Many processes do no mime type lookups, no point keeping in memory
• Keep to a very tiny public interface, lookup by extension / type / filename / content.

[halostatue]

@SamSaffron I am all in favour of adding more data to mime-types-data.