line 66: regex error 10, (invalid repetition count(s)) (FileMagic::FileMagicError)

[CpuID]

Getting this on a specific file, which file --mime-type returns "text/plain" for currently.

This error occurs under MRI 2.0.0-p353 and Rubinius 2.2.6.

nsullivan:~ nsullivan$ file --mime-type /Users/nsullivan/.rbx/09/097e79c388cab23f8981958bea209efff565ae1b
/Users/nsullivan/.rbx/09/097e79c388cab23f8981958bea209efff565ae1b: text/plain

Any ideas?

[blackwinter]

I've never seen this error before. Is there any chance you could provide me with that file? And ideally the exact invocation of the FileMagic method you used.

[CpuID]

No problem. See below for how I have executed it:

nsullivan:rbxtest nsullivan$ chmod +x test.rb 
nsullivan:rbxtest nsullivan$ ./test.rb 
An exception occurred running ./test.rb:

    line 66: regex error 10, (invalid repetition count(s)) (FileMagic::FileMagicError)

Backtrace:

                 Object#__script__ at test.rb:7
  Rubinius::CodeLoader#load_script at kernel/delta/code_loader.rb:66
  Rubinius::CodeLoader.load_script at kernel/delta/code_loader.rb:152
           Rubinius::Loader#script at kernel/loader.rb:649
             Rubinius::Loader#main at kernel/loader.rb:831
nsullivan:rbxtest nsullivan$ rbenv versions
  system
  1.8.7-p352
  1.8.7-p374
  1.9.3-p484
  2.0.0-p353
  2.1.1
  rbx-2.2.5
* rbx-2.2.6 (set by /Users/nsullivan/rbxtest/.ruby-version)
nsullivan:rbxtest nsullivan$ rm .ruby-version 
nsullivan:rbxtest nsullivan$ rbenv versions
  system
  1.8.7-p352
  1.8.7-p374
  1.9.3-p484
* 2.0.0-p353 (set by /usr/local/var/rbenv/version)
  2.1.1
  rbx-2.2.5
  rbx-2.2.6
nsullivan:rbxtest nsullivan$ ./test.rb 
./test.rb:7:in `file': line 66: regex error 10, (invalid repetition count(s)) (FileMagic::FileMagicError)
    from ./test.rb:7:in `<main>'
nsullivan:rbxtest nsullivan$ cat test.rb 
#!/usr/bin/env ruby

require 'ruby-filemagic'
require 'pp'

fm = FileMagic.new(FileMagic::MAGIC_MIME)
file_mime_type = fm.file('/Users/nsullivan/.rbx/09/097e79c388cab23f8981958bea209efff565ae1b')
pp file_mime_type

I'll email you a URL to the file itself, I don't believe it has anything sensitive but just to be safe I won't put it in this public ticket.

[CpuID]

URL to pull the file emailed to your address on your github account.

[blackwinter]

Great, thanks. Unfortunately, I could not reproduce the error with ruby 2.0.0p377, nor ruby 2.1.2p77, nor rubinius 2.2.6 (2.1.0) on x86_64-linux-gnu. Can you try different ruby versions? (2.1.1, 1.9.3-p484) What libmagic version do you have? (5.14 here) What happens if you remove the offending line?

(And just for the record: The tests pass on ruby 2.0.0p353 and rubinius 2.2.6 with libmagic 5.09.)

[CpuID]

Interesting. I am running Ruby on Mac OSX here, using HomeBrew for libmagic, looks like 5.18

nsullivan:~ nsullivan$ brew info libmagic
libmagic: stable 5.18 (bottled)
http://www.darwinsys.com/file/
/usr/local/Cellar/libmagic/5.18 (14 files, 2.9M) *
  Poured from bottle
From: https://github.com/Homebrew/homebrew/commits/master/Library/Formula/libmagic.rb

I have just given it a try on 2.1.1:

nsullivan:rbxtest nsullivan$ rbenv local 2.1.1
nsullivan:rbxtest nsullivan$ bundle install
Fetching gem metadata from https://rubygems.org/.........
Fetching additional metadata from https://rubygems.org/..
Installing builder 3.2.2
Installing excon 0.31.0
Installing formatador 0.2.4
Installing mime-types 2.1
Installing multi_json 1.10.0
Installing net-ssh 2.9.0
Installing net-scp 1.2.1
Installing mini_portile 0.5.3
Installing nokogiri 1.6.1
Installing fog 1.20.0
Installing ruby-filemagic 0.5.2
Installing unf_ext 0.0.6
Installing unf 0.1.3
Using bundler 1.6.1
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.
Post-install message from ruby-filemagic:

ruby-filemagic-0.5.2 [2014-04-24]:

* Use MAGIC_VERSION if available.
* Made tests more robust.

nsullivan:rbxtest nsullivan$ ./test.rb 
./test.rb:7:in `file': line 66: regex error 10, (invalid repetition count(s)) (FileMagic::FileMagicError)
    from ./test.rb:7:in `<main>'
nsullivan:rbxtest nsullivan$ rbenv versions
  system
  1.8.7-p352
  1.8.7-p374
  1.9.3-p484
  2.0.0-p353
* 2.1.1 (set by /Users/nsullivan/rbxtest/.ruby-version)
  rbx-2.2.5
  rbx-2.2.6

I couldn't get 1.9.3-p484 tested due to some gem failures, see below:

Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

        /usr/local/var/rbenv/versions/1.9.3-p484/bin/ruby extconf.rb 
checking for main() in -lstdc++... yes
checking for ruby/encoding.h... yes
creating Makefile

make
compiling unf.cc
cc1plus: error: -Werror=shorten-64-to-32: no option -Wshorten-64-to-32
make: *** [unf.o] Error 1

Gem files will remain installed in /usr/local/var/rbenv/versions/1.9.3-p484/lib/ruby/gems/1.9.1/gems/unf_ext-0.0.6 for inspection.
Results logged to /usr/local/var/rbenv/versions/1.9.3-p484/lib/ruby/gems/1.9.1/gems/unf_ext-0.0.6/ext/unf_ext/gem_make.out
An error occurred while installing unf_ext (0.0.6), and Bundler cannot continue.
Make sure that `gem install unf_ext -v '0.0.6'` succeeds before bundling.

Looks like it was gcc 4.2.1 that Xcode has installed I believe being used, but I could be wrong here... Obviously let's keep this bug to working out why the regex fails rather than why my build of the gem doesnt work for now :)

[blackwinter]

Well, that would suggest that it might have to do with changes in libmagic. But again I wasn't able to reproduce with 5.18 either.

As for your gem install failure: That's unf_ext which is unrelated to ruby-filemagic. I don't think that we have to look at other Ruby versions, though. It seems safe to say that it's platform related somehow, so we can concentrate on 2.1.

Seeing as I don't have access to a Mac, could you try the following and report with your results:

In your shell:

• file -v

In irb -rfilemagic:

• FileMagic::MAGIC_VERSION
• FileMagic.path from 0.5.2.pre1
• FileMagic.mime.file('test.rb')
• FileMagic.mime.buffer(File.read('/Users/nsullivan/.rbx/09/097e79c388cab23f8981958bea209efff565ae1b', 209))
• FileMagic.mime.buffer(File.read('/Users/nsullivan/.rbx/09/097e79c388cab23f8981958bea209efff565ae1b', 218))
• Dir['/Users/nsullivan/.rbx/*/*'].find { |f| FileMagic.mime.file(f) rescue nil } and send me that file if any

I have a feeling that this is not an issue with ruby-filemagic, but rather with your magic file(s) -- the error message seems to indicate as much. Alas, then why would the file command succeed? I hope we can get to the bottom of it with some back-and-forth.

[CpuID]

Awesome, thanks. See below:

nsullivan:~ nsullivan$ file -v
file-5.04
magic file from /usr/share/file/magic

And from irb -rfilemagic:

nsullivan:rbxtest nsullivan$ irb -rfilemagic
irb(main):001:0> FileMagic::MAGIC_VERSION
=> "5.18"
irb(main):003:0> FileMagic.mime.file('test.rb')
=> "text/x-ruby; charset=us-ascii"
irb(main):004:0> FileMagic.mime.buffer(File.read('/Users/nsullivan/.rbx/09/097e79c388cab23f8981958bea209efff565ae1b', 209))
=> "text/plain; charset=us-ascii"
irb(main):005:0> FileMagic.mime.buffer(File.read('/Users/nsullivan/.rbx/09/097e79c388cab23f8981958bea209efff565ae1b', 218))
=> "text/plain; charset=us-ascii"
irb(main):006:0> Dir['/Users/nsullivan/.rbx/*/*'].find { |f| FileMagic.mime.file(f) rescue nil }
=> "/Users/nsullivan/.rbx/00/000144baa96257cbd535c3b229cdfb4ffcecf3fb"
irb(main):007:0> exit

When you said FileMagic.path from 0.5.2.pre1, as in install that gem and try running it? Running 0.5.2 already, is pre1 older?

Emailing you /Users/nsullivan/.rbx/00/000144baa96257cbd535c3b229cdfb4ffcecf3fb in the meantime.

[blackwinter]

Um, now that's less helpful than I had hoped for... Let's step back for a moment:

1. file -i <offending-file> works.
2. FileMagic.mime.file/.buffer works in some cases (even of the same kind), but not with <offending-file>.
3. The line number in the error message does not seem to refer to the file being processed. Otherwise, I would have expected the second call to buffer above to fail. My guess was that it refers to the magic file(s). Can you send me the file in /usr/share/file/magic and maybe also the ones in FileMagic.path? I may be on the wrong track, though.
4. Everything so far works on x86_64-linux, but not on darwin.

I seem to recall that one can have competing/conflicting versions of libraries on OS X (e.g., system version vs. homebrew version). Is it possible that file picks up a different one than FileMagic?

Other than that, I'm afraid I'm out of ideas. Let me think about it for a while. But feel free to poke around some more:

At which line does <offending-file> fail?

fm, b = FileMagic.new(FileMagic::MAGIC_MIME), ''
File.foreach('/Users/nsullivan/.rbx/09/097e79c388cab23f8981958bea209efff565ae1b') { |l| fm.buffer(b << l) rescue break }
p $.

Which of the .rbx files work and which don't?

ok, nok = Dir['/Users/nsullivan/.rbx/*/*'].partition { |f| fm.file(f) rescue nil }
p [ok.size, nok.size]

etc.

When you said FileMagic.path from 0.5.2.pre1, as in install that gem and try running it? Running 0.5.2 already, is pre1 older?

Yeah, that's what I meant. Forgot to increment the version number ;) I just pushed 0.5.3.pre1 instead.

[blackwinter]

By now I'm pretty confident that it's indeed an issue with one of your magic files, not with these bindings. line 66 refers to the "current magic line number", we just have to find out which file it is.

The fact that file behaves differently than FileMagic may still indicate a problem, but that should hardly matter. Just to be sure: All the commands have been executed on the same machine by the same user, right?

[blackwinter]

One more thing to look at:

strace -e open 2>&1 ruby -rfilemagic -e 'FileMagic.mime.file("/Users/nsullivan/.rbx/09/097e79c388cab23f8981958bea209efff565ae1b")' | grep [./]magic

and

strace -e open 2>&1 file --mime-type /Users/nsullivan/.rbx/09/097e79c388cab23f8981958bea209efff565ae1b | grep [./]magic

(The equivalent of strace -e open on OS X should be dtruss -t open.)