Shopify / tapioca

The swiss army knife of RBI generation
MIT License
744 stars 127 forks source link

`bin/tapioca gems --exclude` option not working as expected #1037

Open tusharr opened 2 years ago

tusharr commented 2 years ago

I installed a new gem image_processing in my project. The app runs fine with this new gem but when I run tapioca gem to create the RBI definitions, the process seg faults. I passed the --exclude option, and/or added exclude the sorbet/tapioca/config.yml file, but the command seg faults.

The command I'm running is ./bin/tapioca gems --exclude=image_processing but this gem always gets required by the command.

# Gemfile
gem "image_processing", "~> 1.12", require: false

The output is huge, but the first few 100 lines look like this:

➜  api git:(main) ✗ ./bin/tapioca gems                                                                                 
Removing RBI files of gems that have been removed:

  Nothing to do.

Generating RBI files of gems that are added or updated:

  Requiring all gems to prepare for compiling... /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/ruby-vips-2.1.4/lib/vips/image.rb:467: [BUG] Illegal instruction at 0x000000010d7da38f
ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-darwin20]

-- Crash Report log information --------------------------------------------
   See Crash Report log file under the one of following:                    
     * ~/Library/Logs/DiagnosticReports                                     
     * /Library/Logs/DiagnosticReports                                      
   for more details.                                                        
Don't forget to include the above Crash Report log file in bug reports.     

-- Control frame information -----------------------------------------------
c:0045 p:---- s:0262 e:000261 CFUNC  :vips_image_new_matrix_from_array
c:0044 p:0051 s:0254 e:000253 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/ruby-vips-2.1.4/lib/vips/image.rb:467
c:0043 p:0225 s:0245 e:000244 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/ruby-vips-2.1.4/lib/vips/image.rb:529
c:0042 p:0050 s:0235 e:000234 CLASS  /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/image_processing-1.12.2/lib/image_processing/vips.rb:22
c:0041 p:0036 s:0232 e:000231 CLASS  /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/image_processing-1.12.2/lib/image_processing/vips.rb:18
c:0040 p:0007 s:0229 e:000228 CLASS  /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/image_processing-1.12.2/lib/image_processing/vips.rb:7
c:0039 p:0042 s:0226 e:000225 TOP    /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/image_processing-1.12.2/lib/image_processing/vips.rb:6 [FINISH]
c:0038 p:---- s:0223 e:000222 CFUNC  :require
c:0037 p:0235 s:0218 e:000217 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/bootsnap-1.12.0/lib/bootsnap/load_path_cache/core_ext/kernel_requi
c:0036 p:0066 s:0208 e:000207 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/zeitwerk-2.6.0/lib/zeitwerk/kernel.rb:35 [FINISH]
c:0035 p:---- s:0200 e:000199 CFUNC  :const_get
c:0034 p:0024 s:0194 e:000193 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/runtime/reflection.rb:31
c:0033 p:0040 s:0186 e:000185 BLOCK  /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/runtime/trackers/autoload.rb:24
c:0032 p:0081 s:0182 e:000181 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/runtime/trackers/autoload.rb:47 [FINISH]
c:0031 p:---- s:0175 e:000174 CFUNC  :bind_call
c:0030 p:0248 s:0170 e:000169 BLOCK  /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272 [FINISH]
c:0029 p:0004 s:0164 E:002080 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/runtime/trackers/autoload.rb:19 [FINISH]
c:0028 p:---- s:0160 e:000159 CFUNC  :bind_call
c:0027 p:0248 s:0155 e:000154 BLOCK  /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272 [FINISH]
c:0026 p:0041 s:0149 e:000148 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/commands/gem.rb:151 [FINISH]
c:0025 p:---- s:0144 e:000143 CFUNC  :bind_call
c:0024 p:0248 s:0139 e:000138 BLOCK  /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272 [FINISH]
c:0023 p:0018 s:0133 e:000132 BLOCK  /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/commands/gem.rb:261
c:0022 p:0021 s:0130 e:000129 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor/shell/basic.rb:44
c:0021 p:0051 s:0124 e:000123 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/commands/gem.rb:257 [FINISH]
c:0020 p:---- s:0118 e:000117 CFUNC  :bind_call
c:0019 p:0248 s:0113 e:000112 BLOCK  /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272 [FINISH]
c:0018 p:0028 s:0107 e:000105 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/commands/gem.rb:108 [FINISH]
c:0017 p:---- s:0099 e:000098 CFUNC  :bind_call
c:0016 p:0248 s:0093 e:000092 BLOCK  /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272 [FINISH]
c:0015 p:0246 s:0087 e:000086 BLOCK  /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/cli.rb:234
c:0014 p:0006 s:0081 e:000080 BLOCK  /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca.rb:18
c:0013 p:0012 s:0078 e:000077 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/3.0.0/rubygems/user_interaction.rb:46
c:0012 p:0044 s:0072 e:000071 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca.rb:17 [FINISH]
c:0011 p:---- s:0066 e:000065 CFUNC  :bind_call
c:0010 p:0248 s:0061 e:000060 BLOCK  /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272 [FINISH]
c:0009 p:0012 s:0055 E:002398 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/cli.rb:199
c:0008 p:0054 s:0050 e:000049 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor/command.rb:27
c:0007 p:0040 s:0042 e:000041 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:127
c:0006 p:0235 s:0035 e:000034 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor.rb:392
c:0005 p:0062 s:0022 e:000021 METHOD /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor/base.rb:485
c:0004 p:0122 s:0015 E:000a00 TOP    /Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/exe/tapioca:23 [FINISH]
c:0003 p:---- s:0012 e:000011 CFUNC  :load
c:0002 p:0157 s:0007 E:000118 EVAL   ./bin/tapioca:29 [FINISH]
c:0001 p:0000 s:0003 E:001b90 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
./bin/tapioca:29:in `<main>'
./bin/tapioca:29:in `load'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/exe/tapioca:23:in `<top (required)>'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/cli.rb:199:in `gem'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272:in `block in _on_method_added'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272:in `bind_call'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca.rb:17:in `silence_warnings'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/3.0.0/rubygems/user_interaction.rb:46:in `use_ui'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca.rb:18:in `block in silence_warnings'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/cli.rb:234:in `block in gem'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272:in `block in _on_method_added'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272:in `bind_call'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/commands/gem.rb:108:in `sync'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272:in `block in _on_method_added'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272:in `bind_call'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/commands/gem.rb:257:in `perform_additions'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/thor-1.2.1/lib/thor/shell/basic.rb:44:in `indent'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/commands/gem.rb:261:in `block in perform_additions'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272:in `block in _on_method_added'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272:in `bind_call'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/commands/gem.rb:151:in `require_gem_file'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272:in `block in _on_method_added'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272:in `bind_call'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/runtime/trackers/autoload.rb:19:in `eager_load_all!'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272:in `block in _on_method_added'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/sorbet-runtime-0.5.10098/lib/types/private/methods/_methods.rb:272:in `bind_call'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/runtime/trackers/autoload.rb:47:in `with_disabled_exits'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/runtime/trackers/autoload.rb:24:in `block in eager_load_all!'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/runtime/reflection.rb:31:in `constantize'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/tapioca-0.8.3/lib/tapioca/runtime/reflection.rb:31:in `const_get'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/zeitwerk-2.6.0/lib/zeitwerk/kernel.rb:35:in `require'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/bootsnap-1.12.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/bootsnap-1.12.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/image_processing-1.12.2/lib/image_processing/vips.rb:6:in `<main>'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/image_processing-1.12.2/lib/image_processing/vips.rb:7:in `<module:ImageProcessing>'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/image_processing-1.12.2/lib/image_processing/vips.rb:18:in `<module:Vips>'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/image_processing-1.12.2/lib/image_processing/vips.rb:22:in `<class:Processor>'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/ruby-vips-2.1.4/lib/vips/image.rb:529:in `new_from_array'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/ruby-vips-2.1.4/lib/vips/image.rb:467:in `matrix_from_array'
/Users/tushar/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/ruby-vips-2.1.4/lib/vips/image.rb:467:in `vips_image_new_matrix_from_array'
KaanOzkan commented 2 years ago

I am not able to reproduce this locally using Ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) on an m1 machine. Since it's a Ruby crash it may also be architecture specific. Can you also try using a newer Ruby version?

thegeorgeous commented 6 months ago

@KaanOzkan We are facing the same problem with a private gem. We tried excluding it from the config file as well, but it is always included. tapioca works fine when we remove the gem from the Gemfile and Gemfile.lock

This is the ruby version I have: ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [arm64-darwin23]

KaanOzkan commented 6 months ago

This is a hack but I wonder if you can add a file like sorbet/tapioca/prerequire.rb that's removing the gem using remove_const. --prerequire is a flag that runs before the gems are loaded. That flag will need to be passed into the gem command. I'm not sure if it'll work but it's worth a shot.

# sorbet/tapioca/prerequire.rb
self.class.send(:remove_const, :Foo)
# You might need to remove all constants defined in the gem
# sorbet/tapioca/config.yml
---
gem:
  prerequire: "prerequire.rb"
wesharper commented 5 months ago

FWIW, I recently faced the same issue with the noticed gem in ruby 3.3.0 on a Mac M1. The exclude option did not exclude the gem and only removing it from the Gemfile worked.

wlaurance commented 2 weeks ago

I'm also having trouble with noticed (and active_storage-postgresql) @wesharper

amomchilov commented 3 days ago

It looks like Tapioca intentionally disregards requires: false in the Gemfile (#818). The aim was to pickup more RBIs for more gems, which are conditionally/lazily require-ed in the application (rather than from the Gemfile itself).

The assumption there is that merely requiring a gem would be harmless, and shouldn't be an issue (q: why does requiring these gems cause a crash?), but that appears not to be the case. Ideally we should fix these crashers, but also, Tapioca should have an "escape hatch" to help you circumvent them in the meanwhile.

IMO it's reasonable to change this AutoRequireHook so that it respects require: false for gems that are in the --exclude= list. Would one of you be interested in opening a PR for this?

Edit: Looks like that's already the case:

https://github.com/Shopify/tapioca/blob/0a9f29181270905363d472a041b16443f20bd1a4/lib/tapioca/gemfile.rb#L46

This will need some deeper debugging/investigation. Contributions are very much welcomed