search

ruby / rbs

Type Signature for Ruby
Other
1.96k stars 216 forks source link

`rbs collection install` fails with parse error #1815

Closed Grohden closed 5 months ago

Grohden commented 6 months ago

I'm just following readme instructions here, init works and install gives me this:

rbs collection install 
/Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/3.2.0/psych/parser.rb:62:in `_native_parse': (<unknown>): mapping values are not allowed in this context at line 2 column 11 (Psych::SyntaxError)
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/3.2.0/psych/parser.rb:62:in `parse'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/3.2.0/psych.rb:455:in `parse_stream'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/3.2.0/psych.rb:399:in `parse'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/3.2.0/psych.rb:323:in `safe_load'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/3.2.0/psych.rb:369:in `load'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/lib/rbs/collection/config.rb:37:in `from_path'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/lib/rbs/collection/config/lockfile_generator.rb:32:in `initialize'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/lib/rbs/collection/config/lockfile_generator.rb:26:in `new'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/lib/rbs/collection/config/lockfile_generator.rb:26:in `generate'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/lib/rbs/collection/config.rb:33:in `generate_lockfile'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/lib/rbs/cli.rb:1042:in `run_collection'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/lib/rbs/cli.rb:130:in `run'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/exe/rbs:7:in `<top (required)>'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/bin/rbs:25:in `load'
        from /Users/gabrielrohden/.asdf/installs/ruby/3.2.3/bin/rbs:25:in `<main>'

And I guess that the parsing error might be because the lock file is generated wrong? image

edit: I would help with reproduction steps, but I have no clue on where to start.. what would be needed? minimal gemfile? paths?

pocke commented 6 months ago

And I guess that the parsing error might be because the lock file is generated wrong?

Probably right. The beginning of the rbs_collection.lock.yaml looks removed unexpectedly.

edit: I would help with reproduction steps, but I have no clue on where to start.. what would be needed? minimal gemfile? paths?

This file depends on Gemfile.lock and rbs_collection.yaml. Could you share these files and Ruby and RBS versions?

hsbt commented 6 months ago

FYI: https://github.com/ruby/psych/discussions/607

It caused to mix multiple versions of Psych.

Grohden commented 6 months ago

Right, here's more infos:

rbs --version -> rbs 2.8.4 ruby --version -> ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [arm64-darwin23]

After doing a somewhat binary search commenting the whole gemfile and re-generating gemfile.lock I think that there's a limit on how many rbs files/libs we can have?

I've did some tests commenting and uncommenting the whole gemfile we have and always running

bundle install && rm ./rbs_collection.lock.yaml && rbs collection instal

I did this like 3 times in different regions, and always got stuck with uncomment dependency X -> get error, with random dependencies.. I can comment other dependencies that would lower the rbs numbers and then uncomment others and the error would appear.. it seems the limit is 100 dependencies (assuming the number from the It's done! 92 gems' RBSs now installed. message)

I'll check if I can make a reproduction gemfile with enough dependencies to test

Grohden commented 6 months ago

@pocke I guess you can try this one to get the same error:

```ruby # frozen_string_literal: true source 'https://rubygems.org' git_source(:github) { |repo| "https://github.com/#{repo}.git" } ruby '~> 3' gem 'rails', '~> 7.0.8' gem 'pg', '1.5.6' gem 'redis', '~> 4.0' gem 'active_record_extended' gem 'activerecord-import' gem 'analytics-ruby', require: 'segment' gem 'asana' gem 'aws-sdk-s3' gem 'aws-sdk-secretsmanager' gem 'aws-sdk-sns' gem 'aws-sdk-sqs' gem 'aws-sdk-kms' gem 'aws-sdk-textract' gem 'blazer' gem 'bootsnap', require: false gem 'business_time' gem 'bulma-rails' gem 'capybara' gem 'cloudwatchlogger' gem 'captcha_bypass' gem 'code_owner' gem 'console_watcher' gem 'core-sdk' gem 'devise' gem 'engem' gem 'google-api-client' gem 'geocoder' gem 'hashdiff' gem 'holidays' gem 'faker' gem 'factory_bot', require: false gem 'fast_jsonapi' gem 'feature_tower' gem 'foreman', require: false gem 'koala' gem 'redis-client' gem 'request_store-sidekiq' gem 'retryable' gem 'rubocop' gem 'rbs' gem 'rubyXL' gem 'store_model' gem 'sendgrid-ruby' gem 'stackprof' gem 'sentry-ruby' gem 'sentry-rails' gem 'sentry-sidekiq' gem 'slack-ruby-client' gem 'shortener' gem 'spreadsheet' gem 'steep' gem 'savon' gem 'sidekiq' gem 'sidekiq-unique-jobs' gem 'shakapacker' gem 'twilio-ruby' gem 'image_processing' gem 'faraday' gem 'nokogiri' gem 'view_component' gem 'lookbook' gem 'ox' gem 'omniauth-okta' gem 'omniauth-rails_csrf_protection' gem 'omniauth-saml' gem 'omniauth' gem 'tiddle' gem 'kms_encrypted' gem 'lockbox' gem 'pry-stack_explorer' ```

Btw, I was never able to get above 95 rbs gems

pocke commented 6 months ago

Thanks for providing the Gemfile. Unfortunately I'm busy this week because I'll attend RubyKaigi` 2024. I'll work on this issue after RubyKaigi. Sorry for the inconvenient 🙏

pocke commented 5 months ago

I could reproduce this problem with the following Gemfile. Some gems in your Gemfile do not exist in RubyGems.org, so I commented out them and added some gems to increase number of RBSs.

```ruby # frozen_string_literal: true source 'https://rubygems.org' git_source(:github) { |repo| "https://github.com/#{repo}.git" } ruby '~> 3' gem 'rails', '~> 7.0.8' gem 'pg', '1.5.6' gem 'redis', '~> 4.0' gem 'active_record_extended' gem 'activerecord-import' gem 'analytics-ruby', require: 'segment' gem 'asana' gem 'aws-sdk-s3' gem 'aws-sdk-secretsmanager' gem 'aws-sdk-sns' gem 'aws-sdk-sqs' gem 'aws-sdk-kms' gem 'aws-sdk-textract' gem 'blazer' gem 'bootsnap', require: false gem 'business_time' gem 'bulma-rails' gem 'capybara' gem 'cloudwatchlogger' # gem 'captcha_bypass' # gem 'code_owner' # gem 'console_watcher' # gem 'core-sdk' gem 'devise' # gem 'engem' gem 'google-api-client' gem 'geocoder' gem 'hashdiff' gem 'holidays' gem 'faker' gem 'factory_bot', require: false gem 'fast_jsonapi' # gem 'feature_tower' gem 'foreman', require: false gem 'koala' gem 'redis-client' gem 'request_store-sidekiq' gem 'retryable' gem 'rubocop' gem 'rbs' gem 'rubyXL' gem 'store_model' gem 'sendgrid-ruby' gem 'stackprof' gem 'sentry-ruby' gem 'sentry-rails' gem 'sentry-sidekiq' gem 'slack-ruby-client' gem 'shortener' gem 'spreadsheet' gem 'steep' gem 'savon' gem 'sidekiq' gem 'sidekiq-unique-jobs' gem 'shakapacker' gem 'twilio-ruby' gem 'image_processing' gem 'faraday' gem 'nokogiri' gem 'view_component' gem 'lookbook' gem 'ox' gem 'omniauth-okta' gem 'omniauth-rails_csrf_protection' gem 'omniauth-saml' gem 'omniauth' gem 'tiddle' gem 'kms_encrypted' gem 'lockbox' gem 'pry-stack_explorer' gem 'active_model_serializers' gem 'browser' gem 'cancancan' gem 'carrierwave' gem 'chartkick' gem 'chronic' gem 'chunky_png' gem 'circuitbox' gem 'delayed_job' gem 'delayed_job_active_record' gem 'diff-lcs' gem 'gimei' gem 'moji' gem 'moneta' gem 'octokit' ``` 
$ bundle lock
$ rbs collection install
/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/3.2.0/psych/parser.rb:62:in `_native_parse': (<unknown>): did not find expected key while parsing a block mapping at line 1 column 1 (Psych::SyntaxError)
        from /Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/3.2.0/psych/parser.rb:62:in `parse'
        from /Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/3.2.0/psych.rb:455:in `parse_stream'
        from /Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/3.2.0/psych.rb:399:in `parse'
        from /Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/3.2.0/psych.rb:323:in `safe_load'
        from /Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/3.2.0/psych.rb:369:in `load'
        from /Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/lib/rbs/collection/config.rb:37:in `from_path'
        from /Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/lib/rbs/collection/installer.rb:10:in `initialize'
        from /Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/lib/rbs/cli.rb:1044:in `new'
        from /Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/lib/rbs/cli.rb:1044:in `run_collection'
        from /Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/lib/rbs/cli.rb:130:in `run'
        from /Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-2.8.4/exe/rbs:7:in `<top (required)>'
        from /Users/kuwabara.masataka/.rbenv/versions/3.2.3/bin/rbs:25:in `load'
        from /Users/kuwabara.masataka/.rbenv/versions/3.2.3/bin/rbs:25:in `<main>'

And I've confirmed that this problem does not occur on the latest RBS (v3.4.4).

$ rbs -v
rbs 3.4.4
$ rbs collection install
Using actioncable:7.1 (actioncable@438153d157e)
Using actionmailer:7.0 (actionmailer@438153d157e)
Using actionpack:6.0 (actionpack@438153d157e)
Using actionview:6.0 (actionview@438153d157e)
Using active_model_serializers:0.10 (active_model_serializers@438153d157e)
Using activejob:6.0 (activejob@438153d157e)
Using activemodel:7.0 (activemodel@438153d157e)
Using activerecord:7.0 (activerecord@438153d157e)
Using activerecord-import:1.5 (activerecord-import@438153d157e)
Using activestorage:6.1 (activestorage@438153d157e)
Using activesupport:7.0 (activesupport@438153d157e)
Using addressable:2.8 (addressable@438153d157e)
Using ast:2.4 (ast@438153d157e)
Using aws-sdk-core:3 (aws-sdk-core@438153d157e)
Using aws-sdk-kms:1 (aws-sdk-kms@438153d157e)
Using aws-sdk-s3:1 (aws-sdk-s3@438153d157e)
Using aws-sdk-sns:1 (aws-sdk-sns@438153d157e)
Using aws-sdk-sqs:1 (aws-sdk-sqs@438153d157e)
Using base64:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/base64/0)
Using bcrypt:3.1 (bcrypt@438153d157e)
Using bigdecimal:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/bigdecimal/0)
Using binding_of_caller:1.0 (binding_of_caller@438153d157e)
Using browser:5.3 (browser@438153d157e)
Using business_time:0.13 (business_time@438153d157e)
Using cancancan:3.5 (cancancan@438153d157e)
Using carrierwave:3.0 (carrierwave@438153d157e)
Using cgi:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/cgi/0)
Using chartkick:5.0 (chartkick@438153d157e)
Using chronic:0.10 (chronic@438153d157e)
Using chunky_png:1.4.0 (chunky_png@438153d157e)
Using circuitbox:2.0 (circuitbox@438153d157e)
Using concurrent-ruby:1.1 (concurrent-ruby@438153d157e)
Using connection_pool:2.4 (connection_pool@438153d157e)
Using csv:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/csv/0)
Using date:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/date/0)
Using delayed_job:4.1 (delayed_job@438153d157e)
Using delayed_job_active_record:4.1 (delayed_job_active_record@438153d157e)
Using diff-lcs:1.5 (diff-lcs@438153d157e)
Using erb:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/erb/0)
Using faker:2.23 (faker@438153d157e)
Using faraday:2.5 (faraday@438153d157e)
Using fileutils:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/fileutils/0)
Using forwardable:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/forwardable/0)
Using geocoder:1.8 (geocoder@438153d157e)
Using gimei:1.1 (gimei@438153d157e)
Using globalid:1.1 (globalid@438153d157e)
Using google-protobuf:3.22 (google-protobuf@438153d157e)
Using googleauth:1.11 (googleauth@438153d157e)
Using hashdiff:1.1 (hashdiff@438153d157e)
Using hashie:5.0 (hashie@438153d157e)
Using holidays:8.4 (holidays@438153d157e)
Using httpclient:2.8 (httpclient@438153d157e)
Using i18n:1.10 (i18n@438153d157e)
Using json:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/json/0)
Using jwt:2.5 (jwt@438153d157e)
Using listen:3.2 (listen@438153d157e)
Using logger:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/logger/0)
Using mail:2.8 (mail@438153d157e)
Using marcel:1.0 (marcel@438153d157e)
Using minitest:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/minitest/0)
Using moji:1.6 (moji@438153d157e)
Using moneta:1.6 (moneta@438153d157e)
Using monitor:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/monitor/0)
Using mutex_m:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/mutex_m/0)
Using net-http:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/net-http/0)
Using net-protocol:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/net-protocol/0)
Using net-smtp:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/net-smtp/0)
Using nkf:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/nkf/0)
Using nokogiri:1.11 (nokogiri@438153d157e)
Using octokit:8.0 (octokit@438153d157e)
Using openssl:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/openssl/0)
Using optparse:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/optparse/0)
Using parallel:1.20 (parallel@438153d157e)
Using parser:3.2 (parser@438153d157e)
Using pathname:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/pathname/0)
Using rack:2.2 (rack@438153d157e)
Using rails-dom-testing:2.0 (rails-dom-testing@438153d157e)
Using railties:6.0 (railties@438153d157e)
Using rainbow:3.0 (rainbow@438153d157e)
Using rake:13.0 (rake@438153d157e)
Using redis:4.2 (redis@438153d157e)
Using regexp_parser:2.8 (regexp_parser@438153d157e)
Using request_store:1.5 (request_store@438153d157e)
Using retryable:3.0 (retryable@438153d157e)
Using ripper:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/ripper/0)
Using rubocop:1.57 (rubocop@438153d157e)
Using rubocop-ast:1.30 (rubocop-ast@438153d157e)
Using rubyzip:2.3 (rubyzip@438153d157e)
Using securerandom:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/securerandom/0)
Using sentry-ruby:5.2 (sentry-ruby@438153d157e)
Using sidekiq:7.0 (sidekiq@438153d157e)
Using signet:0.19 (signet@438153d157e)
Using singleton:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/singleton/0)
Using socket:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/socket/0)
Using spreadsheet:1.3 (spreadsheet@438153d157e)
Using stackprof:0.2 (stackprof@438153d157e)
Using strscan:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/strscan/0)
Using tempfile:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/tempfile/0)
Using thor:1.2 (thor@438153d157e)
Using time:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/time/0)
Using timeout:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/timeout/0)
Using tsort:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/tsort/0)
Using uri:0 (/Users/kuwabara.masataka/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/rbs-3.4.4/stdlib/uri/0)
Using yard:0.9 (yard@438153d157e)
It's done! 104 gems' RBSs now installed.

So, could you try the latest RBS?

BTW, I'm still not sure which commit of RBS solved this problem...

Grohden commented 5 months ago

hey @pocke, thank you! sorry for taking so long to test this

Yess, 3.4.4 works for me 🙏, feel free to close this one if you think its done

pocke commented 5 months ago

Thank you for confirming it 👍 I'll close this issue.