search

soundasleep / statgit2

Generate Git repository statistics (version 2)
4 stars 3 forks source link

Database error? #29

Closed mzampare closed 2 years ago

mzampare commented 3 years ago

Hi all,

We imported the latest version of a larger project (> 1300 files in the first commit) into a git repository. statgit has trouble with that repository and produces a database error because it creates a too long list of parameters in its query.

This is the statgit command:

bundle exec ruby generate.rb --workspace cop.workspace/ \ --urlhttps://gitlab.eso.org/dfs/cop \ --database db/databasecop.sqlite3

Which eventually produces the following output (some line breaks inserted in the failing SQL statement, and long list (>1300) of “?” shortened):

[…]

I, [2021-11-09T08:00:47.799586 #986824]  INFO -- : Generating reports...
I, [2021-11-09T08:00:48.114607 #986824]  INFO -- : >> AuthorsReport
I, [2021-11-09T08:00:54.376328 #986824]  INFO -- : >> ChurnReport
I, [2021-11-09T08:00:54.401629 #986824]  INFO -- : >> ConfigReport
I, [2021-11-09T08:00:54.407316 #986824]  INFO -- : >> FilesReport
/diska/statgit2/vendor/bundle/ruby/2.5.0/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:91:in `initialize': SQLite3::SQLException: too many SQL variables:
  SELECT "file_todos".* FROM "file_todos"
  WHERE "file_todos"."commit_file_id" IN (?, ?,
  [...]) (ActiveRecord::StatementInvalid)
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:91:in `new'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/sqlite3-1.3.11/lib/sqlite3/database.rb:91:in `prepare'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/sqlite3_adapter.rb:216:in `block (2 levels) in exec_query'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/sqlite3_adapter.rb:213:in `block in exec_query'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
    from /usr/share/ruby/monitor.rb:226:in `mon_synchronize'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.0/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/sqlite3_adapter.rb:212:in `exec_query'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/database_statements.rb:465:in `select'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/database_statements.rb:57:in `select_all'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/query_cache.rb:101:in `select_all'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/querying.rb:41:in `find_by_sql'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation.rb:554:in `block in exec_queries'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation.rb:578:in `skip_query_cache_if_necessary'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation.rb:542:in `exec_queries'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation.rb:414:in `load'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader/association.rb:106:in `records_for'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader/association.rb:98:in `block in load_records'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader/association.rb:97:in `each'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader/association.rb:97:in `each_slice'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader/association.rb:97:in `each'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader/association.rb:97:in `flat_map'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader/association.rb:97:in `load_records'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader/association.rb:19:in `run'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader.rb:142:in `block (2 levels) in preloaders_for_one'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader.rb:140:in `each'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader.rb:140:in `map'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader.rb:140:in `block in preloaders_for_one'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader.rb:139:in `each'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader.rb:139:in `flat_map'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader.rb:139:in `preloaders_for_one'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader.rb:106:in `preloaders_on'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader.rb:93:in `block in preload'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader.rb:92:in `each'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader.rb:92:in `flat_map'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/associations/preloader.rb:92:in `preload'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation.rb:562:in `block (2 levels) in exec_queries'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation.rb:560:in `each'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation.rb:560:in `block in exec_queries'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation.rb:578:in `skip_query_cache_if_necessary'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation.rb:542:in `exec_queries'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/association_relation.rb:34:in `exec_queries'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation.rb:414:in `load'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation.rb:200:in `records'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/relation/delegation.rb:41:in `each'
    from /diska/statgit2/transforms/files_with_revisions_and_sizes.rb:13:in `map'
    from /diska/statgit2/transforms/files_with_revisions_and_sizes.rb:13:in `call'
    from /diska/statgit2/models/commit.rb:106:in `files_with_revisions_and_sizes'
    from /diska/statgit2/transforms/files_with_most_revisions.rb:11:in `call'
    from /diska/statgit2/models/repository.rb:86:in `files_with_most_revisions'
    from /diska/statgit2/helpers/report_helper.rb:109:in `block in table'
    from /usr/share/ruby/benchmark.rb:308:in `realtime'
    from /diska/statgit2/helpers/report_helper.rb:108:in `table'
    from (haml):38:in `block in render'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/haml-4.0.7/lib/haml/engine.rb:129:in `eval'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/haml-4.0.7/lib/haml/engine.rb:129:in `render'
    from /diska/statgit2/helpers/report_helper.rb:128:in `render_template'
    from /diska/statgit2/reports/abstract_report.rb:26:in `block in render_output'
    from /diska/statgit2/helpers/report_helper.rb:129:in `block in render_template'
    from (haml):14:in `block in render'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/haml-4.0.7/lib/haml/engine.rb:129:in `eval'
    from /diska/statgit2/vendor/bundle/ruby/2.5.0/gems/haml-4.0.7/lib/haml/engine.rb:129:in `render'
    from /diska/statgit2/helpers/report_helper.rb:128:in `render_template'
    from /diska/statgit2/reports/abstract_report.rb:25:in `render_output'
    from /diska/statgit2/reports/abstract_report.rb:14:in `create_file!'
    from /diska/statgit2/reports/abstract_report.rb:21:in `generate!'
    from /diska/statgit2/services/generate_reports.rb:19:in `block in call'
    from /diska/statgit2/services/generate_reports.rb:14:in `each'
    from /diska/statgit2/services/generate_reports.rb:14:in `call'
    from /diska/statgit2/services/command_line_runner.rb:15:in `call'
    from generate.rb:6:in `<main>'

thanks for any help.

soundasleep commented 3 years ago

In order to try and debug this, I've updated the Gem and its dependencies to

It looks like it's due to commit.files.preload(:file_todos), at the time that it's generating the reports, so I'm wondering if the latest versions of Activerecord or Sqlite may have fixed your issue. Can you pull the latest master and try?

Alternatively you could get rid of :file_todos on line 12 of files_with_revisions_and_sizes.rb, it should still work fine but it might be a lot slower...

mzampare commented 2 years ago

The latest changes (which we deployed) fixed the problem, thanks a lot. Merry Christmas & Happy New year.

mzampare commented 2 years ago

Fixed.