search

jcs / rubywarden

An unofficial, mostly Bitwarden-compatible API server written in Ruby (Sinatra and ActiveRecord)
ISC License
593 stars 49 forks source link

Migration from bitwarden-ruby not working #63

Closed siddhantgoel closed 6 years ago

siddhantgoel commented 6 years ago

I tried to migrate from bitwarden-ruby this morning following the directions from https://github.com/jcs/rubywarden/blob/master/AR-MIGRATE.md .

bundle --with migrate worked without problems, after which I ran into the following exception.

bitwarden@pi:~/bitwarden-ruby$ bundle exec ruby tools/migrate_to_ar.rb -e production
/home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:91:in `initialize': SQLite3::SQLException: no such table: folders: SELECT COUNT(*) FROM "folders" (ActiveRecord::StatementInvalid)
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:91:in `new'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:91:in `prepare'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/sqlite3_adapter.rb:212:in `block (2 levels) in exec_query'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6/lib/active_support/dependencies/interlock.rb:46:in `block in permit_concurrent_loads'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6/lib/active_support/concurrency/share_lock.rb:185:in `yield_shares'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6/lib/active_support/dependencies/interlock.rb:45:in `permit_concurrent_loads'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/sqlite3_adapter.rb:209:in `block in exec_query'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract_adapter.rb:613:in `block (2 levels) in log'
        from /home/bitwarden/.rvm/rubies/ruby-2.4.4/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract_adapter.rb:612:in `block in log'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activesupport-5.1.6/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract_adapter.rb:604:in `log'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/sqlite3_adapter.rb:208:in `exec_query'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/database_statements.rb:371:in `select'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/database_statements.rb:42:in `select_all'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/query_cache.rb:97:in `select_all'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/activerecord-5.1.6/lib/active_record/connection_adapters/abstract/database_statements.rb:49:in `select_one'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/yaml_db-0.7.0/lib/yaml_db/serialization_helper.rb:198:in `table_record_count'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/yaml_db-0.7.0/lib/yaml_db/serialization_helper.rb:172:in `dump_table'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/yaml_db-0.7.0/lib/yaml_db/serialization_helper.rb:158:in `block in dump'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/yaml_db-0.7.0/lib/yaml_db/serialization_helper.rb:156:in `each'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/yaml_db-0.7.0/lib/yaml_db/serialization_helper.rb:156:in `dump'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/yaml_db-0.7.0/lib/yaml_db/serialization_helper.rb:18:in `block in dump'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/yaml_db-0.7.0/lib/yaml_db/serialization_helper.rb:17:in `open'
        from /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/yaml_db-0.7.0/lib/yaml_db/serialization_helper.rb:17:in `dump'
        from tools/migrate_to_ar.rb:50:in `<main>'

Not sure what I can do to fix this. Any help would be appreciated. Thank you!

pc-coholic commented 6 years ago

I have to same problem - but to be fair, I am not even able to migrate to AR with the very first commit of AR (5ab123d499d9cd43ef4ee182c78db19e46de3201).

The last working version for me is 0f3da81afa7782eab397a036bdcb2dc2edb0e752 - but that one is obviously still without AR-support.

tanshu commented 6 years ago

The problem is with the location of database file. Before the move to AR it was located at db/production.sqlite3. Now it is expected to be at db/production/production.sqlite3. Just copy the old database into the production directory and the migrate should work properly.

pc-coholic commented 6 years ago

Yes, this solves the issue...

Perhaps the migration script could take care of this automatically - or at least we should add a more prominent hint into the migration-documentation...

I know that the new location is mentioned at the bottom of the migration-guide - but still, it was not clear enough for me to realize, that I need to manually adjust the paths... :-)

siddhantgoel commented 6 years ago

That did solve the issue (thanks!). But now the syncing doesn't seem to work. I see the following exception in the logs -

2018-08-10 11:44:24 - ArgumentError - comparison of Integer with Time failed:
        /home/bitwarden/bitwarden-ruby/lib/helpers/request_helpers.rb:23:in `>='
        /home/bitwarden/bitwarden-ruby/lib/helpers/request_helpers.rb:23:in `device_from_bearer'
        /home/bitwarden/bitwarden-ruby/lib/routes/api.rb:77:in `block (2 levels) in registered'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1635:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1635:in `block in compile!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:992:in `block (3 levels) in route!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1011:in `route_eval'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:992:in `block (2 levels) in route!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1040:in `block in process_route'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1038:in `catch'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1038:in `process_route'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:990:in `block in route!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:989:in `each'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:989:in `route!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1097:in `block in dispatch!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `block in invoke'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `catch'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `invoke'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1094:in `dispatch!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:924:in `block in call!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `block in invoke'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `catch'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `invoke'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:924:in `call!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:913:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-protection-2.0.3/lib/rack/protection/xss_header.rb:18:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-protection-2.0.3/lib/rack/protection/path_traversal.rb:16:in `call'                                                                                                                   
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-protection-2.0.3/lib/rack/protection/json_csrf.rb:26:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-protection-2.0.3/lib/rack/protection/base.rb:50:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-protection-2.0.3/lib/rack/protection/base.rb:50:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-protection-2.0.3/lib/rack/protection/frame_options.rb:31:in `call'                                                                                                                    
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-2.0.5/lib/rack/logger.rb:15:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-2.0.5/lib/rack/common_logger.rb:33:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:231:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:224:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:194:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1958:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1502:in `block in call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1729:in `synchronize'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1502:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-2.0.5/lib/rack/handler/webrick.rb:86:in `service'
        /home/bitwarden/.rvm/rubies/ruby-2.4.4/lib/ruby/2.4.0/webrick/httpserver.rb:140:in `service'
        /home/bitwarden/.rvm/rubies/ruby-2.4.4/lib/ruby/2.4.0/webrick/httpserver.rb:96:in `run'
        /home/bitwarden/.rvm/rubies/ruby-2.4.4/lib/ruby/2.4.0/webrick/server.rb:308:in `block in start_thread'
192.168.178.20 - - [10/Aug/2018:11:44:24 +0000] "GET /api/sync HTTP/1.1" 500 30 0.2418
tanshu commented 6 years ago

I got this error too, but only on my chrome extension. The mobile app synced properly. Somehow the error went away on its own after a few restarts. You can try doing that a few times.

jcs commented 6 years ago

Can you try the new migration script? You may want to delete the whole db/production/ database and re-run the tool to migrate your old db/production.sqlite3 database.

siddhantgoel commented 6 years ago

@jcs Thanks. The new migration script works. After this migration, the mobile app sync fine but the firefox extension doesn't. When I click on "Sync Vault Now", the extension shows an error message "Syncing failed", and the server logs contain the following exception -

2018-08-13 07:46:23 - ArgumentError - comparison of Integer with Time failed:
        /home/bitwarden/bitwarden-ruby/lib/helpers/request_helpers.rb:23:in `>='
        /home/bitwarden/bitwarden-ruby/lib/helpers/request_helpers.rb:23:in `device_from_bearer'
        /home/bitwarden/bitwarden-ruby/lib/routes/api.rb:77:in `block (2 levels) in registered'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1635:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1635:in `block in compile!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:992:in `block (3 levels) in route!'                                                                                                                      
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1011:in `route_eval'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:992:in `block (2 levels) in route!'                                                                                                                      
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1040:in `block in process_route'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1038:in `catch'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1038:in `process_route'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:990:in `block in route!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:989:in `each'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:989:in `route!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1097:in `block in dispatch!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `block in invoke'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `catch'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `invoke'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1094:in `dispatch!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:924:in `block in call!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `block in invoke'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `catch'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1076:in `invoke'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:924:in `call!'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:913:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-protection-2.0.3/lib/rack/protection/xss_header.rb:18:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-protection-2.0.3/lib/rack/protection/path_traversal.rb:16:in `call'                                                                                                                   
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-protection-2.0.3/lib/rack/protection/json_csrf.rb:26:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-protection-2.0.3/lib/rack/protection/base.rb:50:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-protection-2.0.3/lib/rack/protection/base.rb:50:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-protection-2.0.3/lib/rack/protection/frame_options.rb:31:in `call'                                                                                                                    
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-2.0.5/lib/rack/logger.rb:15:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-2.0.5/lib/rack/common_logger.rb:33:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:231:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:224:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:194:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1958:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1502:in `block in call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1729:in `synchronize'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/sinatra-2.0.3/lib/sinatra/base.rb:1502:in `call'
        /home/bitwarden/.rvm/gems/ruby-2.4.4/gems/rack-2.0.5/lib/rack/handler/webrick.rb:86:in `service'
        /home/bitwarden/.rvm/rubies/ruby-2.4.4/lib/ruby/2.4.0/webrick/httpserver.rb:140:in `service'
        /home/bitwarden/.rvm/rubies/ruby-2.4.4/lib/ruby/2.4.0/webrick/httpserver.rb:96:in `run'
        /home/bitwarden/.rvm/rubies/ruby-2.4.4/lib/ruby/2.4.0/webrick/server.rb:308:in `block in start_thread'
192.168.178.20 - - [13/Aug/2018:07:46:23 +0000] "GET /api/sync HTTP/1.1" 500 30 0.0152

I also tried restarting the server and the browser a few times as @tanshu suggested. But the problem still remains.

jcs commented 6 years ago

That looks like your device record has no token_expires_at value for some reason. Can you try logging out in the extension and logging back in? That will trigger Device#regenerate_tokens! which will set token_expires_at.

siddhantgoel commented 6 years ago

Awesome, this worked. Thanks!