search

zilverline / sequent

CQRS & event sourcing framework for Ruby
https://www.sequent.io
MIT License
541 stars 58 forks source link

Relax parser version to allow the latest Rubocop version #410

Closed evsasse closed 5 months ago

evsasse commented 5 months ago

Similar to what we've seen on https://github.com/zilverline/sequent/issues/268 and https://github.com/zilverline/sequent/pull/374

lvonk commented 5 months ago

Ah, the several gemfiles also needs updating apparently, or at least call bundle with those files. Can you add that to this PR?

evsasse commented 5 months ago

Ah, the several gemfiles also needs updating apparently, or at least call bundle with those files. Can you add that to this PR?

bundle install
bundle install --gemfile=gemfiles/ar_6_0.gemfile 
bundle install --gemfile=gemfiles/ar_6_1.gemfile
bundle install --gemfile=gemfiles/ar_7_0.gemfile
bundle install --gemfile=gemfiles/ar_7_1.gemfile

Right? Updated 👍

evsasse commented 5 months ago

Also created a different branch on my fork, if anyone else needs the 7.0.0 release, just with this dependency flexibility, to get a Rubocop update out of the way. Without the upcoming pg_current_snapshot and xact_id changes that are already on the main branch here.

gem "sequent", git: "https://github.com/evsasse/sequent-parser-3.3", branch: "safe-7.0"
lvonk commented 5 months ago

Hi @evsasse thanks! Just wondering if you run into issues with the new main branch and haven't migrated yet?

evsasse commented 5 months ago

Yeah had some trouble that seem to be related to changes added to the main branch, after the 7.0.0 release. Doesn't seem like anything major, but wanted to just get this Rubocop update out of the way...

lvonk commented 5 months ago

Then it seems we've hit some snag on this xact_id, that maybe needed a migration? Didn't dive super deep into it. I then felt more confident to wait for a 7.1/8.0(?) release, so we have a more official/final upgrade guide.

What snag did you hit? It shouldn't need a manual migration. Can you share the error message you got?

evsasse commented 5 months ago

What snag did you hit? It shouldn't need a manual migration. Can you share the error message you got?

I have this "seed" script for the dev env here that basically runs all of these, in order: db:drop, db:create, db:schema:load, sequent:db:create_event_store, sequent:db:create_view_schema, sequent:migrate:online, sequent:migrate:offline, db:seed.

Looks like somewhere on sequent:migrate:online it fails while trying to replay the events, on this newly recreated empty database. Pointing out this xact_id column not existing 🤔.

I, [2024-03-29T12:03:01.066688 #196]  INFO -- : Number of groups 4096
  Sequent::Migrations::Versions Load (0.6ms)  SELECT "sequent_versions".* FROM "sequent_versions" WHERE "sequent_versions"."status" IN ($1, $2, $3) ORDER BY "sequent_versions"."version" ASC LIMIT $4 /*application:App,line:/bundle/ruby/3.3.0/gems/activerecord-7.1.3.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:64:in `internal_exec_query'*/  [["status", 1], ["status", 2], ["status", 3], ["LIMIT", 1]]
I, [2024-03-29T12:03:01.080325 #196]  INFO -- : groups: 4096
I, [2024-03-29T12:03:01.080448 #196]  INFO -- : Start replaying events
  TRANSACTION (0.4ms)  BEGIN
  TRANSACTION (0.3ms)  BEGIN
  TRANSACTION (0.3ms)  BEGIN
   (8.1ms)  declare cursor_807f5fb9eb784c3b8aed9203ca21f7a9 no scroll cursor for SELECT id, event_type, event_json, sequence_number FROM "event_records" WHERE "event_records"."event_type" IN (... REDACTING THIS ...) AND (substring(aggregate_id::text from 1 for 3) in ('001')
) AND (xact_id IS NULL OR xact_id < 11958) ORDER BY aggregate_id ASC, sequence_number ASC /*application:App,line:/bundle/ruby/3.3.0/gems/postgresql_cursor-0.6.8/lib/postgresql_cursor/cursor.rb:250:in `open'*/
   (6.9ms)  declare cursor_226ae1ea82054f7eb03b5487daf2e2ec no scroll cursor for SELECT id, event_type, event_json, sequence_number FROM "event_records" WHERE "event_records"."event_type" IN (... REDACTING THIS ...) AND (substring(aggregate_id::text from 1 for 3) in ('000')
) AND (xact_id IS NULL OR xact_id < 11958) ORDER BY aggregate_id ASC, sequence_number ASC /*application:App,line:/bundle/ruby/3.3.0/gems/postgresql_cursor-0.6.8/lib/postgresql_cursor/cursor.rb:250:in `open'*/
   (8.5ms)  declare cursor_eb01201117314ddebe165678d10514ea no scroll cursor for SELECT id, event_type, event_json, sequence_number FROM "event_records" WHERE "event_records"."event_type" IN (... REDACTING THIS ...) AND (substring(aggregate_id::text from 1 for 3) in ('002')
) AND (xact_id IS NULL OR xact_id < 11958) ORDER BY aggregate_id ASC, sequence_number ASC /*application:App,line:/bundle/ruby/3.3.0/gems/postgresql_cursor-0.6.8/lib/postgresql_cursor/cursor.rb:250:in `open'*/
  TRANSACTION (0.6ms)  BEGIN
  TRANSACTION (1.2ms)  ROLLBACK
E, [2024-03-29T12:03:01.147130 #264] ERROR -- : Replaying failed for ids: ^001 - 001
E, [2024-03-29T12:03:01.147194 #264] ERROR -- : +++++++++++++++ ERROR +++++++++++++++
E, [2024-03-29T12:03:01.147232 #264] ERROR -- : PG::UndefinedColumn: ERROR:  column "xact_id" does not exist
LINE 2: ) AND (xact_id IS NULL OR xact_id < 11958) ORDER BY aggregat...
               ^
/bundle/ruby/3.3.0/gems/activerecord-7.1.3.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:55:in `block (2 levels) in raw_execute'
...
/bundle/ruby/3.3.0/gems/postgresql_cursor-0.6.8/lib/postgresql_cursor/cursor.rb:82:in `each'
/bundle/ruby/3.3.0/bundler/gems/sequent-dbaa11ffbf76/lib/sequent/core/event_store.rb:163:in `replay_events_from_cursor'
/bundle/ruby/3.3.0/bundler/gems/sequent-dbaa11ffbf76/lib/sequent/migrations/view_schema.rb:366:in `replay_events'
/bundle/ruby/3.3.0/bundler/gems/sequent-dbaa11ffbf76/lib/sequent/migrations/view_schema.rb:336:in `block (4 levels) in replay!'
/bundle/ruby/3.3.0/bundler/gems/sequent-dbaa11ffbf76/lib/sequent/util/timer.rb:8:in `time'
/bundle/ruby/3.3.0/bundler/gems/sequent-dbaa11ffbf76/lib/sequent/migrations/view_schema.rb:335:in `block (3 levels) in replay!'
/bundle/ruby/3.3.0/gems/parallel-1.24.0/lib/parallel.rb:627:in `call_with_index'
...
/bundle/ruby/3.3.0/gems/parallel-1.24.0/lib/parallel.rb:300:in `map_with_index'
/bundle/ruby/3.3.0/bundler/gems/sequent-dbaa11ffbf76/lib/sequent/migrations/view_schema.rb:327:in `block (2 levels) in replay!'
/bundle/ruby/3.3.0/bundler/gems/sequent-dbaa11ffbf76/lib/sequent/util/timer.rb:8:in `time'
/bundle/ruby/3.3.0/bundler/gems/sequent-dbaa11ffbf76/lib/sequent/migrations/view_schema.rb:321:in `block in replay!'
/bundle/ruby/3.3.0/bundler/gems/sequent-dbaa11ffbf76/lib/sequent/migrations/view_schema.rb:448:in `with_sequent_config'
/bundle/ruby/3.3.0/bundler/gems/sequent-dbaa11ffbf76/lib/sequent/migrations/view_schema.rb:318:in `replay!'
/bundle/ruby/3.3.0/bundler/gems/sequent-dbaa11ffbf76/lib/sequent/migrations/view_schema.rb:206:in `migrate_online'
/bundle/ruby/3.3.0/bundler/gems/sequent-dbaa11ffbf76/lib/sequent/rake/migration_tasks.rb:175:in `block (3 levels) in register_tasks!'
/bundle/ruby/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `block in execute'
...
/home/web/lib/tasks/sequent.rake:40:in `block in <main>'

PS: that Rubocop update is already out of the way here, so LMK how I can be more helpful here, if you need more details, or an extra hand.

lvonk commented 5 months ago

Ah yes, a migration is needed and missing in the changelog. Will add that. What you need todo is this:

👉🏼 Make sure you have the latest copy of sequent_schema in your own project so setting up from scratch works.

For migrating your existing environments execute the following sql: (replace SCHEMA_NAME with the name of the schema of the event_store in your database):

BEGIN;
ALTER TABLE SCHEMA_NAME.event_records ADD COLUMN xact_id bigint;
COMMIT;

BEGIN;
# SET max_parallel_maintenance_workers = 8; # optionally set this depending on size of your event_records
# ALTER TABLE SCHEMA_NAME.event_records SET (parallel_workers = 8); # optionally set this depending on size of your event_records
CREATE INDEX event_records_xact_id_idx ON SCHEMA_NAME.event_records (xact_id) WHERE xact_id IS NOT NULL;
# ALTER TABLE SCHEMA_NAME.event_records RESET (parallel_workers); # optionally set this depending on size of your event_records
COMMIT;

ALTER TABLE SCHEMA_NAME.event_records ALTER COLUMN xact_id SET DEFAULT pg_current_xact_id()::text::bigint;