comeara / pillar

Pillar manages migrations for your Cassandra data stores.
https://github.com/comeara/pillar
MIT License
111 stars 64 forks source link

Introduce execution stages for up and down queries #19

Closed sadowskik closed 8 years ago

sadowskik commented 9 years ago

Some of the queries cannot be executed via cassandra driver in a single go. GRANT query is one of the examples. In order to effectively setup some grants the one need to run separate query, for each GRANT:

session.execute("GRANT MODIFY ...");
session.execute("GRANT MODIFY ...");
session.execute("GRANT MODIFY ...");

This pull request introduces the notion of stages. Each migration section (up or down) can be segregated with the --stage keyword. Queries defined within each stage are executed in a single go (single session.execute(...) invocation). Example:

-- description: created schema
-- authoredAt: 1370023265
-- up:

-- stage: 1
CREATE TABLE events (
  batch_id text,
  occurred_at uuid,
  event_type text,
  payload blob,
  PRIMARY KEY (batch_id, occurred_at, event_type)
)

-- stage: 2
GRANT MODIFY ON events TO test_user 

-- stage: 3
GRANT SELECT ON events TO test_user 

This migration will be applied by running session.execute() three times:

  1. session.execute("CREATE TABLE events (...")
  2. session.execute("GRANT MODIFY ON...")
  3. session.execute("GRANT SELECT ON...")
PeterLappo commented 9 years ago

Good idea. Was thinking of something similar for bulk loads but separating each command with a semi-colon.

sadowskik commented 9 years ago

Approach with a dedicated stage section allows you to execute many semi-colon separated statements within a single stage:

-- description: created schema
-- authoredAt: 1370023265
-- up:

-- stage: 1
CREATE TABLE events (
  batch_id text,
  occurred_at uuid,
  event_type text,
  payload blob,
  PRIMARY KEY (batch_id, occurred_at, event_type)
);

DROP TABLE old_events;

--stage: 2
GRANT MODIFY ON events TO test_user 

--stage: 3
GRANT SELECT ON events TO test_user

will be executed as:

  1. session.execute("CREATE TABLE events ...; DROP TABLE old_events;")
  2. session.execute("GRANT MODIFY ON...")
  3. session.execute("GRANT SELECT ON...")
PeterLappo commented 9 years ago

Nice

magro commented 9 years ago

Great, I'd love to see this being merged!

magro commented 8 years ago

@comeara Any chance that this will get merged?

comeara commented 8 years ago

This has been merged to master and will be released in 2.3.

comeara commented 8 years ago

2.3.0 was released today.