pact-foundation / pact_broker

Enables your consumer driven contracts workflow
http://pactflow.io
MIT License
708 stars 177 forks source link

feat: reduce contention when updating the contract_data_updated_at field for integrations #671

Closed bethesque closed 8 months ago

bethesque commented 8 months ago

Reduce contention updating the contract_data_updated_at for integrations, which is causing some responses to be slow while locks are waited for.

This is tricky to test in a unit test, especially as the database we use for our unit tests (Sqlite) doesn't support SKIP LOCKED, so I manually tested it using the following code.

script/docker/db-rm.sh # remove any existing db
script/docker/db-start.sh
script/dev/console.rb postgres://postgres:postgres@localhost/postgres

td = PactBroker::Test::TestDataBuilder.new
td.create_consumer("A").create_provider("B").create_integration
td.create_consumer("C").create_provider("D").create_integration
td.create_consumer("E").create_provider("F").create_integration

things1 = [
    OpenStruct.new(consumer: OpenStruct.new(id: 1), provider: OpenStruct.new(id: 2)),
    OpenStruct.new(consumer: OpenStruct.new(id: 3), provider: OpenStruct.new(id: 4))
]

connection.transaction do
       # update the first and second integrations
    PactBroker::Integrations::Repository.new.set_contract_data_updated_at_for_multiple_integrations(things1)
    sleep 15
end

PactBroker::Integrations::Integration.order(:id).all

In another shell, within 15 seconds:

script/dev/console.rb postgres://postgres:postgres@localhost/postgres

# attempt to update second and third integrations
# only the third integration will be updated, as the second one is locked by the first transaction
things2 = [
    OpenStruct.new(consumer: OpenStruct.new(id: 3), provider: OpenStruct.new(id: 4)),
    OpenStruct.new(consumer: OpenStruct.new(id: 5), provider: OpenStruct.new(id: 6))
]

connection.transaction do
    PactBroker::Integrations::Repository.new.set_contract_data_updated_at_for_multiple_integrations(things2)
end

The second script will complete immediately, without waiting for the first one to sleep its 15 seconds. The first script will update integrations 1 and 2, while the second script will attempt to update 2 and 3, but will only actually update 3.

bethesque commented 8 months ago

Note that we do run all our tests against Postgres (multiple versions) as well as Sqlite (and MySQL, but let's not talk about that) in CI.

bethesque commented 8 months ago

Major release for what hairy bits? What is backwards incompatible?

vwong commented 8 months ago

I thought you were hinting at dropping support for other databases