What volumes should I mount on my docker-compose to persist data

[christopher-francisco]

I'm deploying pact broker docker but I don't see docs on what volumes should I mount to persist data between container restarts/destroy of pact-broker container. What volume should I mount?

For the postgres container, I assume it's pgdata:/var/lib/postgresql/data postgres same as the container docs say, can we confirm this?

[bethesque]

Yes. There's nothing else stored on the file system.

[Dutzu]

@bethesque Could you also mention what paths need to be mounted as volumes for working with sqlite?

[bethesque]

Where ever you put your Sqlite database. Only you know that! It's the value that you'll have supplied to thePACT_BROKER_DATABASE_NAME.

[Dutzu]

Awesome. That information would have been nice to have in the readme.

After digging around and poking through the docker image I found that the app itself is under /home/app/pact_broker and I had used the value provided in the readme as the PACT_BROKER_DATABASE_NAME and my problem was that it's a file that doesn't exist and will be created upon first startup and it is in the folder where the rest of the app is so mounting a docker volume is not so straight forward.

Probably a solution would be to put the PACT_BROKER_DATABASE_NAME=/some_sub_folder_to_mount/pact_broker.sqlite

As a workaround I used a binding instead of a volume like so:

version: '3.2'
services:
  pactbroker:
    image: dius/pact-broker
    ports:
      - "7000:80"
    environment:
      - PACT_BROKER_DATABASE_ADAPTER=sqlite
      - PACT_BROKER_DATABASE_NAME=pact_broker.sqlite
      - PACT_BROKER_BASIC_AUTH_USERNAME=*****
      - PACT_BROKER_BASIC_AUTH_PASSWORD=*****
      - PACT_BROKER_BASIC_AUTH_READ_ONLY_USERNAME=****
      - PACT_BROKER_BASIC_AUTH_READ_ONLY_PASSWORD=****
      - PACT_BROKER_PUBLIC_HEARTBEAT=true
    volumes:
      - type: bind
        source: "/data/pact_broker.sqlite"
        target: "/home/app/pact_broker/pact_broker.sqlite"
    restart: always
networks:
  default:
      ipam:
        driver: default
        config:
          - subnet: 10.222.0.0/24

[bethesque]

Can I ask what problem you're trying to solve? It's not normal to use an SQLite database for ongoing usage for this application. It's only used for running the test suites, or starting up the broker for demo purposes.

[Dutzu]

We've been using it like this for over a year now, without a problem, until for some reason the container restarted a couple of times and we lost contracts published and it was annoying to have to re-run all consumers again to re-publish all the contracts.

I get that this is not the way it was intended, and would love to hear what are the reasons a postgres or mysql would be mandatory......but up until know I thought, if it gets the job done, I don't see why I would have to add the extra cost of infrastructure for a database engine or the hassle to monitor the db to see that it's up.

Tbh, I would love to have the broker as a serverless function and keep the state in a file based db or managed serverless db service if possible.

At some point, it might be the only thing left in my application's landscape that is still requiring a VM to live on. (well, I guess I could use something like aws fargate....but it feels like overkill for just the broker)

[mefellows]

I guess you could look at https://github.com/pact-foundation/pact_broker-serverless and use it with EFS (https://aws.amazon.com/blogs/compute/using-amazon-efs-for-aws-lambda-in-your-serverless-applications/)?

For completeness, there is also https://pactflow.io/ (but I understand that may not be an option for you).

[bethesque]

Well, you've made me re-evaluate my thoughts on using SQLite for prod. I just did some googling and came across this page https://www.sqlite.org/whentouse.html

Situations Where A Client/Server RDBMS May Work Better Client/Server Applications If there are many client programs sending SQL to the same database over a network, then use a client/server database engine instead of SQLite. SQLite will work over a network filesystem, but because of the latency associated with most network filesystems, performance will not be great. Also, file locking logic is buggy in many network filesystem implementations (on both Unix and Windows). If file locking does not work correctly, two or more clients might try to modify the same part of the same database at the same time, resulting in corruption. Because this problem results from bugs in the underlying filesystem implementation, there is nothing SQLite can do to prevent it. A good rule of thumb is to avoid using SQLite in situations where the same database will be accessed directly (without an intervening application server) and simultaneously from many computers over a network.

High Concurrency SQLite supports an unlimited number of simultaneous readers, but it will only allow one writer at any instant in time. For many situations, this is not a problem. Writers queue up. Each application does its database work quickly and moves on, and no lock lasts for more than a few dozen milliseconds. But there are some applications that require more concurrency, and those applications may need to seek a different solution.

Is the data separated from the application by a network? → choose client/server

I guess if your usage is quite light, then the likelihood of having concurrent transactions is quite low. I've been using it as the test database since it was first written, and I know that on start up, I occasionally get "file locked" errors when it gets hit too hard, but you probably wouldn't be starting and stopping it that often. If it's worked for you, then that's great!

[mefellows]

I've found tools like terraform have exposed concurrency issues when using pact broker backed by sqllite which makes sense given it is going to do a lot of writing. Choose what makes sense for you I guess!

[bethesque]

I've been getting this error a LOT on my local development machine recently. I don't know how this doesn't happen in production!

16:20:21 E {cognito_sub: 96b88866-8c12-489e-9aec-5e7b3850c2eb} -- SQLite3::BusyException: database is locked: SELECT * FROM `pact_versions` WHERE `id` = 677
16:20:21 E {cognito_sub: 96b88866-8c12-489e-9aec-5e7b3850c2eb} -- Error reference qupXcXPHqI -- Exception: Sequel::DatabaseError: SQLite3::BusyException: database is locked

[Dutzu]

well....context might be different. On your local machine might there by other software that would get a lock on that file? antivirus, IDE, etc....

[bethesque]

It happens when the React UI recompiles and it refreshes in the browser, so a bunch of requests fire off at the same time. Now I think about it, you're on the old server side rendered UI, so you wouldn't get situations with a heap concurrent requests from a browser. That probably explains it.