Losing databases upon deployment or restart to fly.io

[CharlesIrvineKC]

I am deploying my Phoenix app to fly.io. On either a new deployment or even a restart, I seem to be losing all of the contents in my CubDB databases (I have several of them). I'm pretty new to fly.io, so I don't know how to provide anymore info at this point. Can you think of anything that might be causing this? I

The problem does not occur when I run my app locally.

I'll keep digging. Feel free to close the issue if you want.

[DavidOliver]

Are you using a Fly Volume to store the cubdb file?

[CharlesIrvineKC]

No, not unless I'm doing it by accident, since I don't know what a Fly Volume is. I just create the database with:

I just do this:

  @doc false
  def init(_init_arg) do
    {:ok, user_task_db} = CubDB.start_link(data_dir: "database/user_task_db")
    {:ok, completed_process_db} = CubDB.start_link(data_dir: "database/completed_process_db")
    {:ok, process_model_db} = CubDB.start_link(data_dir: "database/process_model_db")
    {:ok, bpm_application_db} = CubDB.start_link(data_dir: "database/bpm_application_db")
    {:ok, process_state_db} = CubDB.start_link(data_dir: "database/process_state_db")

    initial_state = %{
      active_process_groups: %{},
      active_processes: %{},
      restart_state_cache: %{},
      user_task_db: user_task_db,
      completed_process_db: completed_process_db,
      process_model_db: process_model_db,
      bpm_application_db: bpm_application_db,
      process_state_db: process_state_db
    }

    Logger.info("Process service initialized")

    {:ok, initial_state}
  end

FYI - I just now figured out how to get a shell prompt on my server.

[CharlesIrvineKC]

I can see now from my fly shell prompt, that the database files are being reset to empty on each restart. Specifically, I do an "ls" and the file size of 0.cub goes back to a size of 1038 on each restart.

[CharlesIrvineKC]

I just followed your link to "Fly Volume". I gather that is my problem, i.e. that I am not using Fly Volumes. Correct?

[DavidOliver]

Yes.

[CharlesIrvineKC]

Thanks for the tip. I've been trying to do this, but without any luck so far.

First, I created a volume like this:

fly volumes create database

Then in my fly.toml file, I added:

[mounts]
  source = "database"
  destination = '/app/database'

The directory does get created it seems:

root@7811773f270e58:/app# ls
bin  database  erts-14.2.5  lib  releases

However, I'm still losing my database contents on every restart. Hmmmm....

[optikfluffel]

@CharlesIrvineKC I'm doing more or less the same with a Fly Volume + CubDB and it seems to work fine. The only obvious difference I'm seeing at first glance is I'm not mounting it inside the /app directory. Could you check if mounting to /database or something similar might help?

[CharlesIrvineKC]

@optikfluffel Quick question. In your calls to CubDB.start_link/1, does your path start with a leading slash, or without, like this:

 {:ok, user_task_db} = CubDB.start_link(data_dir: "database/user_task_db")

@CharlesIrvineKC I'm doing more or less the same with a Fly Volume + CubDB and it seems to work fine. The only obvious difference I'm seeing at first glance is I'm not mounting it inside the /app directory. Could you check if mounting to /database or something similar might help?

[optikfluffel]

@CharlesIrvineKC with a leading slash, yes

Oh and another difference to your example came to mind, I start my CubDB as part of the Application Supervision Tree in application.ex and not manually with CubDB.start_link. But that shouldn't make a difference I guess.

[CharlesIrvineKC]

@optikfluffel Would you mind showing me your application.ex?

[optikfluffel]

@CharlesIrvineKC sure thing:

defmodule Fluffel.Application do
  @moduledoc false

  use Application

  @impl Application
  def start(_type, _args) do
    if Application.get_env(:sentry, :environment_name) == :prod do
      {:ok, _pid} = Logger.add_backend(Sentry.LoggerBackend)
    end

    children = [
      # Start the Telemetry supervisor
      FluffelWeb.Telemetry,
      # Start the PubSub system
      {Phoenix.PubSub, name: Fluffel.PubSub},
      # Start CubDB
      {CubDB, data_dir: "/metrics_data", auto_compact: true, name: :db},
      # Start Metrics Supervisor
      Fluffel.Metrics,
      # Start the Endpoint (http/https)
      FluffelWeb.Endpoint
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: Fluffel.Supervisor]
    Supervisor.start_link(children, opts)
  end

  # Tell Phoenix to update the endpoint configuration
  # whenever the application is updated.
  @impl Application
  def config_change(changed, _new, removed) do
    FluffelWeb.Endpoint.config_change(changed, removed)
    :ok
  end
end

and the mounts part from my fly.toml:

[mounts]
  source = "metrics_data_volume"
  destination = "/metrics_data"

[CharlesIrvineKC]

I was able to resolve this.