Make sure uniqueness quantum job on the cluster

[denielchiang]

Using quantum and locked at 3.5.0. The cron job has been excuted twice at the time, logs below

2023-07-05T22:00:00Z app[148ed199b34489] sin [info]22:00:00.561 [info] Today 2023-07-06 Topic already be sent
2023-07-05T22:00:00Z app[148ed199b34489] sin [info]22:00:00.561 [info] Sending result: Topic already be sent
2023-07-05T22:00:00Z app[178113d4f5e189] sin [info]22:00:00.733 [info] Today 2023-07-06 Topic already be sent
2023-07-05T22:00:00Z app[178113d4f5e189] sin [info]22:00:00.733 [info] Sending result: Topic already be sent

Cron job configuration

  config :line_reminder, LineReminder.Scheduler,
    timezone: "Asia/Taipei",
    overlap: false,
    jobs: [
      {"0 6 * * *", {LineReminder.Messanger, :send, []}}
    ]

Will find time to fix this duplicate tasking someday.

[denielchiang]

It didn't work, still send twice at the time.

2023-07-06T22:00:01Z app[148ed199b34489] sin [info]22:00:01.272 [info] Today 2023-07-07 Topic already be sent
2023-07-06T22:00:01Z app[148ed199b34489] sin [info]22:00:01.272 [info] Sending result: Topic already be sent
2023-07-06T22:00:01Z app[178113d4f5e189] sin [info]22:00:01.285 [info] Today 2023-07-07 Topic already be sent
2023-07-06T22:00:01Z app[178113d4f5e189] sin [info]22:00:01.285 [info] Sending result: Topic already be sent

I found this post, it seems to make sense to remove quantum and use Oban instead when it is cluster.

Reopen this issue and use Oban to do the scheduling tasks.

[denielchiang]

Did some research, it seems to have serval ways to deal with the "uniqueness quantum job on the cluster".

• Use Oban like previous comment
• Use periodic for the job scheduler, combined with highlander to ensure uniqueness in the cluster. -- quote from Connorlay wrote here, Connorlay even wrote a proof-of-concept.
• Use custom Supervisor module, quoted from Matsa59 wrote on Quantum open issue.

Oban

Oban is a great dependency especially with Oban Web+Pro to have dashboard to monitor those tasks. But I only have one single cron job need to run, so it properly a little bit over-killed in this project.

Periodic/Highlander

Yup, I admire this easy and simple idea. Just lazy to remove the Quantum and re-write periodic tasks. Maybe will try this one after my approach failed one more time.

Supervisor/Libcluster

This one is pretty elegant to me tho, maybe later will try this one too.

Roll the dice

After reading the definition of Highlander

Highlander allows you to run a single globally unique process in a cluster.

What if I just need Highlander to make sure the only one single process, which is the Quantum job, be the unique process in the cluster? like wingyplus says on this comment?

It seems worth to try from wingyplus solution. Since it's less effort and easy to do.

17

[denielchiang]

2023-07-08T22:00:00Z app[148ed199b34489] sin [info]22:00:00.775 [info] Today has no more topic to send
2023-07-08T22:00:00Z app[178113d4f5e189] sin [info]22:00:00.934 [info] Today has no more topic to send

Obviously I understood it wrong, so I re-read the supervisor and previous posts again. Until I read the post - Clustering Your Application on Fly.io. I did some check as the post said.

fly ssh console --pty -C "/app/bin/hello_elixir remote"

expect to have something like this, but didn't.

iex(fly-elixir@fdaa:0:1da8:a7b:ac2:f901:4bf7:2)1> Node.list
[:"fly-elixir@fdaa:0:1da8:a7b:ac4:eb41:19d3:2"]

That's explain lots things to me.

• Why there're 2 machines on Fly.io after I running fly deploy --remote-only?
• How to remain the amounts of these machines by using cluster?

After some digging, I found those answers.

Why there're 2 machines on Fly.io after I running fly deploy --remote-only?

I found here's a doc on fly.io already explain it.

The most basic way to improve resiliency is to create more than one Machine per process group, and we do this by default for process groups with services. A process group with services has mappings to the external internet or to a private (Flycast) network, so its connections and operation can be managed by Fly Proxy.

If a process group has services defined, two Machines are automatically created and started when:

you deploy an app for the first time using fly launch or fly deploy, you redeploy an app using fly deploy after scaling down to zero, or you add a new process group with services in the fly.toml file and then run fly deploy Looks like somehow I defined services without knowing it during the first deployment. And it also can be turned off by this

fly deploy --ha=false

How to remain the amounts of these machines by using cluster?

Okay, since it's a volunteer project for studying. Why not try to make it cluster in the correct way? I decided to follow the instruction that Clustering Your Application said, add libcluster then remain the highlander setting to see what's happened next.

[denielchiang]

Actually got stuck during continue deployment. Since I already wrote my own migration function, I decided to turn off the release_command like below.

Edit fly.toml, comment those lines

#[deploy]
  # release_command = "/app/bin/migrate"

That's it! Deploy success!

[denielchiang]

It's work.