teamhephy / controller

Hephy Workflow Controller (API)
https://teamhephy.com
MIT License
14 stars 26 forks source link

Dockerfile + Procfile disaster recovery issue restoring scale #156

Open kingdonb opened 2 years ago

kingdonb commented 2 years ago

I have a ruby app that is built using the Dockerfile method, and includes a Procfile, (which I assume is important)

Storage and database are both externalized so I can nuke the cluster and bring it back without any persistent volumes or state loss. The app has one entry in Procfile, named cron which does not actually run cron, but looks as shown below and has roughly the effect of a cron job that runs a script at least/about once every 4 hours, similar to Heroku's "cron" behavior.

The scale is not restored correctly after the cluster nuke-and-restore, while some other apps are restored no problem.

(⎈ |oidc@moo:deis) kingdonb@Kingdons-MBP-2:~/hephy/ob-mirror (master u=)$ vi Procfile
> (the file:
cron: ./README; sleep $((15 * 60 * 4 * 4))  # every 4 hours
)
(⎈ |oidc@moo:deis) kingdonb@Kingdons-MBP-2:~/hephy/ob-mirror (master u=)$ deis ps:scale cron=1
Scaling processes... but first, coffee!
done in 0s
=== ob-mirror Processes

> (scale was already "1" so nothing happens)

(⎈ |oidc@moo:deis) kingdonb@Kingdons-MBP-2:~/hephy/ob-mirror (master u=)$ kubens ob-mirror
✔ Active namespace is "ob-mirror"
(⎈ |oidc@moo:ob-mirror) kingdonb@Kingdons-MBP-2:~/hephy/ob-mirror (master u=)$ k get po -w
^C

> (note that there are no `cron` pods)

(⎈ |oidc@moo:ob-mirror) kingdonb@Kingdons-MBP-2:~/hephy/ob-mirror (master u=)$ deis ps:scale cron=0
Scaling processes... but first, coffee!
done in 1s
=== ob-mirror Processes
(⎈ |oidc@moo:ob-mirror) kingdonb@Kingdons-MBP-2:~/hephy/ob-mirror (master u=)$ deis ps:scale cron=1
Scaling processes... but first, coffee!
done in 2s
=== ob-mirror Processes
--- cron:
ob-mirror-cron-65698fbd4b-xngkr up (v4)

> (that worked!)

(⎈ |oidc@moo:ob-mirror) kingdonb@Kingdons-MBP-2:~/hephy/ob-mirror (master u=)$ k get po -w
NAME                              READY   STATUS    RESTARTS   AGE
ob-mirror-cron-65698fbd4b-xngkr   1/1     Running   0          5s
^C

I think we might already have an issue logged for this, but I've seen it a couple of times now and wanted to make sure it got into a report.