Not working at all with latest docker (swarm)

[srigi]

I tried the basic example with clean installation of Docker in Virtualbox VM:

$ docker info

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 18.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: a9qe80a6lq8iigdfsyr54mpqi
 Is Manager: true
 ClusterID: jfkg6jpj5akyw2cxh11ph0hvh
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 10.0.2.4
 Manager Addresses:
  10.0.2.4:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-119-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 488.2MiB
Name: swarm-master-dooku
ID: DH57:OVOC:FFHK:L7LA:5VG3:RET2:2RWC:DTQO:UHID:NPLR:NVM3:AMBK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

First I deployed a CRON service as a stack:

$ docker stack deploy -c docker-compose.yml cron
Creating service cron_cron

$ cat docker-compose.yml
version: '3'

services:
  cron:
    image: vfarcic/docker-flow-cron
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - 8080:8080
    deploy:
      placement:
        constraints: [node.role == manager]

Then I send request do API via curl:

$ curl -XPUT -d '{
  "image": "busybox",
  "command": "date",
  "schedule": "@every 15s"
}' http://127.0.0.1:8080/v1/docker-flow-cron/job/test

{"Status":"OK","Message":"Job test has been scheduled","Job":{"name":"test","servicename":"","image":"busybox","command":"date","schedule":"@every 15s","args":null,"Created":false},"Executions":null}

Then I inspected log of cron service (after more than 15s):

docker service logs -f cron_cron
cron_cron.1.62a3chszlr45@swarm-master-dooku    | Rescheduling jobs
cron_cron.1.62a3chszlr45@swarm-master-dooku    | Starting Web server running on 0.0.0.0:8080
cron_cron.1.62a3chszlr45@swarm-master-dooku    | Scheduling test
cron_cron.1.62a3chszlr45@swarm-master-dooku    | Executing command: docker service create -l "com.df.cron=true" -l "com.df.cron.name=test" -l "com.df.cron.schedule=@every 15s" --name test --replicas 0 -l 'com.df.cron.command=docker service create --restart-condition none busybox date' --restart-condition none busybox date
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | Could not execute command:  docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | Could not execute command:  docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | Could not execute command:  docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | Could not execute command:  docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | Could not execute command:  docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | Could not execute command:  docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1
cron_cron.1.62a3chszlr45@swarm-master-dooku    | docker service scale test=1

Finally I inspected logs of scheduled task to see if it is executing every 15s:

$ docker service logs -f test
test.1.7j305djfe8zv@swarm-agent-grievous    | Mon Apr 16 12:25:31 UTC 2018

As you can see, task was executed only once (even after waiting couple of minutes), so this cron solution for swarm doesn't work as expected. There should be 4 log lines for every minute.

Few more logs:

$ curl http://127.0.0.1:8080/v1/docker-flow-cron/job/test

{"Status":"OK","Message":"","Job":{"name":"test","servicename":"test","image":"busybox","command":"docker service create --restart-condition none busybox date","schedule":"@every 15s","args":null,"Created":false},"Executions":[{"ServiceId":"pg90iu3bjvlcnd2igoso5kwdj","CreatedAt":"2018-04-16T12:25:28.686052105Z","Status":{"Timestamp":"2018-04-16T12:25:31.604537515Z","State":"complete","Message":"finished","ContainerStatus":{"ContainerID":"127d6df82c2fa861b050a4cc29e95f8c36c94362af495f849bfa763507c0d2b8"},"PortStatus":{}}}]}

$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                             PORTS
50jwnckom735        cron_cron           replicated          1/1                 vfarcic/docker-flow-cron:latest   *:8080->8080/tcp
pg90iu3bjvlc        test                replicated          0/1                 busybox

Please provide any feedback how this project is currently standing with respect to latest Docker. So far it looks like it is not working as expected.

Note, I'm using swarm with two nodes (master dooku, worker grievous). Even if task is "constrained" to master node, it executes only once.

[vfarcic]

There were no commits to this project for almost a year. It is, at the moment, abandoned. It would be great if someone (you) could pick it up.

[beenhead]

So this issue is related to a fix applied by the Docker Swarm team, but a workaround has been provided in this note

Using this workaround in a change to the cron.go file, and an update to the Docker version included in this project's Dockerfile I was able to get this working.

I understand this project is abandoned, and I do not believe I am qualified to pick this up moving forward (this is the first time I have modified Golang code) but I wanted to put this note here to suggest it's a fairly easy fix (hack)

Thanks for all your work on this.

[vfarcic]

As you noticed, the project is up for adoption. If you'd like to take it over @beenhead, I can help you with Go.