Data Persistance

[bacco007]

Is there a way to have the data persist after container updates / restarts?

[mikenye]

Hi @bacco007,

Does ModeSMixer have a way to do this when not operating in a container?

I've taken a look at the documentation, and I'm not sure whether it does: http://xdeco.org/?page_id=48

I'm certainly no expert with ModeSMixer - I was asked to create a container by someone who used my other ADS-B containers...

If you're able to tell me how to achieve this with ModeSMixer, I'll make the neccessary changes to the container and documentation.

Thanks.

-Mike

[bacco007]

Hi @mikenye It seems to be a common issue - https://radarspotting.com/forum/index.php/topic,9506.0.html

Happy to close on this basis

[mikenye]

Don't close it just yet...

There might be a way to do this using a process snapshotting tool, ie:

• https://criu.org/
• https://github.com/dmtcp/dmtcp

There are likely others.

I'll have a play with these and see if they work with ModeSMixer reliably. If so, I'll attempt to implement in the container, although I foresee issues with permissions etc...

[mikenye]

Update, had some time to try out criu this morning. It looks promising but won't run in a container. It can be installed as an experimental docker plugin, but not sure this helps us...

I'll try out DMTCP next.

[mikenye]

FYI - output from criu check running in a docker container:

root@8874c553780d:/src/criu# criu check
+ criu check
Error (criu/util.c:624): exited, status=1
Error (criu/util.c:624): exited, status=1
Error (criu/tun.c:85): tun: Unable to create tun: No such file or directory
Warn  (criu/sk-unix.c:229): unix: Unable to open a socket file: Operation not permitted
Error (criu/net.c:3452): Unable create a network namespace: Operation not permitted
Warn  (criu/net.c:3508): NSID isn't reported for network links
Error (criu/kerndat.c:923): Lazy pages are not available
Error (criu/kerndat.c:1143): kerndat_uffd failed when initializing kerndat.
Error (criu/crtools.c:213): Could not initialize kernel features detection.

[mikenye]

Struggling with DMTCP too...

root@1a8231d46870:/# dmtcp_launch --no-coordinator --no-gzip --ckptdir /data --modify-env /usr/local/bin/modesmixer2  --inConnect=reads:30005     --metric     --web=8081
[40000] NOTE at eventwrappers.cpp:216 in epoll_create1; REASON='epoll fd created1'
     ret = 4
     flags = 524288
2020-08-04 11:06:40.684  INFO     ModeSMixer2 v.20190216
[40000] ERROR at mutex.cpp:81 in DmtcpMutexUnlock; REASON='JASSERT(mutex->owner == dmtcp_gettid()) failed'
modesmixer2 (40000): Terminating...

[mikenye]

I've made some progress with DMTCP, however it still isn't working but I think I'm pretty close...

The image in branch data_persistance_dev_dmtcp will take a checkpoint of the ModeSMixer2 process every 60 seconds, and stores the checkpoint to /checkpoints within the container.

If a checkpoint exists on container start, an attempt is made to restore the checkpoint, however this is currently failing.

Container start, checkpoint & stop with DMTCP Checkpointing

$ docker run --hostname mm2 --name mm2 --rm -it -v /tmp/c:/checkpoints -p 8999:8081  mm2test --inConnect readsb:30005 --web 8081
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] 01-cmd: applying...
s6-envuidgid: fatal: unknown user: account
[fix-attrs.d] 01-cmd: exited 0.
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[dmtcp_coordinator] dmtcp_coordinator starting...
[dmtcp_coordinator]     Host: mm2 (172.17.0.6)
[dmtcp_coordinator]     Port: 7779
[dmtcp_coordinator]     Checkpoint Interval: disabled (checkpoint manually instead)
[dmtcp_coordinator]     Exit on last client: 0

ModeSMixer command line arguments: --inConnect readsb:30005 --web 8081

2020-08-04 16:57:29.654  INFO     ModeSMixer2 v.20190216
2020-08-04 16:57:29.656  INFO     inConnect(readsb:30005) connecting readsb:30005
2020-08-04 16:57:29.656  INFO     inConnect(readsb:30005) connected
2020-08-04 16:57:29.819  INFO     inConnect(readsb:30005) BEAST data stream detected
2020-08-04 16:57:42.543  INFO     Coverage area was determined
2020-08-04 16:57:42.543  INFO     Magnetic declination calculated
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

Contents of Checkpoint Directory

$ ls -la
total 7484
drwxr-xr-x  2 root root    4096 Aug  5 09:46 .
drwxrwxrwt 14 root root    4096 Aug  5 09:46 ..
-rw-------  1 root root 7645475 Aug  5 09:46 ckpt_modesmixer2_b6c626-40000-1196c4074e8e42.dmtcp
-rwxr--r--  1 root root    6661 Aug  5 09:46 dmtcp_restart_script_b6c626-40000-1196c4059ec732.sh
lrwxrwxrwx  1 root root      51 Aug  5 00:58 dmtcp_restart_script.sh -> dmtcp_restart_script_b6c626-40000-1196c4059ec732.sh

Attempting Restore from Checkpoint

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] 01-cmd: applying...
s6-envuidgid: fatal: unknown user: account
[fix-attrs.d] 01-cmd: exited 0.
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[dmtcp_coordinator] dmtcp_coordinator starting...
[dmtcp_coordinator]     Host: mm2 (172.17.0.6)
[dmtcp_coordinator]     Port: 7779
[dmtcp_coordinator]     Checkpoint Interval: disabled (checkpoint manually instead)
[dmtcp_coordinator]     Exit on last client: 0

ModeSMixer command line arguments: --inConnect readsb:30005 --web 8081

Resuming from checkpoint...

[252] mtcp_restart.c:406 restore_brk:
  error: new/current break (0x558846c1f000) != saved break (0x5628c22cb000)

...so, I need to figure out a way to prevent this error: new/current break != saved break on restore. I will open an issue with the DMTCP project to see if there's a way to resolve this.

[mikenye]

Raised: https://github.com/dmtcp/dmtcp/issues/854

[mikenye]

Raised: https://github.com/dmtcp/dmtcp/issues/855

[mikenye]

Update, tried BLCR, doesn't work without host kernel modifications.