Optimize cloud-init user data script

[0x2b3bfa0]

On public cloud machines, the startup script needs to be compatible with any modern and sensible GNU+Linux image, and that requires some optimizations and workarounds. For example, we should try to avoid depending on package managers and software other than systemd.

Installing rclone

Whilst rclone binary distributions are self–contained Go executables, they come bundled in zip files, and extracting them in a portable way is hard.

• [x] Short term: #286
• [ ] Mid term: integrate this functionality in the agent by using the rclone module.

Watching files

In order to avoid installing a file watcher,[^1] we've replaced “smart” data+log synchronization by polling, considerably increasing transfer costs.

• [x] Short term: #322 <- #321
  • ~at very least, we should use md5sum and sleep to push logs only when (1) there are new log lines and (2) a significant amount of time has passed since the last synchronization~
• [ ] Mid term: integrate this functionality in the agent by using the fsnotify module.

Issues

• https://github.com/iterative/terraform-provider-iterative/issues/321

Other fixes

• [x] #318
• [x] #313

[^1]: Installing software usually requires resorting to distribution–specific package managers.

[dacbd]

I haven't yet experimented enough so this might not be applicable, but are you open to having an alternative/selectable options for the log capture?

The ideas I have rattling around might be past the scope of what task is trying to provide.

[0x2b3bfa0]

Supporting arbitrary log backends with the same interface doesn't look like a trivial task. Still, you can always install and run software like the CloudWatch Agent as part of the task script.