Feature Request: keep track of modules that have already been run and dont run if unchanged

[emmetog]

This may be a bit against the current philosophy of fa which currently is cronned to run every X hours and always runs all of it's modules, relying on the fact that modules are idempotent. The downside is that changes are not immediate, making changes which break things and their resolutions difficult (for example "Oh no, a change we made in the morning was applied while we're out to lunch and broke everything! And after fixing we need to manually go into the server and run fa apply again as otherwise it won't run until tomorrow lunchtime")

In our case we'd prefer changes to be applied as immediately as possible, so we'd use a really small cron interval (5 mins?). However this may result in modules being run very very often when it may not be needed. Again, if things are idempotent it's not a huge problem, the only downside is running things unnecessarily, maybe resulting in wasting cpu cycles and bandwidth if updates are checked for, etc.

Other "pull provisioners" like chef and puppet poll a central server for modules to apply and only actually run them when a change is detected. Loosely applying this to fa would mean that the --crontab-entry would essentially become the "check for updates poll interval" and fa would need a mechanism to store the state and detect if a module has or has not been already applied to the host. Just throwing ideas out this could possibly be done by dumping the code of the module plus all the variables to a file and calculating the md5 hash of that. If the module code itself changes or if any of the variables which apply to the host change at all, the hash would change and the module would be re-applied (and might even still be idempotent anyway).

I'm aware that this is potentially a big architectural change, just wanted to throw it out as a possibly easier way of using fa.

[P5vc]

Thank you for this suggestion.

We cannot fully implement this change, as Fetch Apply must maintain two important features:

• Module redundancy
  • Some of our users rely on our modules to complete daily maintenance tasks, such as error reporting, log collection, and analytics resets. Others may use the redundancy, for example, as a safety mechanism to ensure that updates don't accidentally alter/replace configuration files in an undesired way.
• Agentless... ish
  • It's important to many of our users that Fetch Apply remain mostly agentless. We cannot setup a Fetch Apply service or undergo a similar approach.

That being said, I do have some good news: Our internal development team has been testing a new feature this past week that could address some of the concerns you bring. It is not slated for release on a specific day yet, however I am pretty confident that it could be pushed to our public repository sometime this month.

This new change brings a feature we call "patches," which, among other things, allows for immediate, customized code execution, to correct problems with, say, the latest module applications. We will also be adding an option to the configuration file to only run modules after a change has been pushed to the remote operations repository.

I cannot go into all of the details at the moment, as this feature is still under development, however I do expect it will address most, if not all of the concerns you raised, even if not quite in the way you articulated. I will keep you updated, and let you know when it's released, as well as provide you with documentation on it once it is released.

[P5vc]

A quick update:

It looks like our main, "patches" feature still won't be ready for some time. However, we did push some of the other code that goes along with that update, including the ability to automate the upgrade process, for installing new versions of Fetch Apply on your system (note: in your case, you will have to manually uninstall and then reinstall Fetch Apply, however the most recent version and all future versions will support automated upgrades, when compatible). We also added a new Fetch Apply configuration: fa_var_APPLY_ON_CHANGE. If you go into your new configuration file (/etc/fetchapply) and set this equal to true, modules will only be applied whenever there has been a change made to the operations repository. This isn't exactly what you described, and will still require modules to be designed with redundancy in mind, however it should still accomplish more or less what you were looking for. In this case, you could set the crontab interval to run once every five minutes, and then Fetch Apply would only apply modules when changes are made to the origin/master operations repository.

Feel free to try out these new changes and let us know what you think. Thanks!

[P5vc]

Our team decided to scrap the "patches" update for a few reasons (too much inefficiency, redundancy, over-complication, etc.), however as it seems that we've been able to successfully address the main concerns you were having, I'm going to go ahead and close this issue. If you would like to add any more details, requests, changes, comments, etc. please feel free to do so, and don't hesitate to re-open this issue if you feel it necessary. Thanks!