State files for non-dispatched components are created during daemon start

[ned21]

After upgrading from 1.2.1 to 16.6 we noticed that we had stale files left in /var/run/quattor-components. We noted that these were for non-dispatched components and had a timestamp that matched the last restart of ncm-cdispd (which we do nightly). This is a bug because we have processes that check for the existence of these files to indicate that a components that did not successfully complete or finish.

Steps to reproduce:

1. Make sure you have at least one component in your profile with dispatch=false
2. sudo rm /var/run/quattor-components/*
3. sudo service ncm-cdispd restart
4. Wait for ncm-ncd to finish successfully with no errors.
5. ls /var/run/quattor-components

Expected result: empty directory Actual result: state files remain for all non-dispatched components.

[stdweird]

The current code has the following usage of state files:

• empty state file of all components from ncm-cdisp (incl inactive ones, missing dispatch or previously existing ones in ICLIST). so this empty statefile indicates that ncm-cdispd is aware of the component, but not necessarily that ncm-ncd is going to pick it up or do anything.
• ncm-ncd sets an empty state file for each somponent before it is going to process it (this is after filtering inactive ones and incl any dependencies)
• components that have error or are broken (meaning failed predep) get non-empty statefile
• when errors are downgraded to warnings, the state files are not cleared (so the initial empty statefile is kept) (is probably a bug, since the statefile gets cleared for regular warnings https://github.com/quattor/ncm-ncd/issues/77 )

as a result:

• non-empty statefile, something went wrong
• missing statefile, all is fine (or did not run, which is also fine)
• empty statefile could come from 3 scenarios
  • something went horribly wrong
  • inactive/non-dispatched components from ncm-cdispd (at least the inactive ones will never get cleaned up by ncm-ncd; the non-dispatched ones could be a dependecy of another component, and thus handled by ncm-ncd)
  • downgraded error -> this is likely a bug

Also good to know: ncm-cdispd runs ncm-ncd with its own statedir configuration; i.e. if the statedir of ncm-ncd.conf and ncm-cdispd.conf does not match, you can get some surprises (manual runs will not afftect the state the ncm-cdispd runs, which ofcourse could also be intentional). by default, they are the same (altough not unittested afaik)

@ned21 wrt your issue: the current code to set the state should be called in the else block at https://github.com/quattor/ncm-cdispd/blob/master/src/main/perl/Utils.pm#L183 rather than at the beginning, so only active and to be dispatched components have the state file touched by ncm-cdispd. (or the remove method in ncm-cdispd should also remove statefiles?) i'm a bit uncertain why ncm-cdispd touches the state files at all, probably to handle cases where ncm-ncd crahses? ncm-cdispd should also use (or try to) the NCD::ComponentProxy->set_state method, instead of its own non-logged implementation. (and the ncm-cdispd script itself should use a FileReader to read the state files, or at least cancel the current Editor before close)

[stdweird]

@ned21 i think all issues are covered and referenced here. i'll leave it to you for further testing and/or closing