File tracing requirements

[ndmitchell]

As requested by @jacereda and @droundy, my rough requirements for something like FSATrace are:

• I need it to run a system command and tell me which files were read and written.
• Must work on at least Windows and Linux, Mac/BSD etc all most welcome.
• It's a nice-to-have optional feature, so needs to either build absolutely everywhere with a simple cabal install or be runtime optional (e.g. dynamically loaded library, separate executable).

And finally an added bonus:

• Would be very nice if I could get notifications before it tries to build a file, so I have a chance to do dependency analysis and build the file. e.g. if gcc -c Main.c stopped and notified me before Main.h was accessed that would be really useful.

Currently fsatrace meets all my needs for the first 3.

[jacereda]

At some point you also requested stat() reporting. Not sure how accurate the current fsatrace implementation is across platforms though.

[ndmitchell]

Reporting on things like doesFileExist/getDirectoryContents etc is useful - the more external state queries you capture the better.

[droundy]

Yes, stat is absolutely needed to track the dependencies of python scripts, for instance, due to their use of cached .pyc files.

On Sat, May 21, 2016, 2:23 AM Neil Mitchell notifications@github.com wrote:

Reporting on things like doesFileExist/getDirectoryContents etc is useful

• the more external state queries you capture the better.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/ndmitchell/shake/issues/455#issuecomment-220768005

[jacereda]

I think the last point could be handled with something like this:

https://github.com/jacereda/traced-fs/commit/f640b659845cc321eb46aa9563b8c57f0025ab5c

The way I think it could work is by having that generator notify via some mechanism a dedicated thread in shake (if we're using -j1 the build will be stopped waiting for completion of the interrupted command).

[ndmitchell]

Yep, anything where I can get a handle on a change. However, in your example do you call generate only when the file is missing? If so, that's still not enough, because a file might be present, merely dirty, and so I'd still need to rebuild it before relying on it.

[jacereda]

I seem to recall we already discussed this, but can't find the reference and I don't remember what the conclusion was. Let's say we need to generate a config.h. When foo.c includes config.h, the generate mechanism will be used to create it. Afterwards, if config.h.in is modified, config.h will be properly regenerated because the dependence on config.h.in will be already recorded in the database, so the result would correct. Am I missing something?

[ndmitchell]

I do remember discussing this vaguely, and also can't remember the conclusion...

I guess the first time around you wouldn't get to generate config, assuming the config already existed - but then it would be kind of weird having a source and also generating the file. I think what you've got is at least "mostly" sufficient, quite possibly "completely" sufficient, and using needed I can make sure it doesn't cause any problems.