how to include fake buildfiles (was: don't crash on duplicate targets)

[silky]

i've asked a related question on StackOverflow - http://stackoverflow.com/questions/20319061/any-way-to-have-a-many-buildfile-structure-in-fake

is is possible, without hacking, to make FAKE not crash on duplicate targets? I realised I could define my own Target inline, and just very offensively check to see if it is in the TargetDict object first.

what I really want to do is have an include structure between .fsx files that works. at the moment, because the load directive runs the script, my targets are all added >= 2 times, and hence fail.

[forki]

I assume you want to do something similar to https://github.com/fsharp/FAKE/issues/244 but maybe using a slightly different approach. Please read carefully through this issue and the referenced proof of concept. Would the |=> operator help to solve your problem?

I don't really under how the loading process of fsx files works and why the files is executed twice. But I don't think it's a good idea to allow overwrite targets. This would silence possible conflicts between different scripts.

[silky]

@forki No I don't believe that |=> helps at all.

The real problem is having this multi-file structure. I just want standard dependencies, but I want to put everything in it's own file, and I want to be able to "import" targets from one file into the other.

The trivial fix is to write something in my "common.fsx" like:

let OldTarget = Target
Target name fun : unit = 
    if not (TargetDict.ContainsKey (toLower name)) then
        OldTarget name fun

Of course, it's a huge hack because I'm using internal details of "TargetHelpers.fsx", and furthermore I have to do it for "FinalTarget", "Description", etc.

I'm not sure of the solution myself. Maybe it's wrong to import other ".fsx" scripts? But it's a pretty standard idiom in NAnt, which is what I was hoping to replace.

I agree that the error on overwriting of targets is actually useful.

I also am mildly confused that the #load directive kind of runs the static initialiser for the module k times, for each time it is #load'd (see my SO link for a runnable example of this.) But I also don't know the solution. There appears to be no way to do this multi-level importing. If you import once. But the problem is this multi-level thing I'm doing. It seems to be a standard idiom; define a common set of tasks, then have each of your projects depend on that, then build "larger" projects by importing all the little "subprojects" and running them appropriately.

I could be convinced that my way of building these "super" projects is wrong. The only alternative seems to go around calling Fake on those .fsx's directly; but that's unpleasant. The reason is you lose the level of detail to look at task dependency trees, start specific targets, etc, etc.

[forki]

@dsyme probably knows why it executes everything multiple times.

[forki]

@shishkin could you please document your approach somewhere?

[shishkin]

What we do is we have our own Contoso.FAKE NuGet package which has a dependency on a strict version of FAKE. This helps us to have a hard-coded (but automated) reference to the FakeLib.dll from a build script. With this approach our build scripts have intellisense in VS without command line fiddling with simple NuGet stock MSBuild package restore.

This custom package also adds three things to the solution:

• a build script template (if it doesn't already have one);
• a powershell boot strapping script;
• and a custom F# script which is loaded into the build script and provides some common stuff.

All this is done with powershell unfortunately, but works in our case just fine. We have this package in an internal feed and can bump its version in order to upgrade to newer FAKE or to roll out new common plumbing.

Regarding the predefined targets, I would rather want FAKE to have targets as function bindings which I can define, redefine and wire together in a directed graph via dependencies. In any case one can just write something like:

Target "MyTest" PredefinedBuildAndRunNUnitTarget
Target "MyCompile" SomePredefinedCompileInRelease

"MyTest" ==> "MyCompile"

I actually tend to put test before compile lately. Tests do compile anyway and if they fail, why bother compiling in release?

[silky]

@shishkin thanks.

Okay, so you generate .fsx templates; you literally write say the "Env" step from my example into each of the "Foo" and "Bar" scripts. I'm not sure how I feel about that. Not good. But perhaps I don't understand it.

Are you able to share a small example of it?

[silky]

Infact I think what I want is more impossible than I anticipated. It's not possible to group previously-independent build scripts because it is necessary to have the "Run" or "RunTargetOrDefault" command at the end of the script. If you have this, it runs the target.

If you #load it, then you necessarily run that line.

This is a bit unfortunate.

What is the standard technique here to combine independent projects? I guess there isn't one. The requirement would be to call FAKE on each of the fsx scripts independently, but as mentioned that has other downsides ...

[silky]

Okay, here is a far less offensive solution I came up with:

• https://gist.github.com/silky/bf304785a46db32b5cc8

It's lets all the projects exist independently, and then allows their recombination into larger projects.

Thoughts?

[shishkin]

@silky , I don't write same steps into each build script of course. I gather all the common stuff in a single fake.fsx script, which I make sure is referenced from each build script. This is done by replication: fake.fsx is copied upon package installation and gets overwritten upon each update. I also have one build script per project.

Your gist with common, foo, bar and marvin targets reminds me of the implicit MSBuild pipeline. All the steps are already predefined, you just install after- and before-hooks. It gets messy, because the pipeline is not expressed in one place, but is scattered across multiple target files.

I believe it's much simpler to have potentially multiple common files with functions, but define the pipeline (targets and dependencies) in a single file for each project.

[silky]

@shishkin I guess defining the "pipeline" in a single file in each project in the Gist above is strictly wrong, right, because it means I'd need to define the "Foo ==> Bar" dependence twice; in Bar and in Marvin.

I don't see any other reasonable way around it at this point.

[shishkin]

@silky Wrong, right :)

Then composition of targets is what you need. If Bar makes no sense without Foo, make a FooBar target and reference it in you script.

What I like about FAKE is that I know exactly which code is going to run in which sequence (except for when I made the sequence non-deterministic myself via dependencies). I also like the ability to Ctrl-Click any name binding in a build script and see its definition. I only tolerate string-based targets as long as they are in the same file. Splitting them across multiple files would result in a maintenance nightmare.

[silky]

@shishkin Bar makes no sense without Foo, indeed.

But Foo can be built independently (of Marvin). But Marvin requires all of Bar (and Foo). I don't see how your proposal helps me.

I don't see why splitting things across files is a nightmare. The IDE support is a completely irrelevent to me (I use vim.) I mean, what I'm doing is not that ridiculous. It's the standard approach in programming - split up all the code into little units, combine it when necessary, etc ...

[shishkin]

For an example of a nightmare look no further than at MSBuild. It also loads multiple files recursively, each of which define and redefine targets and dependencies. Polymorphism is a powerful thing, but often simple functional composition is enough.

I like vim as an editor as well, but I also like the editor to know that the symbol under the cursor isn't arbitrary text, but rather has meaning, defined in a particular place.

[forki]

I think you should distinguish between tasks and targets.

As I see it tasks are the reusable unit and targets are the concrete configurations. I don't reuse targets anymore.

I think if you try to put your reusable parts into tasks it might make more sense.

We used to write target generator functions in referenced files. This worked, but it seems a bit overkill.

[shishkin]

:+1: on this. Targets should be something that makes sense to run on its own. If it doesn't, it probably is a task – a function.

[silky]

@shishkin It'd be great if you could include some code samples with your comments. I honestly am not sure what you are suggesting I actually change to accomplish the exact same functionality I demonstrated in the earlier Gist.

@forki This is what I thought of doing also. I definitely will not be writing any sort of code-generator for this; it seems completely wrong to me.

Unfortunately I'm not sure how to do it without repetition; Note that I want to be able to call the target "Foo" when building the project "Marvin" or building the project "Bar". I do not want to repeat parts of the build dependency tree inside "Bar" that I already specified in "Foo", and similarly for "Marvin".

I feel the need to reiterate that NAnt already allows this. This is not something odd, I don't think ...

[silky]

@forki It seems like you would suggest Foo and Bar now have "foo_funcs.fsx" which contains their little trace messages as functions, then in "Bar" I'd call "foo_funcs" and similarly.

This seems no better than what I have, and also weird and annoying because now the targets are just meaningless wrappers to functions.

Am I missing something? What are you actually suggesting?

[forki]

I don't think I want to implement a feature just because NAnt has it. That's not the way I want to go.

I don't propose a solution to your file loading problems here. I think you have to come up with a different way to structure your projects. Instead of absorbing all the subprojects you may want to keep them modular. This may sound like an excuse for a missing feature and in some sense it is. But I don't want to break the behavior for conflicting targets. And I don't think nesting build scripts is really desirable.

That said: we could come up with a FAKE task which make it easier to call other FAKE builds in a controlled environment.

[silky]

@forki - I understand re not implementing what NAnt has; that's sensible, and not what I wanted.

My question on SO and here is - I don't know how else to structure the project! I claim to need the functionality I get from the earlier Gist - independent projects (Foo can build on its own, Bar can build on its own) but with some hierarchy between them.

In an ideal world I would list say "Foo" as a Nuget package that "Bar" must "obtain" (say by grabbing its output and placing it in some folder). But this is a bit of overkill, because Foo is right there, and the process for obtaining it as a package would be really time consuming.

For a practical example; I have a suite of executables deployed under one umbrella app. All the exes are built independently, within the repo, but I want to combine them into a single distributable (and also perform some post-build tasks, like versioning, etc, etc.) How to achieve this? That's my question.

[silky]

Re this comment:

That said: we could come up with a FAKE task which make it easier to call other FAKE builds in a controlled environment.

This would only be useful if I could include the FAKE targets. Because I want to be able to run any of the "sub-targets" from the main .fsx script.

Say I had

// bar.fsx
Target "Bar" (fun _ -> runFake "foo.fsx" )

This is insufficient. I can't run arbitrary Targets defined in foo.fsx, by running

fake.exe bar.fsx target=Foo

[shishkin]

@silky Now I see that you actually have two independent problems to solve:

1. Orchestration of multiple external FAKE builds from a FAKE build. I think addition of a Fake task with parameters being targets would do the trick.
2. DRY and code reuse in build scripts. This is already possible on the function level, just put your common logic in functions/tasks and call them from targets. I understand that you consider repetition in targets and their dependencies a smell, but I'd rather have my build pipeline be right there explicit in my face, rather than pulling my hair out debugging an implicit pipeline with steps overridden somewhere. I think turning targets into values instead of mutating global TargetDict would greatly help with code reuse via function composition.

[silky]

pipeline with steps overridden somewhere

Nothing is overridden in my setup. Nor do I want it to be.

I think turning targets into values instead of mutating global TargetDict would greatly help with code reuse via function composition.

I probably agree with you.

I think addition of a Fake task with parameters being targets would do the trick.

I don't see how this would lead to me being able to run the sub-targets, as described in my post just above yours.

[forki]

It seems you are not considering to use a different approach so let's discuss how you can help to make this possible.

Since #load is not working we need to define our own version of #load in FAKE:

• Create a hashset with the loaded files in the FSIHelper.fs.
• The hashset should probably be initialized with the current script.
• Create a new load function in the FSIHelper.fs which loads a file only when it is not already loaded (see http://stackoverflow.com/questions/20395062/load-a-file-into-fs-fsi-without-using-load for ideas)
• Add good XML docs to the function
• Bonus points: Write a small turorial in https://github.com/fsharp/FAKE/tree/develop/help
• Send a pull request.

I made this issue "up-for-grabs" so please tell me if you want to do this.

[silky]

@forki Thanks for thinking about how to do this.

Can I ask, though, what do you think is wrong with the approach idea in the Gist I posted earlier - https://gist.github.com/silky/bf304785a46db32b5cc8

It doesn't seem as bad as @shishkin implies, because the dependency chain is still enforced to be correct at run-time. It's a bit unfortunate that you don't see the dependencies in the foo_targets.fsx file, but in some sense it's nice and clean.

Re the idea of trying to solve the loading issue; I'd be happy to have an attempt if you think that the Gist solution is pretty bad.

[forki]

There is another way which might work. If you can reliably detect the file and line no. of target definition we could store this and skip the error when we see the exact same definition again.

We probably also want to store the date of the last modification of the file and maybe some other things.

Regarding the target dependencies I think it's safe to allow to specify the exact same dependency twice.

[silky]

I mentioned a few comments ago that the need for RunTargetOr... actually completely breaks the #loading ability, I think. This is why I went with the _build, _targets approach.

You can't, say, only have the RunTargetOr ... in the "base.fsx" because that means it is executed before the other Targets are defined. And you can't have it in any given "a.fsx" or "b.fsx" because that means nobody else can #load that file.

I think that pattern would also need to be changed. I.e. only the last "RunTargetOr..." would be executed. But how could you ever detect that situation? It's not possible.

[forki]

you can actually check if the target is already executed and prevent it from running again. The only problem is you can specify dependencies for targets that have already been executed

[silky]

you can actually check if the target is already executed and prevent it from running again.

Perhaps, but I still think there is a problem.

Take the "Env, Foo" and "Env, Foo, Bar" path.

Two approaches:

1) Have "base" call a "Default" target that everyone (all children) set up. This is immediately wrong, as this runs before any child targets are setup.

2) Have "Foo" Run itself; have "Bar" Run itself. This is also wrong, I think, because the Env target would run twice (and indeed, the Foo target would run twice). I.e. the #load system would read in Foo, run "Env, Foo", then finish with "foo.fsx", continue processing "bar.fsx", then find a new target to Run, and run again Env, Foo, right? (Because it isn't the same Run invocation that called them both, so it's not clear that the targets have already run, or at least it's more correct if it works that way, and I think it does.)

[silky]

I'm considering this resolved with the One True strategy being that contained in this Gist:

• https://gist.github.com/silky/bf304785a46db32b5cc8

(That is, this is what I've now implemented in our actual system, and it works nicely.)