Cache compiled script on disk

patriksvensson commented 6 years ago

Would be nice if we could cache compiled script to disk if it haven't changed.

guillaume86 commented 6 years ago

FYI I experimented a bit with caching the compiled script. I'd like to use Cake as a task runner and the 4 sec penalty seemed a bit much for running tasks that take under 1 sec on their own.

I just used Emit(path) to generate an assembly and figured out how to call it by reflection, and it seems encouraging:

normal run

Analyzing build script...
Processing build script...

========================================
Default
========================================
Executing task: Default
Hello World!
Finished executing task: Default

Task                          Duration
--------------------------------------------------
Default                       00:00:00.0114282
--------------------------------------------------
Total:                        00:00:00.0114282

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 5
Milliseconds      : 448
Ticks             : 54489236
TotalDays         : 6,30662453703704E-05
TotalHours        : 0,00151358988888889
TotalMinutes      : 0,0908153933333333
TotalSeconds      : 5,4489236
TotalMilliseconds : 5448,9236

compile run (build cache and run)

Analyzing build script...
Processing build script...

========================================
Default
========================================
Executing task: Default
Hello World!
Finished executing task: Default

Task                          Duration
--------------------------------------------------
Default                       00:00:00.0146321
--------------------------------------------------
Total:                        00:00:00.0146321

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 5
Milliseconds      : 347
Ticks             : 53478949
TotalDays         : 6,1896931712963E-05
TotalHours        : 0,00148552636111111
TotalMinutes      : 0,0891315816666667
TotalSeconds      : 5,3478949
TotalMilliseconds : 5347,8949

cached run

Analyzing build script...
Processing build script...

========================================
Default
========================================
Executing task: Default
Hello World!
Finished executing task: Default

Task                          Duration
--------------------------------------------------
Default                       00:00:00.0095663
--------------------------------------------------
Total:                        00:00:00.0095663

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 689
Ticks             : 6890002
TotalDays         : 7,97453935185185E-06
TotalHours        : 0,000191388944444444
TotalMinutes      : 0,0114833366666667
TotalSeconds      : 0,6890002
TotalMilliseconds : 689,0002

It shaved off 4.5 secs for cached runs and is now acceptable to run quick tasks, it will probably help with #2179.

I attached the changed file if anyone is interested (lots of things missing for a proper pull request). RoslynScriptSession.cs.txt

alberto-chiesa commented 5 years ago

Hi, I would like to try to tackle this task and I'm looking for some suggestion on how to proceed. I have some questions:

Where should I store the precompiled assembly? I see two options: (a) beside the build.cake file as a build.cake.dll and (b) inside a subdirectory under the cake.exe runner (like ./tools/cake/cache)
Do you think the last modified date is a good enough heuristic to check if a new compilation is due? I mean, before running the cache is checked and, if a compiled dll is found, the compilation date is compared against the one of the cakefile. If the compilation is after the last modified date of the cakefile, the precached version is used. An alternative is storing the dll as build.cake.md5ofthecakefile.dll and check the hash.
Do you think the precaching should be enabled by default, opt-in or opt-out? In case some command line switch should be added, can I have some pointers on where to look for example code?

Thanks!

gep13 commented 5 years ago

@alberto-chiesa there is an open RFC here: https://github.com/cake-build/rfcs/pull/2 which discusses this, and we would need to get sign off there on the approach before proceeding with any work on this feature.

alberto-chiesa commented 5 years ago

Thanks @gep13 ! I didn't knew about the RFC repo. It seems the linked proposal answers my exact same questions. Nice to see I wasn't too off-target. Continuing there about the points to discuss.

alberto-chiesa commented 5 years ago

Thanks @gep13 ! I didn't knew about the RFC repo. It seems the linked proposal answers my exact same questions. Nice to see I wasn't too off-target. Continuing there about the points to discuss.

jamiegs commented 5 years ago

Is there any updates on this? the slowness of the startup of cake has been fairly annoying for me and I'm about to look into switching to Nuke for this reason only.

patriksvensson commented 5 years ago

@jamiegs No update on this since no one have submitted a PR for it yet.

devlead commented 5 years ago

@jamiegs happy to take contributions on this, it's not that we don't want this Core team currently have a few other things with higher priority atm.

If all you want is a is a pre compiled .NET Core app you could take a look at Frosting or Cake.Bridge

https://github.com/cake-build/frosting

https://github.com/devlead/Cake.Bridge/tree/master/src/CakeConsoleRunner

Frosting is a more Asp.Net core style and Cake.Bridge is more like 1:1 mapping of Cake script to console app.

GeertvanHorrik commented 3 years ago

Our situation:

We currently use (quite large) shared build scripts in all our repositories. Each stage (initialize, build, test, package, deploy, finalize) is a separate stage and requires recompilation of the cake scripts. Compiling the build scripts takes about 20 seconds, so it would be very sweet if we could decrease this to 0.

Since we have different agents (although the build server tries to run everything for a specific build on the same agent), we would like to cache this. I see 2 options:

In the %temp% based on the commit id (a huge improvement for us would be if we could cache based on commit id (saves us compiling 5 times (so 1 instead of 6)).
In the %temp% based on the combined hash of the script files (I am pretty sure that calculating the hash is faster than compiling). Then we could re-use the cache for the same build scripts for different repros
In the repository itself, but then we would have to be aware of recompilation every time. In our case that's not really an issue since we use RepositorySync so we only have to do it once.

My preferred option is 2. If there is anything I can do to help, let me know.

gep13 commented 3 years ago

@GeertvanHorrik thanks for your thoughts. Not sure if you have seen it, but there is an open RFC regarding the addition of caching into Cake. Before moving forward with any implementation, we need to get agreement regarding the way forward. This RFC is here:

https://github.com/cake-build/rfcs/pull/2

If you wanted to read through that (if you haven't already) and add some thoughts to the PR, we can continue to refine the RFC, and then begin on the implementation.

devlead commented 3 years ago

There's an open PR #2650 to be reviewed after 1.0 right?

gep13 commented 3 years ago

@devlead yes, I believe that there is a PR for providing some of the functionality, however, we haven't agreed via the RFC exactly what we are hoping to achieve with this functionality, so until that is addressed, I don't think we can move forward with any PR.

devlead commented 3 years ago

Yip just wanted to make sure we didn't duplicate any effort

cake-build-bot commented 2 years ago

:tada: This issue has been resolved in version v2.2.0 :tada:

The release is available on:

Your GitReleaseManager bot :package::rocket:

RehanSaeed commented 2 years ago

Any reason this is not turned on by default? Is it just off for the initial release? I'd prefer to avoid adding yet another config file to the root of my projects enable this feature if possible.

augustoproiete commented 2 years ago

@RehanSaeed Yes, it's a complex feature... We'll let it stabilize first and consider turning on by default in a future release

devlead commented 2 years ago

New major features generally opt-in to break as few builds as possible and enable iron out any kinks before enabling it for all.

Config file isn't required you can also configure via environment variable or command line arg

https://cakebuild.net/docs/running-builds/configuration/default-configuration-values#cache-compiled-script-on-disk

cake-build / cake