Optimization request of compile time for -t option

[maheswarasunil]

Describe the feature During kapitan compile for the specific target using option -t , we are observing kapitan is processing the entire inventory and generating compiled output for that specific target. Due to this, compile time is very high even though we need to compile one target.

To Reproduce Steps to reproduce the behavior:

1. Have a high number of target files in inventory/targets files.
2. Compile specific target time kapitan compile -t inventory/targets/target01.yml
3. Remove all the targets except inventory/targets/target01.yml
4. Calculate the compiled time time kapitan compile
5. Compile time at step 2 will be more than step 4. More the target files created at step1 more time it will take at step2.

[adrianchifor]

I've done a quick investigation on why this might happen and the culprit seems to be https://github.com/deepmind/kapitan/blob/v0.27.0/kapitan/resources.py#L234, called first at https://github.com/deepmind/kapitan/blob/v0.27.0/kapitan/targets.py#L312. Calls of inventory_reclass() are cached as it's a pretty expensive operation (~2s for 100 targets), so we've already done some optimisation on our side.

This seems to be a side effect of reclass as it can only load all targets in the inventory - AFAIK you can't load single targets, checked reclass docs and couldn't find anything.

When you specify -t <targets> it will only pass the specific target objects around (post-reclass) and compile just those, so it works as expected on those terms. Worst case scenario, your compilation is delayed by 1-2s because of a large inventory, which is an acceptable cost in my view. Additionally, this is only a problem if your single target compilation takes less than a few seconds, which in not the case in most production targets.

So we have a few options here:

1. Optimise your inventory and targets - the smaller the inventory and the less targets you have, the quicker reclass will process it. If you go above 100 targets, consider splitting the project into smaller chunks.
2. We can invest more time on reclass to debug and make a PR to add specific target loading, but not sure if the cost of debugging+development justifies the 1-2s of compilation saved on -t <targets>.

[github-actions[bot]]

This issue is stale because it has been open for 1 year with no activity. Remove the stale label or comment if this issue is still relevant for you. If not, please close it yourself.