[HOW] Avoid regeneration of all outputs when using navigate_results in a separated run

[oliv3r]

I want to run each of the kibot steps in individual pipeline steps. I do this by writing self-contained kibot yaml files for each task. This works great and I have a folder that has all my outputs.

Now I want to use navigate_results, however I think the script uses the input source as reference what to generate. How can I do this, without re-generating everything?

[set-soft]

You don't need to generate the outputs. You must somehow join all the outputs, i.e. using the import mechanism, plus the navigate output. Then you just need to ask KiBot to generate the navigate output, it will check the targets for each output, it will work even if the files aren't generated yet, but you won't get nice previews.

[set-soft]

Oh BTW! Remember you can always have all the outputs defined and ask KiBot to run some of them

[oliv3r]

@set-soft awesome! This works for the most part.

I have:

import:
  - file: 'blender3d.kibot.yaml'
    outputs:
      - '3D_front_30deg_view'
      - '3D_top_view'
      - '3D_bottom_view'

(and other imports) but it seems like it will generate the outputs each time. First I thought it was because I didn't specify the outputs, so it would source everything from the file, then I thought, if I specify the outputs, then it would use the result, but it runs the blender job each time I run the navigate job, thus re-creating the output everytime. This isn't a problem for most jobs, but the blender job takes like 5 minutes :S

[set-soft]

Hi @oliv3r ! I'm not sure what are you running, when you generate the navigate_results output you should ask KiBot to generate only this output, and not the rest. KiBot will collect the files generated by the other outputs in previous runs.

[oliv3r]

@set-soft I have nearly all outputs and my pipeline as I want it, so will start pushing it, that'll make discussion much easier :) Give me some time, though holidays hopefully would be good for that (probably not :p)

But what I do, is I run one job, (locally), to generate an output. Then, I'll run the container again, with navigate, and was hoping based on the import, and the fact that the output existed, it would not re-generate stuff. Basically, I want to achieve what you said, that navigate just generates its own stuff, not caring if the output actually exists or not.

Right now, I'm just stuck at including the output of the Manufacturing compress stages, and have to somehow figure out the name of the job, but with the above issue, that might not actually be possible, as the compress stage relies on other things ... So I might give up on navigate for the time being :(

[oliv3r]

@set-soft See here my pipeline, https://gitlab.com/olliver/lapod/-/jobs/5831839174 this is a job that runs navigate, and generates all outputs (again). The failure is exactly because of this, blender failed to generate its stuff. This probably due to me setting the output in imports? But this was what was required no? So then how can I run the navigate job, without it trying to generate anything.

The idea is, that the pipeline will run, each step creates its own outputs, and then a final step will collect all outputs together (i've added that later in the pipeline commit series).

[set-soft]

Hi @oliv3r !

The Blender export output is run because the output files are missing. I suggest removing this from your workflow until you solve the Blender config issue. Another option could be to just copy dummy files to pretend they were created. Also note that you can play with the run_by_default flag, navigate_results can skip outputs marked as false.

[oliv3r]

So I've played with this a bit but I cannot make it work exactly like I want.

Simple example. I want to first run the ibom. Store those result some where (outputs/ibom). This means, I have to keep run_by_default: true (or omit this as its the default). This generates the ibom if I run kibot -c ibom.kibot.yaml

If I set run_by_default: false, the yaml becomes 'harder to use' as you always need to set it to run via an kibot argument that is not quite obvious. (e.g. kibot -c ibom.kibot.yaml ibom if by luck that is the name of the job ;) ).

Later, (different day, different pipeline, doesn't matter) I want to use the output from the previously run ibom job. As far as I understand, the only way to do this, is to 'import' the job, so that navigate knows there's an ibom output, and where it is stored.

kibot:
  version: 1

import:
  - file: 'ibom.kibot.yaml'

outputs:
  - name: 'navigate_results'
    comment: 'Web page to browse the results.'
    type: 'navigate_results'
    dir: 'outputs/navigate'

This work exactly as I want, as long as run_by_default: false is set in ibom.kibot.yaml, which is what I don't want :p

So then I I looked into disable_run_by_default: true (but also options: skip_not_run: falsein navigate.kibot.yaml, and all combinations of these two options), in the hopes thatdisable_run_by_defaultwould setrun_by_default: false` for all other jobs. However, as the documentation states:

Useful when this output extends another and you don’t want to generate the original. 

Which seems that it only does what I want to do in case of 'extends'.

So now my question, can I keep run_by_default: true in ibom.kibot.yaml (and pdf.kibot.yaml and etc etc), whilst NOT running them when running navigate.kibot.yaml.

I'd really like to avoid having to supply the job to kibot all the time, which may be unknown (though an all special meaning output could be helpful, oth, that's what it is to not supply it, which gets in the way of run_by_default.

Sadly,

import:
  - file: 'ibom.kibot.yaml'
    outputs: all
      run_by_default: false

doesn't seem to work either :(

[oliv3r]

Hmm, I may have success here :)

output:
    run_by_default: @RUN_BY_DEFAULT@

...
definitions:
  RUN_BY_DEFAULT: true

in my jobs, and then in navigate during import:

import:
  - file: 'kici_ibom.kibot.yaml'
    definitions:
      RUN_BY_DEFAULT: false

It feels slightly hacky, but not too bad :)

For now the 'fab' part is something I can't solve this way I think.

kibot:
  version: 1

import:
  - file: 'kici_common.kibot.yaml'

  - file: 'Elecrow'
    definitions:
      _KIBOT_MANF_DIR: 'outputs/fab/Elecrow'

  - file: 'FusionPCB'
    definitions:
      _KIBOT_MANF_DIR: 'outputs/fab/FusionPCB'

  - file: 'JLCPCB'
    definitions:
      _KIBOT_MANF_DIR: 'outputs/fab/JLCPCB'
      _KIBOT_POS_PRE_TRANSFORM: '[''_kicost_rename'', ''_rot_footprint'']'

  - file: 'PCBWay'
    definitions:
      _KIBOT_MANF_DIR: 'outputs/fab/PCBWay'

...
definitions:
  _KIBOT_MANF_DIR_COMP: 'outputs/fab'

As these are all imports themselves, and I'm stuck with whatever run output is there. On the other hand, does navigate_result even handle these? Yes it does :p

[set-soft]

If you have a simple example (not a ton of included and indirectly called) config where things are generated again for the navigate target when you explicitly ask for the navigate target (of course you must do it from the command line, and of course you can create a wrapper script, lets say generate_navigate that asks for the navigate_results output and not all the outputs) post it.

Otherwise I'll assume your problem is just about how you invoke KiBot.

[set-soft]

For future reference: what you can do is have a config like this:

kibot:
  version: 1

outputs:
  - name: 'interactive_bom'
    comment: "Interactive Bill of Materials (HTML)"
    type: ibom
    dir: BoM

  - name: 'Navigate'
    comment: "Generates the web page to navigate the results"
    type: navigate_results
    run_by_default: false

Then you can generate the iBoM output just invoking kibot. To generate the navigation web pages you just need to run kibot Navigate, the iBoM output is not generated again.