mdeweerd commented 3 years ago

Despite the '%v' in the directory for the compress target, the file is written to a directory not including the file_id for the variant defined by the variant global.

'%v' should be used to allow putting the zip file in a variant related directory.

For "gerb_drill", the same is true. Drill files are the same for all variants but they need to be available with each variants output.

global:
  output: "%f_%r_%d-%i.%x"
  # dir: "%f_%r_%d%v"  
  variant: '5V_3btn'

  # Archiver (files compressor):
  # This is used to generate groups of files in compressed file format.
  - name: 'compress example'
    comment: 'Generates a compressed file containing output files.'
    type: 'compress'
    dir: "%f_%r_%d%v/ZIPS"
    options:
      # [string='auto'] [auto,stored,deflated,bzip2,lzma] Compression algorithm. Use auto to let KiBot select a suitable one
      compression: 'auto'
      # [list(dict)] Which files will be included
      files:
        # [string=''] Destination directory inside the archive, empty means the same of the file
        - dest: ''
          # [string='.*'] A regular expression that source files must match
          filter: '.*'
          # [string=''] Collect files from the selected output.
          # When used the `source` option is ignored
          from_output: ''
          # [string='*'] File names to add, wildcards allowed. Use ** for recursive match.
          # Note this pattern is applied to the output dir specified with -d comman line option
          source: 'PCB'
      # [string='ZIP'] [ZIP,TAR,RAR] Output file format
      format: 'ZIP'
      # [string='%f-%i%v.%x'] Name for the generated archive (%i=name of the output %x=according to format). Affected by global options
      # output: '%f-%i%v.%x'

Gerb_drill:

  - name: "gerbers"
    comment: "Gerbers for production"
    type: 'gerb_drill'
    dir: "%f_%r_%d%v/pdf"
    options:
      # [dict|string] [hpgl,ps,gerber,dxf,svg,pdf] Format for a graphical drill map.
      # Not generated unless a format is specified
      map:
        # [string='%f-%i%v.%x'] Name for the map file, KiCad defaults if empty (%i='PTH_drill_map'). Affected by global options
        # output: '%f-%i%v.%x'
        # [string='pdf'] [hpgl,ps,gerber,dxf,svg,pdf] Format for a graphical drill map
        type: 'pdf'
      # [string] Force this replacement for %i when generating NPTH files
      npth_id: None
      # [string='%f-%i%v.%x'] name for the drill file, KiCad defaults if empty (%i='PTH_drill'). Affected by global options
      # output: '%f-%i%v.%x'
      # [string] Force this replacement for %i when generating PTH and unified files
      pth_id: None
      # [dict|string] Name of the drill report. Not generated unless a name is specified
      report:
        # [string=''] Name of the drill report. Not generated unless a name is specified.
        # (%i='drill_report' %x='txt')
        filename: ''
      # [boolean=false] Use the auxiliary axis as origin for coordinates
      use_aux_axis_as_origin: false

set-soft commented 3 years ago

Without a test case I won't investigate it.

mdeweerd commented 3 years ago

OK ;-).

mdeweerd commented 3 years ago

@set-soft The test case is in the following package. tc1.tar.gz

TC1.md

Demonstrates that multiple output directories are created where only 1 is expected.

Usage

Run make to run the default target, which run kibot.

Discussion

The goal of setting a global directory using paramters is to generate all the files in a single output directory that is still dependent on the date ("version"), and variant, and project name, etc.
Generating files to different directory is not helping with that goal.

Therefore, only root output directory is expected which would look like kibom-variant_4__<DATE>_leftvariant .

However, 4 output directories are generated:

kibom-variant_4__2021-11-19_17-50-53_leftvariant
kibom-variant_4__2021-11-20_14-57-29
kibom-variant_4__2021-11-20_14-57-29_leftvariant
kibom-variant_4__2021-11-20_14-57-29_rightvariant

The 'global' setting 'dir' should depend on the global variant used for running the rest of the script.
The goal is to have one output directory for each variant.
Some files may be repeated/regenerated (e.g., the board will always be the same), but at least the files are all located in the 'variant' directory which can be considered a package for distribution.

Also not that when the date is read from the schematic or pcb, only the date is used. When it is coming from the file then the time is added. That's not an issue for me (I'll use the PCB date), but it's not uniform.

We can also see that there are two different dates - one is the date of the schematic and the other the date of the PCB.
While we could suppose that the user notices it, the user may also overlook that the files are not all in the same directory.
Several solutions:

Generate a warning or error when the dates are not the same;
Use the most recent date do determine the global output directory.

Files

Scripts/Documentation

TC1.md - This file
Makefile - Makefile where the default target runs the test case
runInKiAuto - Local script to run kibot in Docker VM using Cygwin on Windows 10

Output files

kibot.log - Trace (console) from testcase run
kibom-variant_4__* - Output directories generated using kibot tests

Input files

tests/yaml_samples/step_variant_2.kibot.yaml - Main configuration for this test case
tests/yaml_samples/PCBWay2.kibot.yaml - Configuration to generate PCBWay production files - variation on example
tests/board_samples/kicad_5/kibom-variant_4.pro - Project file for Variant 4 project
tests/board_samples/kicad_5/kibom-variant_4.sch - Schematic for Variant 4 project
tests/board_samples/kicad_5/kibom-variant_4.kicad_pcb - PCB Design file for Variant 4 project
tests/board_samples/kicad_5/steps - Directory with some steps used in the PCB footprint models

set-soft commented 3 years ago

The 'global' setting 'dir' should depend on the global variant used for running the rest of the script.

Nope. This can't be this way. You could want to generate more than one variant from the same script in the same run. A variant is an attribute for an output. The fact that you can define a global default doesn't mean it will override local definitions. Also: the global dir is just a default for the outputs dir, and then dir is expanded for each particular output.

You are defining topvariant globally, but then you are generating the STEP files for top and left variants. They won't go to the same directory if you use %v in the dir. I think you should generate them in different runs. Or you could generate both, but then use compress with hardcoded names to separate them.

And the main problem I see with your example is that compress and gerber wasn't meant to be related to a variant. This can be changed. More below.

Another problem I see here:

kibom-variant_4__2021-11-19_17-50-53_leftvariant kibom-variant_4__2021-11-20_14-57-29_leftvariant

This is because you are using the %d value and it will expand to the pcb/sch date from metadata if available, file modification date otherwise and not %D which expands to date the script was started.

Another detail that confused me is that topvariant has a file_id _leftvariant ... and the comments for top and left are the same.

Anyways, from all of this I can rescue the same idea exposed in the subject Expand '%v' to variant file_id in directory specification for compress & gerb_drill target. Which in practice translates to: When using %v for a target not related to variants try to expand the file_id for the global variant.

This is the patch that I'm applying. Feel free to open other issue to discuss any other detail.

mdeweerd commented 3 years ago

The 'global' setting 'dir' should depend on the global variant used for running the rest of the script.

Nope. This can't be this way. You could want to generate more than one variant from the same script in the same run. A variant is an attribute for an output. The fact that you can define a global default doesn't mean it will override local definitions. Also: the global dir is just a default for the outputs dir, and then dir is expanded for each particular output.

Currently the "script" can be run for one global variant at a time - this is the variant that I'ld expect in the "dir".
I'ld expect "dir" to be expanded before applying it, especially when the output dir is specified as '+SUBDIR'.
Yes a specific output can have its own specific variant - there are some good reasons to not use the global variant. It should not change the global directory.
'lef' and 'right' in the test case was not meant to be used for it's meaning, but for what it does. Better semantic examples:
- Always provide a complete BoM for reference, in case a component must be added later, but the toolset is not going to be run again;
- Always provide a full step file (for similar reasons, to ensure that all components will fit);
- Always provide CPL for all components.

You are defining topvariant globally, but then you are generating the STEP files for top and left variants. Was meant as technical example of two different variants, I could have named them a and b.

And the main problem I see with your example is that compress and gerber wasn't meant to be related to a variant. This can be changed. More below.

This is because you are using the %d value and it will expand to the pcb/sch date from metadata if available, file modification date otherwise and not %D which expands to date the script was started.

Yes, that is what I want as the output directory is meant to correspond to the dates in the sch/pcb. That way all documents are "in line" with each other.

Another detail that confused me is that topvariant has a file_id _leftvariant ... and the comments for top and left are the same. Ok, sorry, didn't mean the semantics.

Anyways, from all of this I can rescue the same idea exposed in the subject _Expand '%v' to variant file_id in directory specification for compress & gerbdrill target. Which in practice translates to: _When using %v for a target not related to variants try to expand the fileid for the global variant.

Ok, this helps for gerbers, compress (not sure it helps for the JLCPCB cpl and bom example).

Would be happy if you agree to apply the same if the directory is specified as '+Something" - for completeness, if '+Something' is '+%v', and global dir also has '%v', then the '%v' in the global dir would be the global variant, and the '%v' in the output would be the local variant.

This is the patch that I'm applying. Feel free to open other issue to discuss any other detail.

Ok, for the moment I replied here, if you do not pick up the discussion, I'll copy this.

set-soft commented 3 years ago

The 'global' setting 'dir' should depend on the global variant used for running the rest of the script.

Nope. This can't be this way. You could want to generate more than one variant from the same script in the same run. A variant is an attribute for an output. The fact that you can define a global default doesn't mean it will override local definitions. Also: the global dir is just a default for the outputs dir, and then dir is expanded for each particular output.

Currently the "script" can be run for one global variant at a time - this is the variant that I'ld expect in the "dir".

I guess you mean: Currently I intend to ... because KiBot doesn't impose this limitation.

I'ld expect "dir" to be expanded before applying it, especially when the output dir is specified as '+SUBDIR'.

This isn't the way things work. As I told you dir is an attribute of output (as the YAML describes).

Yes a specific output can have its own specific variant - there are some good reasons to not use the global variant. It should not change the global directory.

And it doesn't. The global directory is the one you specify using -d command line option, and comes from outside. The dir value in global section is the default value for the dir attribute of each output. And if you use a pattern there you'll get a pattern, then the pattern will be locally expanded. The expansion patterns needs context. As an example %i is very output dependant, and others like %d are really dependant on the file you are processing (schematic vs PCB).

'lef' and 'right' in the test case was not meant to be used for it's meaning, but for what it does. Better semantic examples:

Always provide a complete BoM for reference, in case a component must be added later, but the toolset is not going to be run again;

Always provide a full step file (for similar reasons, to ensure that all components will fit);

Always provide CPL for all components.

You are defining topvariant globally, but then you are generating the STEP files for top and left variants. Was meant as technical example of two different variants, I could have named them a and b.

And the main problem I see with your example is that compress and gerber wasn't meant to be related to a variant. This can be changed. More below.

This is because you are using the %d value and it will expand to the pcb/sch date from metadata if available, file modification date otherwise and not %D which expands to date the script was started.

Yes, that is what I want as the output directory is meant to correspond to the dates in the sch/pcb. That way all documents are "in line" with each other.

But they dates are different! So you'll get one directory name when processing the schematic and another when processing the PCB.

Another detail that confused me is that topvariant has a file_id _leftvariant ... and the comments for top and left are the same. Ok, sorry, didn't mean the semantics.

Anyways, from all of this I can rescue the same idea exposed in the subject _Expand '%v' to variant file_id in directory specification for compress & gerbdrill target. Which in practice translates to: _When using %v for a target not related to variants try to expand the fileid for the global variant.

Ok, this helps for gerbers, compress (not sure it helps for the JLCPCB cpl and bom example).

Would be happy if you agree to apply the same if the directory is specified as '+Something" - for completeness, if '+Something' is '+%v', and global dir also has '%v', then the '%v' in the global dir would be the global variant, and the '%v' in the output would be the local variant.

This looks like magic, I can spect a user to understand it. We could add another pattern, I don't know, may be %g to be expanded as default variant declared in the global section. What do you think?

This is the patch that I'm applying. Feel free to open other issue to discuss any other detail.

Ok, for the moment I replied here, if you do not pick up the discussion, I'll copy this.

If the above method looks good please open a request to add it, so you can have better tracking.

set-soft commented 3 years ago

The above commit is missleading, should be #108 not #104 sorry.

INTI-CMNB / KiBot

Expand '%v' to variant file_id in directory specification for compress & gerb_drill target #104

TC1.md

Usage

Discussion

Files

Scripts/Documentation

Output files

Input files