Closed oliv3r closed 7 months ago
I hadn't run the jlcpcb output for a while, and hadn't spotted the warning before, but it seems like jlcpcb wants the 3D environment, but it's not using it from the global environment. My repo is self-contained, in that the 3D models are in the repo itself.
global: environment: models_3d: '${KIPRJMOD}/packages3D'
Ironically, the render3d and blender jobs generate the 3d models just fine, which also require 3D models of course.
The only thing I found interesting, is that I see files such as
./kibot/resources/config_templates/JLCPCB_with_THT.kibot.yaml
and I also know that kibot generates THT parts on the fly for the blender renders ... but for the fab step, this sounds odd to me. After all, these steps should only be generating gerbers, drill files and assembly instructions?
KiBot always looks for the KiCad environment variables, no matter which outputs are defined.
In the blender run, i see these lines in my log (why are they for both 6 and 7?)
DEBUG:Using KICAD7_3DMODEL_DIR="/workspace/hello/packages3D" (from environment) (kibot - config.py:387) DEBUG:Using KICAD6_3DMODEL_DIR="/workspace/hello/packages3D" (from environment) (kibot - config.py:387)
Because you are using KiCad 7 and KiCad 7 libs needs both. All the official footprints uses KICAD6_3DMODEL_DIR (a bug IMHO), but all the environment variables are named KICAD7_. KiCad is "smart" enough to use any of them, but I don't trust in such a "smart" behavior and defining both is more robust. Here KiBot is informing you that it will apply the same "smart" behavior found in KiCad: use KICAD7_3DMODEL_DIR where KICAD6_3DMODEL_DIR is needed.
So if its working for blender, why not for jlcpcb?
Without an example is hard to know. My guess is that you aren't defining models_3d when invoking the JLCPCB outputs, only when invoking the render outputs, after all you think this isn't needed. And in fact isn't needed and this is why you get a warning, not an error.
I don't think that's true, also because all other fab outputs do work just fine, locally I run without -W
and all other outputs are happy. As you can see from the pipeline (which has full log) and all my files, you'll see that I have 1 file that defines my globals, which all jobs use, so it should work exactly the same for all jobs?
https://gitlab.com/olliver/sandbox/bugs/kici/-/pipelines/1147243394
I'll check now, but could it be that the jlcpcb job defines the global:environment
section, which of course would overwrite any previous imports ... but I can't find it.
Looking at the logs, I see both do:
DEBUG:YAML after expanding definitions:
global:
environment:
models_3d: '${KIPRJMOD}/packages3D'
If you think this is a bug, and not a bizarre use case, create a small example that shows it.
Reduce the workflow to the minimal needed to expose the issue.
Not sure how complex the pipeline is; but here a simple pipeline with just one job, the fab output.
https://gitlab.com/olliver/sandbox/bugs/kici/-/pipelines/1148310601
If I copy the contents of my imported file instead, it does work just fine; so it is related to the fact that things are being imported.
What remains a mystery however, is that PCBway works, but JLCPCB does not.
Looking at the logs, it seems neither gets the globals imported, but only jlcpcb cares ...
I have 2 import statements, which should have been combined into one :)
I think the bug report stands, but I figured it out :) Why does JLC need the 3D stuff, and the others not.
https://gitlab.com/olliver/sandbox/bugs/kici/-/jobs/5992162608
Here's a job that will run just the fab stage, without the import, for pcbway and jlc, I think this is 'smallest' way to show the bug.
Hi @oliv3r !
This is really far from simple. I need something I can feed into KiBot and get the error, not tons of scripts that handles tons of particular details. I need a few KiBot configuration files.
@set-soft not sure what you mean, there's 1 single kibot script in that pipeline,and both kicad projects (hello and world) fail on it.
So to trigger the bug locally, I do:
docker run --pull always --user "$(id -u):$(id -g)" --env HOME=/tmp --rm -it -v $(pwd):/workspace -w /workspace ghcr.io/inti-cmnb/kicad7_auto_full:dev /bin/bash
Status: Downloaded newer image for ghcr.io/inti-cmnb/kicad7_auto_full:dev
I have no name!@f6d0af5f0d76:/workspace$ cd hello
I have no name!@f6d0af5f0d76:/workspace/hello$ kibot -c .kibot/kici_fab.kibot.yaml
Using SCH file: hello.kicad_sch
- 'Gerbers compatible with PCBWay' (_PCBWay_gerbers) [gerber]
- 'Drill files compatible with PCBWay' (_PCBWay_drill) [excellon]
- 'Gerbers compatible with JLCPCB' (_JLCPCB_gerbers) [gerber]
- 'Drill files compatible with JLCPCB' (_JLCPCB_drill) [excellon]
- 'Pick and place file, JLCPCB style' (_JLCPCB_position) [position]
- 'BoM for JLCPCB' (_JLCPCB_bom) [bom]
WARNING:(W008) Unable to find KiCad configuration file (/tmp/.config/kicad/7.0/kicad_common.json) (kibot - config.py:199)
WARNING:(W010) Unable to find KiCad user templates (kibot - config.py:433)
WARNING:(W010) Unable to find KiCad 3D models (kibot - config.py:433)
- 'ZIP file for PCBWay' (_PCBWay_compress) [compress]
- 'ZIP file for JLCPCB' (_JLCPCB_compress) [compress]
Found 3 unique warning/s (3 total)
(or kibot -c ../.kibot/kici_fab.kibot.yaml
does the same, so it's not related to the symlink or anything.)
E.g. To reprodcue only 4 files are needed, 3 kicad files and 1 fab.yaml file. nothing else, no scripts or anything strange :)
(I forgot to do kicad-cli --version
to remove W008, and I haven't seen the user-templates one, but that one doesn't show in the pipeline, so not too bothered there).
@set-soft not sure what you mean, there's 1 single kibot script in that pipeline,and both kicad projects (hello and world) fail on it.
But take a look at your CI/CD workflow:
# SPDX-License-Identifier: AGPL-3.0-or-later
#
# Copyright (C) 2024 Olliver Schinagl <oliver@schinagl.nl>
include:
# Do not use $CI_COMMIT_SHA in any project, this is for testing here only!
- component: $CI_SERVER_HOST/$CI_PROJECT_PATH/gitlab-component@$CI_COMMIT_SHA
inputs:
stage_lint: 'lint'
stage_build: 'build'
attach_diff: 'true'
projects: 'hello world'
- project: 'ci-includes/masslinter'
ref: 'master'
file: 'all-linters.yml'
- project: 'ci-includes/pipeline-deployers'
ref: 'master'
rules:
- if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'
file:
- 'announce-release.yml'
- 'release-branch.yml'
workflow:
rules:
- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
- if: '$CI_COMMIT_BRANCH =~ /^release\/v\d+.\d+$/'
- if: '$CI_COMMIT_TAG'
- if: '$CI_MERGE_REQUEST_IID'
- if: '$CI_PIPELINE_SOURCE == "web"'
stages:
- lint
- build
- branch
- deploy
- complete
default:
tags:
- docker
Where is the KiBot call here? I won't investigate it ...
So to trigger the bug locally, I do:
docker run --pull always --user "$(id -u):$(id -g)" --env HOME=/tmp --rm -it -v $(pwd):/workspace -w /workspace ghcr.io/inti-cmnb/kicad7_auto_full:dev /bin/bash Status: Downloaded newer image for ghcr.io/inti-cmnb/kicad7_auto_full:dev I have no name!@f6d0af5f0d76:/workspace$ cd hello I have no name!@f6d0af5f0d76:/workspace/hello$ kibot -c .kibot/kici_fab.kibot.yaml Using SCH file: hello.kicad_sch - 'Gerbers compatible with PCBWay' (_PCBWay_gerbers) [gerber] - 'Drill files compatible with PCBWay' (_PCBWay_drill) [excellon] - 'Gerbers compatible with JLCPCB' (_JLCPCB_gerbers) [gerber] - 'Drill files compatible with JLCPCB' (_JLCPCB_drill) [excellon] - 'Pick and place file, JLCPCB style' (_JLCPCB_position) [position] - 'BoM for JLCPCB' (_JLCPCB_bom) [bom] WARNING:(W008) Unable to find KiCad configuration file (/tmp/.config/kicad/7.0/kicad_common.json) (kibot - config.py:199) WARNING:(W010) Unable to find KiCad user templates (kibot - config.py:433) WARNING:(W010) Unable to find KiCad 3D models (kibot - config.py:433) - 'ZIP file for PCBWay' (_PCBWay_compress) [compress] - 'ZIP file for JLCPCB' (_JLCPCB_compress) [compress] Found 3 unique warning/s (3 total)
Next time modify your workflow to just do this.
(or
kibot -c ../.kibot/kici_fab.kibot.yaml
does the same, so it's not related to the symlink or anything.)E.g. To reprodcue only 4 files are needed, 3 kicad files and 1 fab.yaml file. nothing else, no scripts or anything strange :)
(I forgot to do
kicad-cli --version
to remove W008, and I haven't seen the user-templates one, but that one doesn't show in the pipeline, so not too bothered there).
Running the example you show the only problem is that you forgot to import kici_common.kibot.yaml:
diff --git a/.kibot/kici_fab.kibot.yaml b/.kibot/kici_fab.kibot.yaml
index 60a2849..ce1e88a 100644
--- a/.kibot/kici_fab.kibot.yaml
+++ b/.kibot/kici_fab.kibot.yaml
@@ -2,6 +2,7 @@ kibot:
version: 1
import:
+ - file: kici_common.kibot.yaml
- file: 'PCBWay'
definitions:
_KIBOT_MANF_DIR: 'outputs/fab/PCBWay'
This removed the warning, and remember to also remove the unneeded run_erc: false stuff
@set-soft not sure what you mean, there's 1 single kibot script in that pipeline,and both kicad projects (hello and world) fail on it.
But take a look at your CI/CD workflow:
include: # Do not use $CI_COMMIT_SHA in any project, this is for testing here only! - component: $CI_SERVER_HOST/$CI_PROJECT_PATH/gitlab-component@$CI_COMMIT_SHA inputs: stage_lint: 'lint' stage_build: 'build' attach_diff: 'true' projects: 'hello world'
Where is the KiBot call here? I won't investigate it ... Right here in the template; but it's not relevant, no need to investigate :)
So to trigger the bug locally, I do:
docker run --pull always --user "$(id -u):$(id -g)" --env HOME=/tmp --rm -it -v $(pwd):/workspace -w /workspace ghcr.io/inti-cmnb/kicad7_auto_full:dev /bin/bash Status: Downloaded newer image for ghcr.io/inti-cmnb/kicad7_auto_full:dev I have no name!@f6d0af5f0d76:/workspace$ cd hello I have no name!@f6d0af5f0d76:/workspace/hello$ kibot -c .kibot/kici_fab.kibot.yaml Using SCH file: hello.kicad_sch - 'Gerbers compatible with PCBWay' (_PCBWay_gerbers) [gerber] - 'Drill files compatible with PCBWay' (_PCBWay_drill) [excellon] - 'Gerbers compatible with JLCPCB' (_JLCPCB_gerbers) [gerber] - 'Drill files compatible with JLCPCB' (_JLCPCB_drill) [excellon] - 'Pick and place file, JLCPCB style' (_JLCPCB_position) [position] - 'BoM for JLCPCB' (_JLCPCB_bom) [bom] WARNING:(W008) Unable to find KiCad configuration file (/tmp/.config/kicad/7.0/kicad_common.json) (kibot - config.py:199) WARNING:(W010) Unable to find KiCad user templates (kibot - config.py:433) WARNING:(W010) Unable to find KiCad 3D models (kibot - config.py:433) - 'ZIP file for PCBWay' (_PCBWay_compress) [compress] - 'ZIP file for JLCPCB' (_JLCPCB_compress) [compress] Found 3 unique warning/s (3 total)
Next time modify your workflow to just do this. But the workflow IS just this, you can see here: https://gitlab.com/olliver/sandbox/bugs/kici/-/blob/simple_test/templates/gitlab-component.yml?ref_type=heads#L139
kibot \ ${DEBUG_TRACE_CI:+--verbose --verbose --verbose --verbose} \ --defs-from-env \ --no-warn 9,132 \ --plot-config "${_job}" \ --stop-on-warnings ${KIJOB_ATTACH_DIFF_PDF:+atta
the rest of the pipeline just creates some folders and exports some variables.
>
> > (or `kibot -c ../.kibot/kici_fab.kibot.yaml` does the same, so it's not related to the symlink or anything.)
> > E.g. To reprodcue only 4 files are needed, 3 kicad files and 1 fab.yaml file. nothing else, no scripts or anything strange :)
> > (I forgot to do `kicad-cli --version` to remove W008, and I haven't seen the user-templates one, but that one doesn't show in the pipeline, so not too bothered there).
>
> Running the example you show the only problem is that you forgot to import _kici_common.kibot.yaml_:
I removed it altogether, I made it as simple as possible, just to show, that PCBWAY does NOT require the 3d environment variable, but JLCPCB DOES require the 3d environment variable!
>
> ```diff
> diff --git a/.kibot/kici_fab.kibot.yaml b/.kibot/kici_fab.kibot.yaml
> index 60a2849..ce1e88a 100644
> --- a/.kibot/kici_fab.kibot.yaml
> +++ b/.kibot/kici_fab.kibot.yaml
> @@ -2,6 +2,7 @@ kibot:
> version: 1
>
> import:
> + - file: kici_common.kibot.yaml
> - file: 'PCBWay'
> definitions:
> _KIBOT_MANF_DIR: 'outputs/fab/PCBWay'
> ```
>
> This removed the warning, and remember to also remove the unneeded _run_erc: false_ stuff
Ofcourse this removes the warning, as that set the needed environment variable. However, JLCPCB needs it, PCBWAY does not. That was my point :)
the rest of the pipeline just creates some folders and exports some variables.
Next time just put it as the only action, not included from some other place.
Ofcourse this removes the warning, as that set the needed environment variable. However, JLCPCB needs it, PCBWAY does not. That was my point :)
KiBot needs it. If you don't like the warning just filter it or provide a definition.
KiBot needs it. If you don't like the warning just filter it or provide a definition.
I'm fine with kibot needing it, but why not with PCBWAY (and the other fabs), why ONLY with JLCPCB?
I'm fine with kibot needing it, but why not with PCBWAY (and the other fabs), why ONLY with JLCPCB?
Different outputs, different data needs to be collected.
KiBot tries to avoid collecting everything and then do the work, instead collects data as needed. But some data comes in packs, so when you get the KiCad environment all the environment is solved, not just one variable.
I'm fine with kibot needing it, but why not with PCBWAY (and the other fabs), why ONLY with JLCPCB?
Different outputs, different data needs to be collected.
And that is perfectly and fully understandable. I can reason with logic :) A job that requires the 3d models, will require the 3d paths to be setup :)
KiBot tries to avoid collecting everything and then do the work, instead collects data as needed. But some data comes in packs, so when you get the KiCad environment all the environment is solved, not just one variable.
Sure, which is why I said initially, I figured it out, my include was wrong, I can work around the problem by just always defining the environment, no problem.
However, the question remains, what is so different about JLCPCB that requires 'more' of the variables? With that 'pack' coment though, I figure, because we also need POS files and other things for JLC's assembly service, that requires more environment variables, which 'accidentally' also needs the 3d environment stuff, even though JLCPCB doesn't really do anything with the 3D models.
Could have just said that :p but it's mostly clear now. So if you think it's not worth the effort to 'fix' this, I fully agree and understand.
Hi @oliv3r !
The above patch detects official docker images, GitLab and GitHub CI/CD workflows. In these cases W009 is omitted and two cases of W010 are also ignored.
We could add more warnings that are always found on CI/CD workflows.
Of course this behavior can be disabled, just using --warn-ci-cd
I hadn't run the jlcpcb output for a while, and hadn't spotted the warning before, but it seems like jlcpcb wants the 3D environment, but it's not using it from the global environment. My repo is self-contained, in that the 3D models are in the repo itself.
Ironically, the render3d and blender jobs generate the 3d models just fine, which also require 3D models of course.
The only thing I found interesting, is that I see files such as
./kibot/resources/config_templates/JLCPCB_with_THT.kibot.yaml
and I also know that kibot generates THT parts on the fly for the blender renders ... but for the fab step, this sounds odd to me. After all, these steps should only be generating gerbers, drill files and assembly instructions?In the blender run, i see these lines in my log (why are they for both 6 and 7?)
So if its working for blender, why not for jlcpcb?
The JLCPCB log bit below