[HOW] blender_export missing 3D Models with kicad_3d_url set

[Nudelsalad]

Hi, some time ago I opened an Issue also having problems with the 3D Files missing in the rendering. In a production setup the runner storage (~200Gib) was full within 2 weeks, the 3D models installed and no garbage collector... I thought to give this another try for a bit more storage efficient. I have the official Kicad library with my own symbols and footprints running on my gitlab server. The symbols are a 1v1 clone, every footprint references a 3D Model with the KICAD6_3DMODEL_DIR variable. Here is my config copied from your example file:

kibot:
  version: 1

outputs:
  - name: '3d_export'
    comment: "Exports the PCB in blender format"
    type: blender_export
    options:
      pcb3d:
        solder_paste_for_populated: true
        kicad_3d_url: 'https://gitlab.example.de/hardware/kicad_library/3d_models/-/raw/main/'
      render_options:
        transparent_background: true
        samples: 10
      point_of_view:
        rotate_x: 30
        rotate_z: -20
      outputs:
        - type: blender
        - type: render

The log tells me:

DEBUG:Using `/root/.cache/kibot/3d` as dir for downloaded 3D models (kibot - out_base_3d.py:479)
ERROR:Unable to expand `KICAD6_3DMODEL_DIR` in `${KICAD6_3DMODEL_DIR}/Resistor_SMD.3dshapes/R_0603_1608Metric.wrl` (kibot - log.py:155)
WARNING:(W011) Missing default symbol library table (kibot - config.py:519)
DEBUG:Downloading `https://gitlab.example.de/hardware/kicad_library/3d_models/-/raw/main/Resistor_SMD.3dshapes%2FR_0603_1608Metric.wrl` (kibot - out_base_3d.py:167)
DEBUG:Downloading `https://gitlab.example.de/hardware/kicad_library/3d_models/-/raw/main/Resistor_SMD.3dshapes%2FR_0603_1608Metric.step` (kibot - out_base_3d.py:167)

What I've tried so far:

1. gitlab cache -> referenced with KIBOT_3D_MODELS -> same error
2. setting 3d_models to the cache folder removed the error but also no 3d models inside rendering
3. using another variable instead of KICAD6_3DMODEL_DIR, setting path and rewriting out_base_3d.py -> no errors, files get downloaded and cached but however not in the rendering

Obviously it expects the KICAD6_3DMODEL_DIR variable of the 3d models to be set for expansion, but somehow scrolling through existing issues and your examples make me think that it can not be that complex. Any help appreciated, Thanks

[set-soft]

Hi @Nudelsalad !

I don't fully understand what is the problem. What is exactly failing? You don't get the models in the render? Because they seems to be downloaded. What happens when you use the KiCad repo to get the 3D models?

And you don't need to define KICAD6_3DMODEL_DIR if the models will be downloaded.

[Nudelsalad]

Hi @set-soft, again thank you very much for that bump into the right direction, it was indeed the same gitlab access problem of the runner as mentioned in issue #414 . The whole time it was not able to download actually anything only the files were generated without content (me dumb sry). Building the string actually work with kicad_3d_url and kicad_3d_url_suffix, but however I saw somewhat strange behavior. In your code I noticed that you are sometimes forcing to download also a .wrl model if the .step was pulled and V.V. :

DEBUG:Downloading `https://gitlab.example.de/api/v4/projects/88/repository/files/dynamics_3dmodels%2FIND_WE-SLM.stp/raw?ref=main&private_token=[token]` (kibot - out_base_3d.py:167)
DEBUG:Downloading `https://gitlab.example.de/api/v4/projects/88/repository/files/dynamics_3dmodels%2FIND_WE-SLM.stp/raw?ref=main&private_token=[token].wrl` (kibot - out_base_3d.py:167)
WARNING:(W048) Failed to download `https://gitlab.example.de/api/v4/projects/88/repository/files/dynamics_3dmodels%2FIND_WE-SLM.stp/raw?ref=main&private_token=[token].wrl` (kibot - out_base_3d.py:174)

So as it downloads succesfully the first model it tries also to download the .wrl model which gets appended after the suffix and logically then fails. The rendering looks very good now thank you for your support!

[set-soft]

Hi @Nudelsalad !

Oops! I see, part of the code wasn't aware of the suffix. The above patch should fix it.

Note that KiCad needs wrl + step pairs, in particular: you can't convert a WRL to STEP (but you can create a WRL surface from a STEP file)

[Nudelsalad]

Thank you very much @set-soft, i fiddled a bit around with the CI now everything works as expected thank you very much!

One question left: When having to access artifacts, files or whole repositories from Gitlab CI the CI_JOB_TOKEN comes often into play. It is an easy way to access everything the runner is permitted to access, gets created at runtime of the CI and can be expanded with $CI_JOB_TOKEN. Expanding variables in a yaml file does not seem to work

kicad_3d_url: 'https://gitlab.example.de/api/v4/projects/88/repository/files/'
kicad_3d_url_suffix: '/raw?ref=main&job_token=$CI_JOB_TOKEN'

Is there another way?

[set-soft]

Is there another way?

Here is explained:

https://kibot.readthedocs.io/en/latest/configuration/substitution.html

Use @CI_JOB_TOKEN@ and the --def-from-env command line option

[Nudelsalad]

Thanks for the suggestion, replacing the vars inside the yaml works fine, I feel sorry for asking. However using the CI_JOB_TOKEN currently does not work, I tried every combination possible:

https://gitlab-ci-token:@CI_JOB_TOKEN@@gitlab.example.de/... does not work
https://gitlab.example.de/api/v4/projects/88/repository/files/[file]/raw?ref=main&job_token=@CI_JOB_TOKEN@ does not work
https://gitlab.example.de/api/v4/projects/88/repository/files/[file]/raw?ref=main&private_token=@CI_JOB_TOKEN@ does not work

What does work is using a private access token or creating an access token for the repository and handing that into the yaml.

After reading this: Link I cannot find that the job token even has scope to repositories file api like the personal access token has LINK

May be useful for anyone reading through the issues and being stuck at the same point. For now i go with that solution. Maybe gitlab will implement this anytime. Thanks however @set-soft