site for hosting InteractiveHtmlBom for open hardware projects?

[aewallin]

Is there a site that is willing to host InteractiveHtmlBom pages for open hardware projects?

Similar to Travis-builds, testing-dashboards etc? Ideally the BOM-page would autogenerate once you commit to the git-repo of the project.

thanks!

[qu1ck]

You should be able to set this up with github pages like I did with demo repository. I generate my demos with a script but setting up travis build to do it for you should be straightforward.

[damped]

I have been trying to build BOMs using gitlab's CI/CD build service and then hosting it there.

image: rjw57/kicad     #This is just the first kicad image I found on docker hub

pages:
        script:
                - ls
                - apt-get update -qq && apt-get install -y git
                - git clone https://github.com/openscopeproject/InteractiveHtmlBom.git
                - mkdir public
                - python2 InteractiveHtmlBom/InteractiveHtmlBom/generate_interactive_bom.py /pcb/pcb.kicad_pcb
                #- mv out.html /public/out.html    #will change this to the output file name when it works
                - ls

        artifacts:
                paths:
                        - public
        only:
                - master

I'm not an expert on this stuff so I have been getting an error when generate_interactive_bom.py is run. Unable to access the X Display, is $DISPLAY set properly? I'm guessing this is because its running headless?

Do you know how I can run the script without a display?

Thanks for making such an amazing tool!

[qu1ck]

Yes you need to install dummy X server xserver-xorg-video-dummy package should do it on ubuntu

See also #59

- mv out.html /public/out.html #will change this to the output file name when it works

You can use --dest-dir flag to point script to desired output directory

[damped]

Thanks so much for your help. I found this project which managed to run kicad headless. They used Xvfb :0 -screen 0 1024x768x16 &> xvfb.log & to simulate a display. I will try your suggestion next.

image: python:2.7

pages:
        script:
                - ls
                - apt-get update -qq && apt-get install -y xvfb git kicad
                - Xvfb :0 -screen 0 1024x768x16 &> xvfb.log  &
                - git clone https://github.com/openscopeproject/InteractiveHtmlBom.git
                - mkdir public
                - python2 InteractiveHtmlBom/InteractiveHtmlBom/generate_interactive_bom.py /pcb/pcb.kicad_pcb --dest-dir public

        artifacts:
                paths:
                        - public
        only:
                - master

I noticed that the docker image was giving me kicad 4 so I changed some things. Now I get an import error. ImportError: No module named pcbnew

Do you think I should be using a different version of python?

On my local machine pcbnew is only available to python 2.7.15

Sorry for hijacking your question aewallin. If this works, we will be able to host our stuff on gitlab (and probably github too) by including this code in a .gitlab-ci.yml file in a repo with the pcb files.

[qu1ck]

You don't need to actually run KiCad, you only need to run a fake X server. Xvfb is one option, package I mentioned is newer replacement for it, I'm not sure if it's available on all systems.

Python 2.7 is fine, actually that is exactly the one you need. Your issue now is that pcbnew module is not installed even though i see apt-get install kicad in your script. Do you know what system runs gitlab CI? Which repository did you get kicad from? It's possible that the package you use does not have python scripting enabled.

[damped]

In the latest script I changed from using a pre built docker container that has kicad 4 installed to using the python 2.7 container which is apparently is based on debian wheezy. So that's probably why Kicad is so old.

Do you have a recommendation for a ubuntu version that still has python2.7 and a decent version of Kicad?

A docker image of Arch linux seems easier at this point.

[qu1ck]

Both ubuntu 14 and 16 will work. Official ppa has packages for them https://launchpad.net/~js-reynaud/+archive/ubuntu/kicad-5

Don't use 18 since kicad built for that one doesn't support python scripting at the moment.

[damped]

This is what I have so far.

image: ubuntu:16.04

pages:
        script:
                - ls
                - apt-get update -qq && apt-get install -y software-properties-common
                - add-apt-repository ppa:js-reynaud/kicad-5
                - apt-get update -qq && apt-get install -y xvfb git kicad

                - Xvfb :0 -screen 0 1024x768x16 &> xvfb.log  &
                - git clone https://github.com/openscopeproject/InteractiveHtmlBom.git
                - mkdir public
                - python2 InteractiveHtmlBom/InteractiveHtmlBom/generate_interactive_bom.py /pcb/pcb.kicad_pcb --dest-dir public

        artifacts:
                paths:
                        - public
        only:
                - master

Its been running for 15 minutes getting 3d models but it looks good so far. Is there a base package of kicad that will still meet the requirements of InteractiveHtmlBom?

I think I'll call it a night though. Thanks qu1ck, it's been great working with you.

[damped]

It just finished... We're back to Unable to access the X Display, is $DISPLAY set properly?

I'll look at it tomorrow.

[qu1ck]

You only need kicad base package, no need to install libraries or 3d models. Probably apt-get pulls it because they are recommended, try --no-install-recommends flag. To debug display issue check $DISPLAY and also xvfb.log should tell you if something went wrong.

[damped]

The --no-install-recommends worked great! Cut the run time down to 2 minutes.

I saved xvfb.log and echoed $DISPLAY and they are both empty.

I will try xserver-xorg-video-dummy package but I think I need to edit some config files.

[qu1ck]

It appears that DISPLAY will not be set automatically when xvfb is run, try setting it explicitly in your script DISPLAY=:0 Also take a look at xvfb-run http://manpages.ubuntu.com/manpages/xenial/man1/xvfb-run.1.html

[damped]

It's finally working! Looks like we came to the same conclusion with xvfb-run. I should have just refreshed the page...

.gitlab-ci.yml
--------------

image: ubuntu:16.04
#Will save to https://<usrname>.gitlab.io/<repository>/ibom.html

pages:
        script:
                #PCB path name relitive to the root of the repository
                - PCBPATH="pcb/pcb.kicad_pcb"
                - ls
                - apt-get update -qq && apt-get install -y software-properties-common           #For add-apt-repository
                - add-apt-repository ppa:js-reynaud/kicad-5
                - apt-get update -qq && apt-get install -y --no-install-recommends kicad
                - apt-get install -y xvfb git
                - git clone https://github.com/openscopeproject/InteractiveHtmlBom.git

                - mkdir public
                - CURRENTDIR=$(pwd)
                - xvfb-run --auto-servernum --server-args "-screen 0 1024x768x24" python2 InteractiveHtmlBom/InteractiveHtmlBom/generate_interactive_bom.py --no-browser $PCBPATH --dest-dir $CURRENTDIR/public

        artifacts:
                paths:
                        - public
        only:
                - master

This is what it looks like when it's in a gitlab repository: Denford CNC GRBL Controller It should be similar for github.

Thanks for your help qu1ck! Is there anything you would improve?

[qu1ck]

Thanks for sharing final config, I'll put this on wiki for visibility.

Only thing I would do is make sure that if something is broken (either my plugin, which I try to avoid but it happens, or pcb has some unsupported feature etc.) and output is not generated, the script fails and you get some sort of notification to investigate.

Also since your board has nothing of interest on the bottom side you can pass --layer-view F to plugin script and by default layout of the bom will be front only.

[damped]

I added your argument. There might be a better way to do this other than a variable.

.gitlab-ci.yml
--------------
image: ubuntu:16.04
#Will save to https://<usrname>.gitlab.io/<repository>/ibom.html

pages:
        script:
                #PCB path name relitive to the root of the repository
                - PCBPATH="pcb/pcb.kicad_pcb"
                #Arguments to pass to IHB seperated by spaces
                - ARGS="--layer-view F"
                - ls
                - apt-get update -qq && apt-get install -y software-properties-common           #For add-apt-repository
                - add-apt-repository ppa:js-reynaud/kicad-5
                - apt-get update -qq && apt-get install -y --no-install-recommends kicad
                - apt-get install -y xvfb git
                - git clone https://github.com/openscopeproject/InteractiveHtmlBom.git

                - mkdir public
                - CURRENTDIR=$(pwd)
                - xvfb-run --auto-servernum --server-args "-screen 0 1024x768x24" python2 InteractiveHtmlBom/InteractiveHtmlBom/generate_interactive_bom.py --no-browser $ARGS $PCBPATH --dest-dir $CURRENTDIR/public

        artifacts:
                paths:
                        - public
        only:
                - master

When the job fails I get an email. I played with exporting logs on failure but it seemed harder to find those then just looking at the output from the failed job and for some reason when it didn't fail it didn't give me my artifacts.

                paths:
                        - public

                when: on_failure
                paths:
                        - logs
                        - public

Awesome that it's making it on the wiki!

[qu1ck]

Might as well include --no-browser into $ARGS

[damped]

I was keeping it in the big long command since it's not optional and it seemed more intuitive to only have the args that you can change in $ARGS.

If I remove --no-browser I get

2019-01-17 23:46:30,014 INFO Opening file in browser
Traceback ...
OSError: [Errno 2] No such file or directory

You can post it on the wiki how you like it though

[qu1ck]

That makes sense. It is optional, html will still be generated, it just fails to open it in browser later. I still should probably catch that error and print sensible error message.

Anyway, I'll add this info to the wiki in a bit, this bug can be closed.

[ccoenen]

Hey, everyone! I am currently working on a woodpecker CI config, and this conversation helped me tremendously! The preliminary result is currently here: https://codeberg.org/ccoenen/Dustbuster-Communicator/src/branch/main/.woodpecker/kicad_ibom.yml The resulting html is not yet stored anywhere, I'm going to be working on that next.

[damped]

That's awesome @ccoenen! I wonder if there are any updates to this method with KiCad 7 now having a CLI interface which hopefully means it can run truly headless. I haven't been keeping up to date with it though so I'll have to see when I get some time.

[qu1ck]

You can set INTERACTIVE_HTML_BOM_NO_DISPLAY env variable and then ibom will not need a valid display.

[ccoenen]

I also found kibot, which uses ibom: https://github.com/INTI-CMNB/KiBot I'm very much back into experimenting right now :-D