Proposal to create IFC test case build server

[Moult]

I am envisioning a workflow where:

1. A server has a repository of many IFC test files
2. The server has the latest IfcOpenBot build of IfcOpenShell
3. The server automatically headlessly opens up each IFC test file in turn with Blender, and takes a render snapshot from a particular angle
4. Results are published online for us to review. If it segfaults, it reports that it failed.

This will help prevent regressions, spot mesh processing errors, and efficiently test IFC geometry processing, which is perhaps quite an arduous process.

Would this be worthwhile doing? I can probably cook something up. I don't think I can get a bunch of different OCC versions, though. Going through Blender is also less efficient than doing it via OCC directly and some sort of OpenGL render, but I don't have the skills for the latter... perhaps there are other benefits for doing it through Blender - it can double up to test BlenderBIM Add-on functionality ...

Edit: I can just do a geometry iterator to check for segfaults as a first pass, but I think a visual render would be very useful :)

[lcpt]

Some years ago I've used CImg to compute the difference between two images to return a measure of how different they are. Maybe this can be useful to automate the tests.

[aothms]

We're quite far with this to only test segfault and return status on all files using IfcConvert.

I like the idea of image comparison as well and probably blender background mode (blender -b -P script.py) is the a good way to get an image from an IFC file. OpenGL to file is a pain, you need mesa offscreen and such and you can't assume it works on a VM due to the OpenGL versions and such. These days with puppeteer you can render a chrome WebGL canvas to a file (but since we have already embraced Blender it should probably be our tool of choice.

@Moult if you can somehow program BlenderBIM to render a screenshot of an IFC to a file from the command line that'd be good progress in that direction (and useful for other purposes too I imagine).

[Moult]

Very cool :) I will find some time and cook something up. I agree that image comparison is cool too! I'll keep you all posted.

A trick I've done in the past for image comparison is to use imagemagick to overlay RB channels from image X and GB channels from from image Y. This gives a red/green overlay - showing red portions removed, green portions added in new image, and greyscale = unchanged. You can calculate the changes numerically, as well as visually spot the changes easily :)

[EAzari]

@Moult Dion, there's a long time I want to say "you're in a wrong path" in IfcDiff but I didn't

Just compression is not a good way, you need "matching technique" too :)

Then the tool will give you: Modified, Deleted, and Unchanged parts

[Moult]

@aothms et all: we now have a thing! https://ci.blenderbim.org/

It is very basic, and I'm feeding it three files at the moment. As you can see, with INCLUDE_CURVES enabled, ifcopenhouse walls go missing :) I can disable INCLUDE_CURVES, but ultimately I think we do need much more control over how different representation contexts are imported.

Also, the third one of the bcf-ifc-pile demonstrates what happens when it borks. You can check the process log to see the python traceback. I've fixed the bug since, but I'm leaving this up so you can see what it looks like when an error occurs. The process log includes the Blender logger output, which can show you which element it last attempted to import before borking (in that case, ifcproject), and it also has stderr appended to the end.

It's very basic still, so no historical log, no image comparison, but it'll improve.

I'd like to feed it bigger and complex files in https://github.com/Moult/ifc-test-files - and then also https://github.com/IfcOpenShell/files - thoughts? Suggestions?

[derbosper]

I think I have a solution for this, which doesn't require comparing images. My idea was to quantify the geometry of each element in an IFC file using OCC, serialize it and then compare with the same serialized element from a environment.

I build a proof-of-concept tool for this and it worked OK with two models and three environments. You can have a look here. The tool does the following things:

• builds all environments from a directory using Docker
• generates OBJ files and CSVs for every environment and model from IFCs in repo
• prints report with elements which geometries are different from the reference environment ones

Report prints failed elements like this:

• for Railing (2y5uU5j0b4Jgf1kOnRnSCh)

  Diff in v0_6_0
  Reference: Product(guid=2y5uU5j0b4Jgf1kOnRnSCh, name=Railing:Handrail - Pipe:178438, has_valid_shape=True, length=468.9775, area=3.0300, volume=0.0178)
  Compared:  Product(guid=2y5uU5j0b4Jgf1kOnRnSCh, name=Railing:Handrail - Pipe:178438, has_valid_shape=True, length=404.3419, area=2.4219, volume=0.0129)

  And the meshes loaded from OBJ files:

• for Wall (3EmpdCRsPEQvlc3klyAdM$)

  Diff in v0_6_0
  Reference: Product(guid=3EmpdCRsPEQvlc3klyAdM$, name=Basic Wall:Generic - 6":379199, has_valid_shape=True, length=171.9472, area=48.0486, volume=3.2206)
  Compared:  Product(guid=3EmpdCRsPEQvlc3klyAdM$, name=Basic Wall:Generic - 6":379199, has_valid_shape=True, length=92.2528, area=56.8102, volume=4.0646)

  Meshes:

Given the GUIDs of faulty products and their meshes in OBJ files, I guess that selected elements can be rendered and uploaded somewhere for preview, along with some better structured report.

Also, for some reason, the environment with ifcopenshell 0.6.0a1 seems to give the best results, but that's another topic.

What do you think about it?

[Moult]

@mieszk absolutely! I never intended the image to be used as the primary means of audit - simply that an image is useful to have when scanning through many test files :) Your solution is much nicer as a primary means of audit.

The OBJ conversion helps test one aspect of IfcOpenShell, that is, the meshifying capabilities of it. That can be in addition to the Blender import conversion, which tests the ability of the BlenderBIM Add-on to process IFCs, which is a bit richer than OBJ, as it needs to retain parametric geometry, multiple materials + surface styles, and object relationships.

I had a look at your code and unfortunately the server running the CI thing I set up right now is on Gentoo, so I can't just drop in your apt-get scripts.

[derbosper]

@Moult In this case the shape volume/area/length is read after creating shapes and before OBJ serialization, which is done mainly with the purpose of viewing meshes later (and OCC meshing is usually done right from what I've seen).

Anyway, I will try to plug there your IFC importer and do similar thing with blender meshes. I'd also like to test it against some more environments and models and see where it goes.

[aothms]

Very cool @mieszk

and OCC meshing is usually done right from what I've seen

For simple things yes, but for more complex situations (advanced breps with nurbs and all) the quantities can all be ok, but meshing might still give improper results (triggered by IfcOpenShell or not)

This does overlap a bit with the --validate option [0] that IfcConvert has. @johanrd has put a lot of effort in annotating files with several properties e.g. [1] and IfcConvert will check these quantities after conversion returning a non-zero exit code if they don't match the value written in the model.

[0] https://github.com/IfcOpenShell/IfcOpenShell/blob/v0.6.0/src/ifcgeom/IfcGeomFunctions.cpp#L1859 [1] https://github.com/IfcOpenShell/files/blob/7502def933685eb5c3f024c0603a8d23abd5c9b6/516--wall--missing-subtractions--augmented.ifc#L648