DWG Object Handles imported via ODA do not match Navis

[sebjf]

Description

When importing DWGs via ODA, the Entity ID in the metadata is different to that shown in the Navis tree, as is the layer name in the Tree.

• This is happening in: both
• This requires a hotfix to production: no

Steps to replicate

1. Import the file in the attached zip boxes.zip

Current Behaviour

• The Entity IDs are 253 & 261, whereas in Navisworks they are 30C and 308 (the same as in AutoCAD, where the file was originally authored).

Use the LIST command to check the Handles of selected objects in AutoCAD.

Expected Behaviour

The Entity IDs should match those in Navisworks.

Pictures/ Gifs of the error

This issue is part of: https://github.com/3drepo/3D-Repo-Product-Team/issues/387

Solution

From the investigation below, the DGN type itself is not capable of holding the Autodesk metadata required to marry up the records between different Autodesk files.

The solution is to write a dedicated DWG importer, instead of relying on the DWG to DGN conversion.

Tasks

Note this task list is incomplete - more tasks will be added as we import more complex files and compare to Navis.

• [x] Change FileProcessorDwg inheritance to remove reliance on FileProcessorDgn
• [x] Add a new DataProcessorDwg class to handle the import
• [x] Update cmake files to support loading DWGs natively into an OdDbDatabase
• [x] Ensure we can vectorize geometry from the OdDbDatabase
• [x] Ensure that material conversion methods are called
• [ ] Initialise the palette for colour conversion
• [x] Find the base type of the OdGiDrawable that can be used to extract hierarchical information
• [x] Make sure colour by pen works
• [x] Make sure colour by layer works
• [x] Make sure colour by block works
• [x] Make default material settings look better (no specular highlights etc)
• [x] Set up layers to match hierarchy shown in Navisworks
• [x] Hook up metadata
• [x] Make sure blocks work
• [x] Make sure class names come through
• [x] Don't import geo position markers (or at least stop them appearing as part of another node)
• [x] Create unit tests

[sebjf]

The problem is also seen in the OdaDgnApp sample, which shows the same Ids as the viewer.

A few other things worth remarking on:

1. Exporting from AutoCAD to a DGN results in different Handles in ODA, compared to loading the DWG into OdaDgnApp directly - however these are still not the same as in AutoCAD/Navis.
2. Dumping the DWG to XML with OdDwgCdaDumpEx shows the correct Object Handles under the UniqueSourceID property.
3. When dumping the DGN to XML with OdDgnCdaDumpEx, the UniqueSourceID shows the same Handle IDs as in OdaDgnApp.
4. When importing a DGN into Navisworks, there is no Entity Id available.

[sebjf]

@carmenfan it seems your suspicion is correct and that the DWG->DGN conversion is lossy.

Since DGN is not an Autodesk type, and doesn't seem to contain Autodesk style references in Navis that the clash detection could return, can we assume users won't care about those Ids in DGNs?

Since the references are in the DWGs read by ODA, just not in the Db returned by OdDgnExport, we can try and write a dedicated processor to DWGs and hopefully this should preserve them.

[carmenfan]

Since the references are in the DWGs read by ODA, just not in the Db returned by OdDgnExport, we can try and write a dedicated processor to DWGs and hopefully this should preserve them.

Yea I think the ideal way to go is this (Sorry 😢 )

And leave DGN do it's thing exclusively for DGN

[sebjf]

Hi @AsiteZane @carmenfan,

I now have something substantially functioning. A few questions about how you'd like to behave:

1. Tree

In Navisworks, components resulting from Block references appear under branches of that blocks name. So far, I have set up the DWG exporter to make sure that Entities are in the correct layer, regardless of whether they come from a Block Reference or are assigned directly.

Should I also add branches for the Blocks so it matches Navis exactly, or leave it like it is now?

2. Metadata

In Navis, DWG entities have some metadata associated with them. Navis (via ODA and the plugin) exports these properties. Do we want the DWG exporter to do the same, or just export the Entity Handle?

(These properties are based on the concrete types of each Entity defined by AutoCAD, rather than there being a generic properties array. That is, we'd need to build all of these entries 'by hand', so its straightforward, but quite time consuming).

(For comparison, the only metadata the DGN exporter had was the layer name.)

3. DWGPROPS

Do we want to support the custom properties the user can create at the file-level in DWGPROPS? Assuming its possible to add metadata to the root node of the tree...

4. Names

Should we try and match the names in the Navis tree? This is, make them user friendly versions of the class name instead of the handle? Or leave them as the handles to match the DGN importer.

[carmenfan]

@AsiteZane might think otherwise (in which case go for whatever he says 😆 ), but my thoughts:

1. No, those are instanced groups, we don't want to present them,. Though Layer 2 seems to have one more node in 3drepo.io than navis?
2. No, they're not BIM attributes so I don't think it's relevant.
3. Seems like they only ever store information about the file I'd say leave for now.
4. Ideally yes. but nice to have

[sebjf]

@carmenfan thank you!

Though Layer 2 seems to have one more node in 3drepo.io than navis?

There is a 3D Solid above the Block Reference, so the number of entries in this case is the same. Though there is an issue with Text objects resulting in two entries (a mesh and line) in .io, which should appear as one entity in the tree, which I am trying to figure out...

[carmenfan]

There is a 3D Solid above the Block Reference, so the number of entries in this case is the same. Though there is an issue with Text objects resulting in two entries (a mesh and line) in .io, which should appear as one entity in the tree, which I am trying to figure out...

I've already counted the 3D Solid 😆 Base on the tree symbol in navis, the block reference is an instance group, in which case I would only expect one node being represented in the tree (may be several unnamed geometry nodes underneath it, but those would be hidden from the tree. I.e. I would only expect 2 leaf-looking nodes on the tree instead of 3. Which I guess is what you're discribing with the mesh and line! 👍

[sebjf]

Ah I understand, good you clarified because I was writing it the other way! Though yes the logic I am creating for the text can be applied to that as well.

[sebjf]

Hi @carmenfan @AsiteZane, quick question about how you want the tree to work:

So in AutoCAD, if a Block contains objects on multiple players, when the Block is Referenced, anything on layer 0 adopts the Block Reference's layer, but objects on any other layer remain on their specified layer.

For example, if I had a block with a drawing of a fence on Layer 2, and I referenced it on Layer 3, when I toggle Layer 2, the fence in the reference will toggle, even though the Block Reference itself is on Layer 3. (And turning off Layer 3 will hide everything except the fence.)

I am guessing this is the reason for a Block's objects being split accross the Layer nodes in the Navisworks tree - to allow users to easily switch layers on and off and see the same behaviour as AutoCAD.

Do we want to preserve this behaviour? Or just put all Block Elements under a single node, on the Layer that the Reference is on?

[sebjf]

Hi @carmenfan,

Also, when Block References are nested, do we want to show all levels in the tree, or only the top one?

For example, above Block Text is a Block that is referenced directly by Layer 0, but also by My Block.

[carmenfan]

Hi @carmenfan @AsiteZane, quick question about how you want the tree to work:

So in AutoCAD, if a Block contains objects on multiple players, when the Block is Referenced, anything on layer 0 adopts the Block Reference's layer, but objects on any other layer remain on their specified layer.

For example, if I had a block with a drawing of a fence on Layer 2, and I referenced it on Layer 3, when I toggle Layer 2, the fence in the reference will toggle, even though the Block Reference itself is on Layer 3. (And turning off Layer 3 will hide everything except the fence.)

I am guessing this is the reason for a Block's objects being split accross the Layer nodes in the Navisworks tree - to allow users to easily switch layers on and off and see the same behaviour as AutoCAD.

Do we want to preserve this behaviour? Or just put all Block Elements under a single node, on the Layer that the Reference is on?

Just going to paraphrase what you said to make sure I know what you mean. In essence, block reference is an instance that is being referenced in multiple layers, and in AutoCAD, you're saying if you hide a block reference, it'll hide all instances of references?

Assuming that's what you mean, I don't think we need to preserve this behaviour - it'll be more problematic than anything. Since we duplicate instances in 3drepo right now we may as well just keep doing it.

Also, when Block References are nested, do we want to show all levels in the tree, or only the top one?

Just the top one, so in your example here, I would only expect to see this:

[sebjf]

Hi @carmenfan ,

In essence, block reference is an instance that is being referenced in multiple layers, and in AutoCAD, you're saying if you hide a block reference, it'll hide all instances of references?

Not quite... I'll try and rephrase:

Within a Block, objects can be on different layers, and when a Block is instantiated, those objects remain on the layer that they were assigned inside the block editor.

The only exception is objects on Layer 0, which adopt the Layer that the Block Reference is on.

So if you hide a layer in AutoCAD, it will hide all objects on that Layer in all Block References, even if the Block References themselves are on a different layer.

Just the top one, so in your example here, I would only expect to see this:

👍

[carmenfan]

@sebjf 🤔 how does navis deal with that?

I think we can just match the behaviour with navis. This seems a bit too complex to be represented in our tree?

[sebjf]

🤔 how does navis deal with that?

Navis puts a Node for a Block Reference under each Layer that the Block references, containing only those objects from the Block belonging to that layer.

So in the tree above, there is only one instance of "My Block" in the drawing, and the Block contains three lines. However two of those lines are on Layer 2 and one is on Layer 1, so Navis creates two entries for "My Block", one under each.

[carmenfan]

so they basically split the block reference into multiple objects then?

Would it be difficult to do the same thing?

[sebjf]

@carmenfan

so they basically split the block reference into multiple objects then?

Yes.

Would it be difficult to do the same thing?

I don't think so!

The question is do we want the names of elements within the blocks under the block reference nodes in that case?

[sebjf]

This is a version without:

This implementation splits block entities by layer, but collapses all nested references.

[sebjf]

Hi @carmenfan, a couple of quick questions:

1. What should we do with Geo Position Marker geometry? These indicate GIS positions on a map. They are not entities and do not have Ids.

Navisworks doesn't show them in the 3D view or the tree.

Since they don't have Ids, we can either put them on their own layer/tree node that we name, or just not import the geometry.

2. What should we do with XLines? (Infinity lines)

Navis doesn't show them at all. Currently we import them but they are clipped (by ODA) to the bounds of the view. They are entities and do have IDs (its more effort to not import it, than to just leave it, if you don't care!)

[carmenfan]

1. Ignore is fine (though potentially super useful to know for project settings related stuff for a different feature!)

2. If it's already there let's leave them in (@AsiteZane tagging you just incase you have more thoughts!)

[AsiteZane]

Well, I completely missed this :D

[carmenfan]

Well, I completely missed this :D

@AsiteZane lol not really, you know of it probably just not in the form of this issue

This came from the work in Clash, where we cannot reliably identify dwg objects because we're not importing the IDs right 😆

[carmenfan]

@sebjf just looking at https://github.com/3drepo/3drepoPlugin/issues/514 and it made me wonder...

Are you importing the element ID enclosed in []?

If so we may have a discrepancy with the plugins and navis metadata