"Invalid XDATA structure" error

[jmacura]

Hi there,

While reading a DXF file (version 'AC1027') I end up with following error:

raise DXFXDataError('Invalid XDATA structure, only group code >=1000 allowed in XDATA section') ezdxf.lldxf.const.DXFXDataError: Invalid XDATA structure, only group code >=1000 allowed in XDATA section

From my understanding of AutoCAD and ezdxf, I suspect that ezdxf does not recognize "basepoint" object in a block definition. That's because if I use a file without these basepoints in the blocks, it runs just fine.

The simple code I use follows:

dxf = ezdxf.readfile("file.dxf")
for b in dxf.blocks:
    print(b.name)
    if not b.name.startswith('*'):
        for bb in b:
            print(bb)

Thanks in advance for any improvement or a simple reply Jan

[mozman]

Hi!

The exception says what it is, an error in the XData structure:

http://ezdxf.readthedocs.io/en/latest/dxfinternals/dxftags.html#extended-data

At the end of the entity is a XData section, starting with a tag (1001, ...), and has a following tag with group code < 1000 which is not allowed. This is an error in the DXF structure.

If you created the DXF file by ezdxf, it would be nice to see the Python code to fix the error in ezdxf. If you created the DXF file by AutoCAD, it would be nice to see the DXF file, to investigate this new feature of AutoCAD.

regards, Manfred

[kwohlfahrt]

I am running into the same error. An example file produced by 'Civils 3D 2018' is available here. Previous files produced from this program were OK. Copying the contents to a new file and exporting a different DXF version all had the same problem.

[jmacura]

Thanks Manfred for your quick reply!

The DXF file was produced in AutoCAD Map 3D 2017. You can find it under following link: https://drive.google.com/file/d/19jjNo0UmDZpe_rF4mJXS5tszQx6-OE1Y/view?usp=sharing

I am quite convinced that the issue is caused by the presence of a "basepoint" in blocks. I tried to open several DXF files today and only those containing "basepoint" features failed to read in ezdxf.

Thanks Jan

[mozman]

Hi!

First a fix, but saving this file with ezdxf damages the entity BLOCKBASEPOINTERPARAMETER:

import ezdxf
ezdxf.options.check_entity_tag_structures = False

The entity BLOCKBASEPOINTERPARAMETER uses extended group codes not at the end of the entity.

DXF reference for extended data.

If an entity contains extended data, it follows the entity's normal definition data.

This entity uses xdata group codes as usual entity group codes.

CAUTION: If you save DXF files with such extended data, ezdxf DAMAGES the DXF file.

Is the entity BLOCKBASEPOINTERPARAMETER created by an Autodesk application or an 3rd party add-on?

[mozman]

The class definition seems to be from Autodesk AcDb...:

[mozman]

BTW:

Autodesk Trueview 2018 does not load this DXF:

Invalid or incomplete DXF input

... same for AutoCAD LT 2018

[jmacura]

Well, I guess it's possible that certain flavors of AutoCAD (Map 3D, Civil 3D) produce invalid (or at least non-standard) DXF files for describing their special features...

Yet I would think that BlockBasepointParameter is something which is present in all AutoCAD versions...

[mozman]

Hi!

The dev branch can now open Civil 3D and Map 3D files, and saving without damaging the DXF structure should also work but without using this applications, I can't guarantee that. (And NO I don't install the test version)

[jmacura]

Hi,

I have installed the dev version of ezdxf and can confirm that the error raises no more, thanks Manfred! Do you plan to implement the Basepoint as a separate entity class in ezdxf? I guess it's the only way, how to resolve the position of inserted blocks in WCS - by reading the coordinates of the basepoint....

[mozman]

No, I don't (can't) implement not documented DXF entities and further more I can not support special applications like Map 3D or Civil 3D.

Block references (INSERT) entity has a DXF attribute "insert", which defines the insert position of the block reference, basic DXF feature since ever. And the block definition has a DXF attribute "base_point" which defines the insert point inside the block definition (default is 0, 0, 0).

[jmacura]

I fully understand, that it's painful to implement undocumented entities, thanks for the work you have done so far! Yet still I am surprised that it's an application specific thing and I believe there will be also other versions of AutoCAD which use Basepoint entity (they might be saving them in DXF differently, though).

Well, this is probably not the best place to ask, but: insert attribute of INSERT entity gives me nonsense values. They certainly not correspond to the WCS of the drawing, neither the paper coordinates. I haven't defined any UCS, so they must be in some OCS, which seems to be related to the position of the Block definition (first INSERT entity has always insert value (0,0,0)). But coordinates of Block definition's base_point say (0,0,0). So I hoped that by getting the placement of a Basepoint entity in Block definition might do solve my troubles. Try the file I have shared in one of the previous posts with the following code and you can see on your own:

for e in dxf.modelspace():
    if e.dxf.layer == 'VO_body+cisla' and e.dxftype() == 'INSERT':
        print(e.dxf.name)
        print(e.dxf.insert)
        print(e.dxf.extrusion)
        print(e.dxf.xscale)
        print(e.dxf.yscale)
        print(e.dxf.zscale)

I have no idea, how to find out the coordinates of Insert entities in WCS...

[mozman]

Hi!

First: Your Map 3D DXF file opens in AutoCAD LT 2018 and in TrueView 2018 without complaints, so this file is at least DXF compatible, by my definition:

If AutoCAD opens a DXF file without complaints, it is DXF compatible according to the DXF Reference.

The Civil 3D DXF example I got, is not DXF compatible by this definition.

Next: The insertion point of INSERT is really in OCS, I didn't realize this fact until now, but OCS is only relevant if the extrusion vector diverges from (0, 0, 1) the world Z-axis orientation.

I don't know how to interpret the insert coordinates for this special application or how to translate them into georeferenced coordinates.

But there is a GEODATA entity defined in the DXF file and this entity is defined in the DXF reference, so I added read support for this object (dev branch), but don't know how to use this data. Especially the mesh defined in GEODATA seems to be a projection function from source to target coordinates.

from pathlib import Path
from pprint import pprint
import ezdxf

DXF_TEST = Path(r'D:\Source\dxftest')
PRODUCTS = DXF_TEST / 'AutodeskProducts'

dwg = ezdxf.readfile(PRODUCTS/'Map3D_2017.dxf')
msp = dwg.modelspace()

for block_ref in msp.query("INSERT[layer=='VO_body+cisla']"):
    print('\nINSERT')
    pprint(block_ref.dxfattribs())
    # GEODATA handle is stored in the associated BLOCK_RECORD
    block_record = dwg.block_records.get(block_ref.dxf.name)
    try:  # GEODATA handle is stored in an extension dictionary
        xdict = block_record.get_extension_dict()
    except ezdxf.DXFValueError:
        continue
    try:  # GEODATA handle is stored by key ACAD_GEOGRAPHICDATA
        geodata_handle = xdict['ACAD_GEOGRAPHICDATA']
    except ezdxf.DXFKeyError:
        pass
    else:  # but this seems never the case for a block reference
        print('uses GEODATA(#{})'.format(geodata_handle))

# GEODATA entity is only referenced by the model space
try:  # GEODATA handle is stored in an extension dictionary
    # layouts store their associated BLOCK_RECORD
    xdict = msp.block_record.get_extension_dict()
except ezdxf.DXFValueError:
    pass
else:
    try:  # GEODATA handle is stored by key ACAD_GEOGRAPHICDATA
        geodata_handle = xdict['ACAD_GEOGRAPHICDATA']
    except ezdxf.DXFKeyError:
        pass
    else:
        print('Model space uses GEODATA(#{})'.format(geodata_handle))
        geodata = dwg.get_dxf_entity(geodata_handle)
        pprint(geodata.dxfattribs())

# Or you can query all GEODATA entities in the objects section
# because all GEODATA objects know their host BLOCK_RECORD
print()
for geodata in dwg.objects.query('GEODATA'):
    print(geodata)
    block_record = dwg.get_dxf_entity(geodata.dxf.block_record)
    print('GEODATA object used by {}'.format(block_record.dxf.name))
    pprint(geodata.dxfattribs())
    pprint(geodata.coordinate_system_definition())
    pprint(geodata.get_mesh_data(), width=160)

At Last: Look at the query method

Tutorial

Reference

The query method works also with not supported entities like BLOCKBASEPOINTPARAMETER, but there are no DXF attributes defined (which is meant by unsupported), so you have to work with the basic DXF tags.

for base_pointer in dwg.objects.query('BLOCKBASEPOINTPARAMETER'):
    base_pointer.tags  # this is an ExtendedTags() object

[jmacura]

Thanks! This looks promising, I will try this approach soon and will let you know, if it solves my troubles.

[mozman]

Simplified GEODATA access:

geodata = msp.get_geodata()
print('Model space uses: ' + str(geodata))
print(geodata.dxfattribs())
print(geodata.get_coordinate_system_definition())

[jmacura]

So, to be brief: the GEODATA object doesn't seem helpful for me, but I have been able to reconstruct some of the coordinates I was looking for by summing up the basepoint property of BlockReference and the insert property of each Insert (some +/- switching was necessary). Because it only works when an actual Basepoint entity is defined in block, and the data I am working with are quite noisy (some blocks have the Basepoint entity defined, some doesn't, etc.), this all is still kind of a mistery to me...

Anyway, thank you very much for the updates you have made and for all the work on this library!