Error reading large VTK XLM-files ( e.g. vtu-files)

sfladi commented 6 months ago

Describe the bug, what's wrong, and what you expected.

Pyvista raises an error, when reading large VTK XML-files on Windows. In my current use case this is an approximately 7GB .vtu-file holding a large mesh with finite element result data.

 pv.read(path_to_large_vtu_file)

The encountered error message is:

ERROR:root:Cannot read cell data array "E_IntegrationPoints" from PointData in piece 0. The data array in the element may be too short. D:\000_temp.venv\Lib\site-packages\pyvista\core\utilities\fileio.py:203: UserWarning: The VTK reader vtkXMLUnstructuredGridReader in pyvista reader XMLUnstructuredGridReader('D:\000_temp\large_vtu_file.vtu') raised an errorwhile reading the file. "Cannot read cell data array "E_IntegrationPoints" from PointData in piece 0. The data array in the element may be too short." warnings.warn(

This seems to be related to a an issue recently resolved in ParaView, c.f. https://gitlab.kitware.com/paraview/paraview/-/issues/22201 or https://gitlab.kitware.com/paraview/paraview/-/issues/21145. In fact, I encountered literally the identical error, when trying to open my large .vtu-file in ParaView, as i did when using pyvista. Upgrading my ParaView installation to the most recent version resolved the issue in ParaView, indicating for my .vtu-file to be valid.

When trying to reproduce the bug, i would advise to use the .vtu-file provided in the ParaView issue 22201 mentioned above. This will produce a different but seemingly related error message, documented below.

Steps to reproduce the bug.

import pyvista as pv
pv.read("cfd_hlcrm_medium_volume.vtu")  # from https://gitlab.kitware.com/paraview/paraview/-/issues/22201

This produces the following error message:

2024-05-16 14:09:18.480 ( 840.005s) [29CCA2AE3643EC84] vtkXMLDataParser.cxx:640 ERR| vtkXMLDataParser (0000016980440340): Error reading uncompressed binary data header. Read 0 of 8 bytes. ERROR:root:Error reading uncompressed binary data header. Read 0 of 8 bytes. ERROR:root:Cannot read cell offsets from Cells in piece 0 because the "offsets" array is not long enough. D:\000_temp.venv\Lib\site-packages\pyvista\core\utilities\fileio.py:203: UserWarning: The VTK reader vtkXMLUnstructuredGridReader in pyvista reader XMLUnstructuredGridReader('D:\000_temp\cfd_hlcrm_medium_volume.vtu') raised an errorwhile reading the file. "Cannot read cell offsets from Cells in piece 0 because the "offsets" array is not long enough." warnings.warn(

System Information

--------------------------------------------------------------------------------
  Date: Thu May 16 13:59:16 2024 Mitteleuropäische Sommerzeit

                OS : Windows (10 10.0.19045 SP0 Multiprocessor Free)
            CPU(s) : 8
           Machine : AMD64
      Architecture : 64bit
       Environment : Python
        GPU Vendor : NVIDIA Corporation
      GPU Renderer : Quadro P600/PCIe/SSE2
       GPU Version : 4.5.0 NVIDIA 528.89
  MathText Support : False

  Python 3.12.1 | packaged by conda-forge | (main, Dec 23 2023, 07:53:56) [MSC
  v.1937 64 bit (AMD64)]

           pyvista : 0.43.8
               vtk : 9.3.0
             numpy : 1.26.4
        matplotlib : 3.9.0
            scooby : 0.10.0
             pooch : 1.8.1
            pillow : 10.3.0
--------------------------------------------------------------------------------

Screenshots

No response

sfladi commented 6 months ago

As far as I understand this is an issue with VTK and is already patched in VTK https://gitlab.kitware.com/vtk/vtk/-/merge_requests/10629

Im sorry I have little insight into this topic, but is it just a matter of waiting for dependency updates to propagate through to pyvista?

tkoyama010 commented 6 months ago

As far as I understand this is an issue with VTK and is already patched in VTK https://gitlab.kitware.com/vtk/vtk/-/merge_requests/10629

Im sorry I have little insight into this topic, but is it just a matter of waiting for dependency updates to propagate through to pyvista?

You are right. It seems that it has not been added to vtk v9.3.0. We need to wait until the next vtk version and close this issue when you check that it is reflected.

Edit: To make sure it is still not on release. We can see that the patch is not in the dev at release note directory. https://gitlab.kitware.com/vtk/vtk/-/blob/2b2257711e4b4fed98422894939c123004e23b6b/Documentation/release/dev/expat-large-size-fix.md

sfladi commented 6 months ago

I understand, thank you for looking into this!

sfladi commented 4 months ago

This is fixed with the release of VTK v9.3.1: https://discourse.vtk.org/t/vtk-9-3-1-is-out/14141

tkoyama010 commented 4 months ago

This is fixed with the release of VTK v9.3.1: https://discourse.vtk.org/t/vtk-9-3-1-is-out/14141

Awesome! Thanks for sharing this.

pyvista / pyvista