search

asmaloney / libE57Format

Library for reading & writing the E57 file format
Boost Software License 1.0
137 stars 66 forks source link

E57_ERROR_INTERNAL exceptions when trying to read data3d with 0 points #262

Closed nh2 closed 1 year ago

nh2 commented 1 year ago

Hi, thanks for the library.

I observed that when creating a .reader() for a data3d entry that has 0 points, E57_ERROR_INTERNAL gets thrown, likely incorrectly.

The 2 throw locations are these (linking code from v2.2.0 but that code seems the same with master):

Case 1:

https://github.com/asmaloney/libE57Format/blob/65c0d2380711ddf3c7ec8a7fc9f03d5b453b9d7c/src/Packet.cpp#L97-L101

Case 2:

https://github.com/asmaloney/libE57Format/blob/65c0d2380711ddf3c7ec8a7fc9f03d5b453b9d7c/src/CompressedVectorReaderImpl.cpp#L103-L110

With my real E57 file I encountered the issue on, the first one is thrown, and with a mini synthetic empy E57 point cloud I tried to craft with help of CloudCompare and Meshlab, the second one is thrown.

For the second one, here is a file to repro: empty-cloud.e57.zip

I believe that for such a 0-point point cloud, creating a .reader() from it should work, simply reading 0 points, and not give E57_ERROR_INTERNAL.

asmaloney commented 1 year ago

Hi Niklas.

I'm trying to reproduce this.

Are you getting this with the latest release or an older version?

When you are talking about .reader() for a data3d what functions are you calling specifically? i.e. is it failing when creating the e57::Data3DPointsFloat, when calling SetUpData3DPointsData, or when calling .read()?

(Do you mind if I add this E57 file to the test suite?)

asmaloney commented 1 year ago

I was able to reproduce the second one (after removing a specific check for 1 or more points in E57SimpleData.cpp). I don't currently see a path to reproducing the first one.

Not quite sure how to fix it at the moment because the SetUpData3DPointsData function creates and returns the CompressedVectorReader (which is what's throwing), and the function signature does not allow for an error to be returned. I guess I could at least throw from there with a clearer error message and how to solve it?

The simple solution is to not call that if you have 0 points :-) You do have that information from the Data3D header before calling into SetUpData3DPointsData.

Here's a test that outlines it:

TEST( SimpleReaderData, NoPoints )
{
   e57::Reader *reader = nullptr;

   E57_ASSERT_NO_THROW( reader = new e57::Reader( TestData::Path() + "/self/no-points.e57", {} ) );

   ASSERT_TRUE( reader->IsOpen() );
   EXPECT_EQ( reader->GetImage2DCount(), 0 );
   EXPECT_EQ( reader->GetData3DCount(), 1 );

   e57::E57Root fileHeader;
   ASSERT_TRUE( reader->GetE57Root( fileHeader ) );

   CheckFileHeader( fileHeader );
   EXPECT_EQ( fileHeader.guid, "{EC1A0DE4-F76F-44CE-E527-789EEB818347}" );

   e57::Data3D data3DHeader;
   ASSERT_TRUE( reader->ReadData3D( 0, data3DHeader ) );

   ASSERT_EQ( data3DHeader.pointCount, 0 );

   const uint64_t cNumPoints = data3DHeader.pointCount;

   if ( cNumPoints > 0 )
   {
      e57::Data3DPointsFloat pointsData( data3DHeader );

      auto vectorReader = reader->SetUpData3DPointsData( 0, cNumPoints, pointsData );

      const uint64_t cNumRead = vectorReader.read();

      vectorReader.close();

      EXPECT_EQ( cNumRead, cNumPoints );
   }

   delete reader;
}
nh2 commented 1 year ago

Are you getting this with the latest release or an older version?

I'm testing with v2.2.0.

When you are talking about .reader() for a data3d what functions are you calling specifically?

I'm calling from pye57 reader = points.reader(dest_buffers), where points is the field in a data3d vector entry.

I think this calls CompressedVectorNode::reader()

Here are the GDB stack traces for Case 1 from the issue description:

(gdb) bt
#0  0x00007ffff7eafff8 in _Unwind_RaiseException () from /nix/store/843dqq10jdkalr2yazaz6drx334visrb-gcc-12.2.0-lib/lib/libgcc_s.so.1
#1  0x00007fff8b8b525a in __cxa_throw () from /nix/store/7c0yrczwxn58f9gk9ipawdh607vh067k-gcc-12.2.0-lib/lib/libstdc++.so.6
#2  0x00007fff7cec7b1a in e57::PacketReadCache::lock(unsigned long, char*&) [clone .cold] ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#3  0x00007fff7ceeacc4 in e57::CompressedVectorReaderImpl::CompressedVectorReaderImpl(std::shared_ptr<e57::CompressedVectorNodeImpl>, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >&) () from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#4  0x00007fff7cee8231 in e57::CompressedVectorNodeImpl::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >, bool) ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#5  0x00007fff7cefb66d in e57::CompressedVectorNode::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool) ()

My debug print I added prints e.errorCode() 11 e.context() packetLogicalOffset=0.

I'll try to produce the one for Case 2 in a moment as well.

(Do you mind if I add this E57 file to the test suite?)

You can definitely use it. The XML looks a bit odd when viewed in less, but that might be because of embedded binary data?

Meshlab crashed in the middle of writing it (I filed https://github.com/cnr-isti-vclab/meshlab/issues/1433 for that) but it seems to still have written it.

asmaloney commented 1 year ago

I'm testing with v2.2.0.

Ah. The "simple reader" stuff was significantly revamped for version 3, so it's possible nothing I do here will apply to 2.x.

The XML looks a bit odd...

The XML is OK. The E57 format stores binary data followed by XML and the whole file includes CRC checksums on each 1020 bytes, so that's why you get problems with less. Here's what it looks like when read properly:

<?xml version="1.0"?>
<e57Root type="Structure" xmlns="http://www.astm.org/COMMIT/E57/2010-e57-v1.0" xmlns:nor="http://www.libe57.org/E57_NOR_surface_normals.txt">
    <formatName type="String"><![CDATA[ASTM E57 3D Imaging Data File]]></formatName>
    <guid type="String"><![CDATA[{EC1A0DE4-F76F-44CE-E527-789EEB818347}]]></guid>
    <versionMajor type="Integer">1</versionMajor>
    <versionMinor type="Integer" />
    <e57LibraryVersion type="String"><![CDATA[E57Format-2.2.0-x86_64-linux-gcc122]]></e57LibraryVersion>
    <data3D type="Vector" allowHeterogeneousChildren="1">
        <vectorChild type="Structure">
            <guid type="String"><![CDATA[{50c4acef-3526-4a44-b6b8-2d32c8457814}]]></guid>
            <pose type="Structure">
                <rotation type="Structure">
                    <w type="Float">1.00000000000000000e+00</w>
                    <x type="Float" />
                    <y type="Float" />
                    <z type="Float" />
                </rotation>
                <translation type="Structure">
                    <x type="Float" />
                    <y type="Float" />
                    <z type="Float" />
                </translation>
            </pose>
            <points type="CompressedVector" fileOffset="0" recordCount="0">
                <prototype type="Structure">
                    <cartesianX type="Float" precision="single" />
                    <cartesianY type="Float" precision="single" />
                    <cartesianZ type="Float" precision="single" />
                    <nor:normalX type="Float" precision="single" minimum="-1.0000000e+00" maximum="1.0000000e+00" />
                    <nor:normalY type="Float" precision="single" minimum="-1.0000000e+00" maximum="1.0000000e+00" />
                    <nor:normalZ type="Float" precision="single" minimum="-1.0000000e+00" maximum="1.0000000e+00" />
                </prototype>
                <codecs type="Vector" allowHeterogeneousChildren="1" />
            </points>
        </vectorChild>
    </data3D>
    <images2D type="Vector" allowHeterogeneousChildren="1" />
</e57Root>

(Edit: Oh - and I'm afraid that stack trace doesn't really give any useful information about where it's coming from. I don't know the Python stuff, so it's possible it needs to add the check for 0 points before trying to read like I have in the test I posted above?)

nh2 commented 1 year ago

I'll try to produce the one for Case 2 in a moment as well.

Here we go, for the repro file attached above:

(gdb) bt
#0  0x00007ffff7eafff8 in _Unwind_RaiseException () from /nix/store/843dqq10jdkalr2yazaz6drx334visrb-gcc-12.2.0-lib/lib/libgcc_s.so.1
#1  0x00007fff8b8b525a in __cxa_throw () from /nix/store/7c0yrczwxn58f9gk9ipawdh607vh067k-gcc-12.2.0-lib/lib/libstdc++.so.6
#2  0x00007fff7ceb0ba7 in e57::CompressedVectorReaderImpl::CompressedVectorReaderImpl(std::shared_ptr<e57::CompressedVectorNodeImpl>, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >&) [clone .cold] () from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#3  0x00007fff7cee8341 in e57::CompressedVectorNodeImpl::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >, bool) ()
   from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#4  0x00007fff7cefb77d in e57::CompressedVectorNode::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool) ()

My debug print prints

e.errorCode() 11 e.context() imageFileName=/path/to/empty-cloud.e57 cvPathName=/data3D/0/points
nh2 commented 1 year ago

I don't currently see a path to reproducing the first one.

@asmaloney I have a real-world E57 file I have that causes it, which is probably the fastest way to repro Case 1.

But it's 38 GB and I can't post it publicly. May I share it with you privately, and would you be up for sending me an email agreeing that you can use it only for the purpose of investigating this bug and to not distribute it further?

I also tried to downsample it with CloudCompare for easier sharing in the hope that this would preserve the 0-points data3d entry (of which there's only one in the file), but CloudCompare crashes on it with


[E57] Error: bad SourceDestBuffer (E57_ERROR_BAD_BUFFER)
    context: pathName=cartesianX
An error occurred while loading 'MainHouse': the third-party library in charge of saving/loading the file has thrown an exception
```, 

CloudCompare also uses `libE57Format` according to https://github.com/CloudCompare/CloudCompare/blob/e243efbb3e6ae65a96e42da7620fd46771f90fbe/CHANGELOG.md?plain=1#L972-L974

However, here the error is different, and this downsampling problem might be CloudCompare's own problem.
asmaloney commented 1 year ago

So it sounds like there are a couple of possible issues here:

  1. you're using a Python wrapper
  2. it's using an old version of the code

This library uses exceptions everywhere (certainly not my choice!), not error return. So the "right" way to use it is to wrap calls to the API with try/catch. This would have to be done in the Python wrapper.

Two solutions for the Python wrapper code:

  1. Add try/catch to handle exceptions
  2. Check the number of points before trying to read
asmaloney commented 1 year ago

But it's 38 GB

😆 Yikes. I don't think I can deal with that right now.

Can you post a full stacktrace? The frames in the one you posted don't really tell us much other than were the exception was thrown.

What software produced the file? (Hopefully the e57LibraryVersion in the XML in the file should tell us.)

asmaloney commented 1 year ago

(I sent an email to the address in your profile to followup.)

nh2 commented 1 year ago

Two solutions for the Python wrapper code:

Some clarifications:

Can you post a full stacktrace? The frames in the one you posted don't really tell us much other than were the exception was thrown.

Here are the full stack traces for the two cases, but I don't think it's too helpful because the immediate frame on top of what I posted is already pybind11, Python's C++ binding:

Case 1:

(gdb) bt
#0  0x00007ffff7eafff8 in _Unwind_RaiseException () from /nix/store/843dqq10jdkalr2yazaz6drx334visrb-gcc-12.2.0-lib/lib/libgcc_s.so.1
#1  0x00007fff8b8b525a in __cxa_throw () from /nix/store/7c0yrczwxn58f9gk9ipawdh607vh067k-gcc-12.2.0-lib/lib/libstdc++.so.6
#2  0x00007fff7cec7b1a in e57::PacketReadCache::lock(unsigned long, char*&) [clone .cold] ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#3  0x00007fff7ceeacc4 in e57::CompressedVectorReaderImpl::CompressedVectorReaderImpl(std::shared_ptr<e57::CompressedVectorNodeImpl>, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >&) () from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#4  0x00007fff7cee8231 in e57::CompressedVectorNodeImpl::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >, bool) ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#5  0x00007fff7cefb66d in e57::CompressedVectorNode::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool) ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#6  0x00007fff7cfa2907 in pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<e57::CompressedVectorReader, e57::CompressedVectorNode, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v>(e57::CompressedVectorReader (e57::CompressedVectorNode::*)(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&)::{lambda(e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool)#1}, e57::CompressedVectorReader, e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v>(pybind11::cpp_function::initialize<e57::CompressedVectorReader, e57::CompressedVectorNode, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v>(e57::CompressedVectorReader (e57::CompressedVectorNode::*)(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&)::{lambda(e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool)#1}&&, e57::CompressedVectorReader (*)(e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#7  0x00007fff7cf93979 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) ()
   from /nix/store/jaswcrnsbcgdkm7b5gybzicdlrr7kl2z-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#8  0x00007ffff7c65103 in cfunction_call () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#9  0x00007ffff7b1f930 in _PyObject_MakeTpCall () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#10 0x00007ffff7b28592 in method_vectorcall () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#11 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#12 0x00007ffff7a62ae3 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#13 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#14 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#15 0x00007ffff7a62ae3 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#16 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#17 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#18 0x00007ffff7a63221 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#19 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#20 0x00007ffff7aad76c in PyVectorcall_Call () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#21 0x00007ffff7a62c7a in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#22 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#23 0x00007ffff7a62c7a in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#24 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#25 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#26 0x00007ffff7a61fc5 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#27 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#28 0x00007ffff7a62c7a in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#29 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#30 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#31 0x00007ffff7a61fc5 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#32 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#33 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#34 0x00007ffff7a61fc5 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#35 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#36 0x00007ffff7b285f4 in method_vectorcall () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#37 0x00007ffff7c6b258 in thread_run () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#38 0x00007ffff7a749c2 in pythread_wrapper () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
--Type <RET> for more, q to quit, c to continue without paging--
#39 0x00007ffff789fe24 in start_thread () from /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libc.so.6
#40 0x00007ffff79219b0 in clone3 () from /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libc.so.6

Case 2:

(gdb) bt
#0  0x00007ffff7eafff8 in _Unwind_RaiseException () from /nix/store/843dqq10jdkalr2yazaz6drx334visrb-gcc-12.2.0-lib/lib/libgcc_s.so.1
#1  0x00007fff8b8b525a in __cxa_throw () from /nix/store/7c0yrczwxn58f9gk9ipawdh607vh067k-gcc-12.2.0-lib/lib/libstdc++.so.6
#2  0x00007fff7ceb0ba7 in e57::CompressedVectorReaderImpl::CompressedVectorReaderImpl(std::shared_ptr<e57::CompressedVectorNodeImpl>, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >&) [clone .cold] () from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#3  0x00007fff7cee8341 in e57::CompressedVectorNodeImpl::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> >, bool) ()
   from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#4  0x00007fff7cefb77d in e57::CompressedVectorNode::reader(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool) ()
   from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#5  0x00007fff7cfa2b47 in pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<e57::CompressedVectorReader, e57::CompressedVectorNode, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v>(e57::CompressedVectorReader (e57::CompressedVectorNode::*)(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&)::{lambda(e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool)#1}, e57::CompressedVectorReader, e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v>(pybind11::cpp_function::initialize<e57::CompressedVectorReader, e57::CompressedVectorNode, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::arg_v>(e57::CompressedVectorReader (e57::CompressedVectorNode::*)(std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&)::{lambda(e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool)#1}&&, e57::CompressedVectorReader (*)(e57::CompressedVectorNode*, std::vector<e57::SourceDestBuffer, std::allocator<e57::SourceDestBuffer> > const&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) ()
   from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#6  0x00007fff7cf93bb9 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) ()
   from /nix/store/107b92i3jp2p6rvzavs57gyrvbzi5hiv-python3-3.10.12-env/lib/python3.10/site-packages/pye57/libe57.cpython-310-x86_64-linux-gnu.so
#7  0x00007ffff7c65103 in cfunction_call () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#8  0x00007ffff7b1f930 in _PyObject_MakeTpCall () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#9  0x00007ffff7b28592 in method_vectorcall () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#10 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#11 0x00007ffff7a62ae3 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#12 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#13 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#14 0x00007ffff7a62ae3 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#15 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#16 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#17 0x00007ffff7a63221 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#18 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#19 0x00007ffff7aad76c in PyVectorcall_Call () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#20 0x00007ffff7a62c7a in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#21 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#22 0x00007ffff7a62c7a in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#23 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#24 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#25 0x00007ffff7a61fc5 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#26 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#27 0x00007ffff7a62c7a in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#28 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#29 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#30 0x00007ffff7a61fc5 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#31 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#32 0x00007ffff7a5ec78 in call_function () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#33 0x00007ffff7a61fc5 in _PyEval_EvalFrameDefault () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#34 0x00007ffff7ba7206 in _PyEval_Vector () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#35 0x00007ffff7b285f4 in method_vectorcall () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#36 0x00007ffff7c6b258 in thread_run () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#37 0x00007ffff7a749c2 in pythread_wrapper () from /nix/store/bc45k1n0pkrdkr3xa6w84w1xhkl1kkyp-python3-3.10.12/lib/libpython3.10.so.1.0
#38 0x00007ffff789fe24 in start_thread () from /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libc.so.6
#39 0x00007ffff79219b0 in clone3 () from /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libc.so.6

What software produced the file? (Hopefully the e57LibraryVersion in the XML in the file should tell us.)

The Faro SCENE proprietary laser scan processing software (e57LibraryVersion: SCENE_E57RefImpl-1.1.312).

asmaloney commented 1 year ago

OK - thanks. Yes those stack traces are not useful 😄

It still seems worthwhile to figure out whether this error should be raised. From my understanding, E57_ERROR_INTERNAL should not surface if the caller does everything correctly, which it seems to do in my case.

One of the design principles from the original lib was that all objects should be in a reasonable state and that no pointers are used for the objects in the API. That means no returning pointers & checking for nullptr on error for example.

So the problem we have in this case is that SetUpData3DPointsData returns a CompressedVectorReader, but a CompressedVectorReader is only valid when there are > 0 points. So we are kind of stuck because of the design.

One solution might be to change the function signature for SetUpData3DPointsData to do something different, but again the rest of the library doesn't return errors; it uses exceptions. Another might be to modify CompressedVectorReader to allow for 0 points. This is probably a better solution and I will look into this (though I'm hesitant to touch this code 😆 ).

For now, I agree about returning E57_ERROR_INTERNAL. I will change it to catch this issue earlier in SetUpData3DPointsData and throw a more useful error.

The Faro SCENE proprietary laser scan processing software (e57LibraryVersion: SCENE_E57RefImpl-1.1.312).

Ah interesting - that's a (possibly modified) earlier version of the library that E5Format was derived from (E57RefImpl 1.1.332). It will likely suffer from the some of the same things as this library. Even though it was called a reference library it didn't follow the standard in some cases...

nh2 commented 1 year ago

Copying here one part from our email exchange:

@asmaloney found that my SCENE-written file I provided certainly does not write the 0-point cloud spec-compliantly, because it does not write a data index, but the spec requires that:

The original E57RefImpl (and libE57Format because it comes from E57RefImpl) doesn't create/store any indices. It's required by the standard.

E57 Standard: 9.3.5:

A CompressedVector shall contain at least one index packet and at least one data packet.

This means that all E57-writing programs using E57RefImpl will likely write the 0-points pointcloud wrong.

We haven't figured out what the way to do it correctly is; quoting @asmaloney:

It's unclear what one is supposed to write when you have 0 points - both in the header and in the binary data. It's not specified in the standard. With E57 Standard: 9.3.5 (see next comment), an index and data packet are also required. Maybe it's possible to write each of those with zero entries, but now it's getting ridiculous...

I agree with that, it's definitely weird for the spec to require these to exist.

asmaloney commented 1 year ago

OK - I'm going to call this "done".

  1. Trying to read a Data3D with an ill-formed header will now give an ErrorData3DReadInvalidZeroRecords exception instead of ErrorInternal. (#264) It looks like this:

    trying to read an invalid Data3D with zero records - check for zero records before trying to read this Data3D section (ErrorInvalidZeroRecordsData3D): fileOffset cannot be 0; cvPathName=/data3D/0/points imageFileName=ZeroPointsInvalid.e57

  2. Properly write a Data3D section with 0 points by including an empty data packet. (#266)
  3. Properly read a Data3D section with 0 points if it includes an empty data packet. (#267)

Note that this doesn't address the missing (required) index packet. That is a failure to comply with the standard, but does not throw any exceptions.

nh2 commented 1 year ago

@asmaloney Thanks!