Segmentation fault while building multiple laz files

[Timmarh]

I'm trying to entwine build a large amount (1021) of laz files. I'm using a config.json:

{
        "input": "/path/to/colored/laz/**",
        "output": "/output/path",
        "reprojection": {
                "in": "EPSG:28992",
                "out": "EPSG:28992",
                "hammer": true
            },
        "dataType":"zstandard",
        "threads":8
}

All the files are found and listed, but after the files are listed (up until 1021/1021), the building stops with: segmentation fault (core dumped)

this is the output of gdb:

[Thread 0x7fffec0b9700 (LWP 31157) exited]
[Thread 0x7fffef0bf700 (LWP 31150) exited]

Thread 6 "entwine" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffed8bc700 (LWP 31154)]
__memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:326
326     ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
(gdb) trace
Tracepoint 1 at 0x7ffff6de2121: file ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S, line 326.
(gdb) bt
#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:326
#1  0x00007ffff6f9f302 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace(unsigned long, unsigned long, char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007ffff6c39692 in pdal::Utils::escapeJSON(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /usr/local/lib/libpdal_util.so.8
#3  0x00007ffff7aac6c5 in pdal::MetadataNode::jsonValue[abi:cxx11]() const () from /usr/local/lib/libpdal_base.so.8
#4  0x00007ffff7aaa812 in pdal::(anonymous namespace)::toJSON(pdal::MetadataNode const&, std::ostream&, int) () from /usr/local/lib/libpdal_base.so.8
#5  0x00007ffff7aaa20a in pdal::(anonymous namespace)::subnodesToJSON(pdal::MetadataNode const&, std::ostream&, int) ()
   from /usr/local/lib/libpdal_base.so.8
#6  0x00007ffff7aaaa47 in pdal::(anonymous namespace)::toJSON(pdal::MetadataNode const&, std::ostream&, int) () from /usr/local/lib/libpdal_base.so.8
#7  0x00007ffff7aaa20a in pdal::(anonymous namespace)::subnodesToJSON(pdal::MetadataNode const&, std::ostream&, int) ()
   from /usr/local/lib/libpdal_base.so.8
#8  0x00007ffff7aaac12 in pdal::Utils::toJSON(pdal::MetadataNode const&, std::ostream&) () from /usr/local/lib/libpdal_base.so.8
#9  0x00007ffff7aaab3b in pdal::Utils::toJSON[abi:cxx11](pdal::MetadataNode const&) () from /usr/local/lib/libpdal_base.so.8
#10 0x00007ffff7f5f1a2 in entwine::ScanInfo::ScanInfo(pdal::Stage&, pdal::QuickInfo const&) () from /usr/local/lib/libentwine.so.2
#11 0x00007ffff7f5f6db in entwine::ScanInfo::create(pdal::Stage&) () from /usr/local/lib/libentwine.so.2
#12 0x00007ffff7f65607 in entwine::Executor::preview(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer>, bool) const () from /usr/local/lib/libentwine.so.2
#13 0x00007ffff7f366f8 in entwine::Scan::add(entwine::FileInfo&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) ()
   from /usr/local/lib/libentwine.so.2
#14 0x00007ffff7f38657 in entwine::Scan::add(entwine::FileInfo&)::{lambda()#1}::operator()() const [clone .isra.635] () from /usr/local/lib/libentwine.so.2
#15 0x00007ffff7e963ec in std::thread::_State_impl<std::thread::_Invoker<std::tuple<entwine::Pool::go()::{lambda()#1}> > >::_M_run() ()
   from /usr/local/lib/libentwine.so.2
#16 0x00007ffff6f30630 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#17 0x00007ffff704b182 in start_thread (arg=<optimized out>) at pthread_create.c:486
#18 0x00007ffff6d76b1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

It sounds like there's a problem with writing a json (something in ept-sources?). The files are organized in separate folders. As a temporary fix I wrote a loop that builds each directory individually. This seems to work. Probably one of the files or directories contains an error somehow. Any idea why this error occurs and how to solve it?

[Timmarh]

I narrowed the problem down to one file. Seems like the file is corrupted. Cloud Compare and pdal also give errors when trying to open the file.

[connormanning]

Interesting, thanks for the update. By the way, if the purpose of your no-op reprojection filter is simply to set an output SRS, you can use the --srs CLI flag or srs configuration option instead. This option is documented in the entwine build CLI help but I just realized that it was not in the configuration docs - I've just added it here.

[Timmarh]

Perfect, thanks Connor!