Closed 107dipan closed 2 years ago
Previous discussion with vespa team : https://github.com/vespa-engine/vespa/issues/22755#issue-1248016484
The crash is likely due to readdir() not returning all entries in a directory when some entries are removed during the directory scan.
We have not been able to reproduce this particular problem. As a possible improvement we have rewritten the code that deletes directories (as part of removing a document type) to using std::filesystem. See https://github.com/vespa-engine/vespa/pull/22851 (in 7.594.7).
Describe the bug The cluster state is going down when removing schemas from the cluster. Currently I have 18 content node cluster. I have only one cluster type defined and all the schemas are part of the same cluster type.
To Reproduce Steps to reproduce the behavior:
Screenshots [2022-05-25 12:15:04.180] WARNING : searchnode proton.transactionlog.server Failed deleting someDocTyoe domain. Exception = IoException: DIRECTORY HAVE CONTENT: rmdir(tls/tls/someDocTyoe, recursive): Failed, errno(39): Directory not empty at rmdir in /builddir/build/BUILD/vespa-7.559.12/vespalib/src/vespa/vespalib/io/fileutil.cpp:580\nBacktrace:\n /opt/vespa/lib64/libvespalib.so(vespalib::IoException::IoException(vespalib::stringref, vespalib::IoException::Type, vespalib::stringref, int)+0x41) [0x7f6e017d96f1]\n /opt/vespa/lib64/libvespalib.so(+0x195d74) [0x7f6e01674d74]\n /opt/vespa/lib64/libsearchlib.so(search::transactionlog::TransLogServer::deleteDomain(FRT_RPCRequest)+0x3bf) [0x7f6e0397295f]\n /opt/vespa/lib64/libsearchlib.so(search::transactionlog::TransLogServer::run()+0x34b) [0x7f6e03973f9b]\n /opt/vespa/lib64/libstaging_vespalib.so(document::Runnable::Run(FastOS_ThreadInterface, void*)+0x4b) [0x7f6e0192aa2b]\n /opt/vespa/lib64/libfastos.so(FastOS_ThreadInterface::Hook()+0x109) [0x7f6e014c72f9]\n /opt/vespa/lib64/libfastos.so(FastOS_ThreadHook+0x9) [0x7f6e014c7499]\n /opt/vespa/lib64/vespa/malloc/libvespamalloc.so(+0x11fab) [0x7f6e04950fab]\n /lib64/libpthread.so.0(+0x7ea5) [0x7f6e01094ea5]\n /lib64/libc.so.6(clone+0x6d) [0x7f6df87a1b0d] [2022-05-25 12:15:04.181] FATAL : searchnode proton.proton.server.proton_disk_layout Failed to remove tls domain someDocTyoe [2022-05-25 12:15:04.182] ERROR : searchnode proton /builddir/build/BUILD/vespa-7.559.12/searchcore/src/vespa/searchcore/proton/server/proton_disk_layout.cpp:90: Abort called. Reason: Failed to remove tls domain [2022-05-25 12:15:04.184] WARNING : searchnode stderr /builddir/build/BUILD/vespa-7.559.12/searchcore/src/vespa/searchcore/proton/server/proton_disk_layout.cpp:90: Abort called. Reason: Failed to remove tls domain
*Environment (please complete the following information):
Vespa version 7.559.12