mldbai / mldb

MLDB is the Machine Learning Database
http://mldb.ai
Apache License 2.0
665 stars 102 forks source link

Race between procedure destruction and procedure application #946

Open jeremybarnes opened 3 years ago

jeremybarnes commented 3 years ago
MLDBFB-724_classifier_exp_segfault_test.py FAILED
. virtualenv/bin/activate; PYTHONPATH=build/arm64/bin build/arm64/bin/mldb_runner -h localhost -p '11700-12700'  --run-script mldb/testing/MLDBFB-724_classifier_exp_segfault_test.py --mute-final-output --config-path mldb/container_files/mldb.conf --watchdog-timeout=120 
reading configuration from file: 'mldb/container_files/mldb.conf'

MLDB ready

creating SYMLINK /var/folders/ch/fv_rwtr97j55bhh8lz6p2dl80000gp/T/REVv50/main.py -> /Users/jeremy/projects/mldb/mldb/testing/MLDBFB-724_classifier_exp_segfault_test.py
loading from: /var/folders/ch/fv_rwtr97j55bhh8lz6p2dl80000gp/T/REVv50/main.py
2021-08-30 10:00:41.042 script runner plugin SELECT {_inception.* EXCLUDING(image_url)} AS features,
                 ds.cei AS label
                 FROM _inception
                 INNER JOIN ds ON _inception.image_url=ds.image_url
2021-08-30 10:00:41.045 script runner plugin [
    [
        "_rowName"
    ]
]
2021-08-30 10:00:41.059 script runner plugin test_it (__main__.Mldbfb724ClassifierExpSegfaultTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.152s

OK

exception in accept: Operation canceled
exception in accept: Operation canceled
ServicePeer [2021-08-30T10:00:41.059-4:00] warning WARNING: peer mldb lost its own entry in discovery.  Letting it come back
peer mldb connection to mldb changed state to 3
peer mldb connection to mldb changed state to 3
=================================================================
==67440==ERROR: AddressSanitizer: heap-use-after-free on address 0x00011834aa08 at pc 0x0001053dd2cc bp 0x00016c5030c0 sp 0x00016c5030b8
READ of size 8 at 0x00011834aa08 thread T34
    #0 0x1053dd2c8 in MLDB::SelectExpressionDescription::printJsonTyped(MLDB::SelectExpression const*, MLDB::JsonPrintingContext&) const sql_expression.cc:3616
    #1 0x1081408cc in MLDB::StructureDescriptionBase::printJson(void const*, MLDB::JsonPrintingContext&) const value_description.cc:583
    #2 0x1053f7150 in MLDB::InputQueryDescription::printJsonTyped(MLDB::InputQuery const*, MLDB::JsonPrintingContext&) const sql_expression.cc:4429
    #3 0x1081408cc in MLDB::StructureDescriptionBase::printJson(void const*, MLDB::JsonPrintingContext&) const value_description.cc:583
    #4 0x10d7fab20 in Json::Value MLDB::jsonEncode<MLDB::ExperimentProcedureConfig>(MLDB::ExperimentProcedureConfig const&) value_description.h:650
    #5 0x10d7cb33c in MLDB::ExperimentProcedureConfig MLDB::Procedure::applyRunConfOverProcConf<MLDB::ExperimentProcedureConfig>(MLDB::ExperimentProcedureConfig const&, MLDB::ProcedureRunConfig const&) const procedure.h:161
    #6 0x10d7c0b54 in MLDB::ExperimentProcedure::run(MLDB::ProcedureRunConfig const&, std::__1::function<bool (Json::Value const&)> const&) const experiment_procedure.cc:308
    #7 0x106c74730 in MLDB::ProcedureRun::ProcedureRun(MLDB::Procedure*, MLDB::ProcedureRunConfig, std::__1::function<bool (Json::Value const&)> const&) procedure.cc:85
    #8 0x10721bfe4 in MLDB::ProcedureRunCollection::construct(MLDB::ProcedureRunConfig, std::__1::function<bool (Json::Value const&)> const&) const procedure_run_collection.cc:262
    #9 0x1071f3e44 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::constructCancellable(MLDB::ProcedureRunConfig, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>) const rest_collection_impl.h:1415
    #10 0x1072e0f6c in std::__1::shared_ptr<MLDB::ProcedureRun> std::__1::__invoke_void_return_wrapper<std::__1::shared_ptr<MLDB::ProcedureRun> >::__call<std::__1::__bind<std::__1::shared_ptr<MLDB::ProcedureRun> (MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::*)(MLDB::ProcedureRunConfig, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>) const, MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>*, MLDB::ProcedureRunConfig, std::__1::placeholders::__ph<1> const&, std::__1::placeholders::__ph<2> const&>&, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool> >(std::__1::__bind<std::__1::shared_ptr<MLDB::ProcedureRun> (MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::*)(MLDB::ProcedureRunConfig, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>) const, MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>*, MLDB::ProcedureRunConfig, std::__1::placeholders::__ph<1> const&, std::__1::placeholders::__ph<2> const&>&, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>&&) __functional_base:317
    #11 0x107258400 in MLDB::RestCollection<MLDB::Utf8String, MLDB::ProcedureRun>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::ProcedureRun> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::ProcedureRun>)> const&, bool, MLDB::Any)::'lambda'()::operator()() const rest_collection_impl.h:586
    #12 0x107257a58 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, MLDB::RestCollection<MLDB::Utf8String, MLDB::ProcedureRun>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::ProcedureRun> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::ProcedureRun>)> const&, bool, MLDB::Any)::'lambda'()> >(void*) thread:291
    #13 0x18e887874 in _pthread_start+0x13c (libsystem_pthread.dylib:arm64e+0x7874)
    #14 0x18e8825dc in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x25dc)

0x00011834aa08 is located 72 bytes inside of 360-byte region [0x00011834a9c0,0x00011834ab28)
freed by thread T0 here:
    #0 0x109b18f64 in wrap__ZdlPv+0x74 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x4cf64)
    #1 0x10d7d9fe8 in MLDB::ExperimentProcedureConfig::~ExperimentProcedureConfig() experiment_procedure.h:60
    #2 0x10d7d410c in MLDB::ExperimentProcedure::~ExperimentProcedure() experiment_procedure.h:133
    #3 0x1089dbcf8 in std::__1::__tree<std::__1::__value_type<MLDB::Utf8String, MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::Entry>, std::__1::__map_value_compare<MLDB::Utf8String, std::__1::__value_type<MLDB::Utf8String, MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::Entry>, std::__1::less<MLDB::Utf8String>, true>, std::__1::allocator<std::__1::__value_type<MLDB::Utf8String, MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::Entry> > >::destroy(std::__1::__tree_node<std::__1::__value_type<MLDB::Utf8String, MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::Entry>, void*>*) __tree:1836
    #4 0x1089424d4 in MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::clear() rest_collection_impl.h:195
    #5 0x1049fae6c in MLDB::MldbServer::shutdown() mldb_server.cc:436
    #6 0x1049fa690 in MLDB::MldbServer::~MldbServer() mldb_server.cc:114
    #7 0x1049630b4 in main mldb_runner.cc:429
    #8 0x18e8a544c in start+0x0 (libdyld.dylib:arm64e+0x1844c)

previously allocated by thread T33 here:
    #0 0x109b18b4c in wrap__Znwm+0x74 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x4cb4c)
    #1 0x1053f6284 in MLDB::InputQueryDescription::parseJsonTyped(MLDB::InputQuery*, MLDB::JsonParsingContext&) const sql_expression.cc:4409
    #2 0x10814cfe4 in std::__1::__function::__func<MLDB::StructureDescriptionBase::parseJson(void*, MLDB::JsonParsingContext&) const::$_0, std::__1::allocator<MLDB::StructureDescriptionBase::parseJson(void*, MLDB::JsonParsingContext&) const::$_0>, void ()>::operator()() functional:1727
    #3 0x1081a9d08 in MLDB::StructuredJsonParsingContext::forEachMember(std::__1::function<void ()> const&) json_parsing.cc:1894
    #4 0x10813fcd4 in MLDB::StructureDescriptionBase::parseJson(void*, MLDB::JsonParsingContext&) const value_description.cc:555
    #5 0x10d7bf6e8 in MLDB::ExperimentProcedureConfig MLDB::Any::convert<MLDB::ExperimentProcedureConfig>(MLDB::ValueDescription const&) const any_impl.h:68
    #6 0x10d7bf180 in MLDB::ExperimentProcedure::ExperimentProcedure(MLDB::MldbEngine*, MLDB::PolyConfig, std::__1::function<bool (Json::Value const&)> const&) experiment_procedure.cc:289
    #7 0x10d7fdea4 in std::__1::shared_ptr<MLDB::EntityType<MLDB::Procedure> > MLDB::registerProcedureType<MLDB::ExperimentProcedure, MLDB::ExperimentProcedureConfig>(MLDB::Package const&, MLDB::Utf8String const&, MLDB::Utf8String const&, std::__1::function<MLDB::RestRequestMatchResult (MLDB::RestDirectory*, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&)>, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >)::'lambda'(MLDB::RestDirectory*, MLDB::PolyConfig, std::__1::function<bool (Json::Value)> const&)::operator()(MLDB::RestDirectory*, MLDB::PolyConfig, std::__1::function<bool (Json::Value)> const&) const procedure.h:241
    #8 0x10d7fd6e0 in MLDB::Procedure* std::__1::__invoke_void_return_wrapper<MLDB::Procedure*>::__call<std::__1::shared_ptr<MLDB::EntityType<MLDB::Procedure> > MLDB::registerProcedureType<MLDB::ExperimentProcedure, MLDB::ExperimentProcedureConfig>(MLDB::Package const&, MLDB::Utf8String const&, MLDB::Utf8String const&, std::__1::function<MLDB::RestRequestMatchResult (MLDB::RestDirectory*, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&)>, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >)::'lambda'(MLDB::RestDirectory*, MLDB::PolyConfig, std::__1::function<bool (Json::Value)> const&)&, MLDB::RestDirectory*, MLDB::PolyConfig, std::__1::function<bool (Json::Value)> const&>(MLDB::ExperimentProcedure&&...) __functional_base:317
    #9 0x10719dff8 in MLDB::PolyCollection<MLDB::Procedure>::doConstruct(MLDB::RestDirectory*, MLDB::PolyConfig, std::__1::function<bool (Json::Value const&)> const&) poly_collection_impl.h:365
    #10 0x10719ca38 in MLDB::PolyCollection<MLDB::Procedure>::construct(MLDB::PolyConfig, std::__1::function<bool (Json::Value const&)> const&) const poly_collection_impl.h:114
    #11 0x10897b8d4 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::constructCancellable(MLDB::PolyConfig, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>) const rest_collection_impl.h:1415
    #12 0x108a22c48 in std::__1::shared_ptr<MLDB::PolyEntity> std::__1::__invoke_void_return_wrapper<std::__1::shared_ptr<MLDB::PolyEntity> >::__call<std::__1::__bind<std::__1::shared_ptr<MLDB::PolyEntity> (MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::*)(MLDB::PolyConfig, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>) const, MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>*, MLDB::PolyConfig, std::__1::placeholders::__ph<1> const&, std::__1::placeholders::__ph<2> const&>&, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool> >(std::__1::__bind<std::__1::shared_ptr<MLDB::PolyEntity> (MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::*)(MLDB::PolyConfig, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>) const, MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>*, MLDB::PolyConfig, std::__1::placeholders::__ph<1> const&, std::__1::placeholders::__ph<2> const&>&, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>&&) __functional_base:317
    #13 0x1089f2310 in MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::PolyEntity> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::PolyEntity>)> const&, bool, MLDB::Any)::'lambda'()::operator()() const rest_collection_impl.h:586
    #14 0x1089f1968 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::PolyEntity> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::PolyEntity>)> const&, bool, MLDB::Any)::'lambda'()> >(void*) thread:291
    #15 0x18e887874 in _pthread_start+0x13c (libsystem_pthread.dylib:arm64e+0x7874)
    #16 0x18e8825dc in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x25dc)

Thread T34 created by T7 here:
    #0 0x109b055b0 in wrap_pthread_create+0x54 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x395b0)
    #1 0x107257650 in std::__1::thread::thread<MLDB::RestCollection<MLDB::Utf8String, MLDB::ProcedureRun>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::ProcedureRun> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::ProcedureRun>)> const&, bool, MLDB::Any)::'lambda'()&, void>(MLDB::Utf8String&&, MLDB::ProcedureRun&&...) thread:307
    #2 0x1071c6a78 in MLDB::RestCollection<MLDB::Utf8String, MLDB::ProcedureRun>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::ProcedureRun> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::ProcedureRun>)> const&, bool, MLDB::Any) rest_collection_impl.h:621
    #3 0x1071e3fc0 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::handlePutItl(MLDB::Utf8String, MLDB::ProcedureRunConfig, std::__1::function<void (std::__1::shared_ptr<MLDB::ProcedureRun>)> const&, bool) rest_collection_impl.h:1767
    #4 0x1071e3324 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::handlePut(MLDB::Utf8String, MLDB::ProcedureRunConfig, bool) rest_collection_impl.h:1795
    #5 0x1071e66fc in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::handlePost(MLDB::Utf8String, MLDB::ProcedureRunConfig, bool) rest_collection_impl.h:1852
    #6 0x1072b1d3c in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::RouteManager::addPostRoute()::'lambda0'(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&)::operator()(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&) const rest_collection_impl.h:1649
    #7 0x107c0109c in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:386
    #8 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #9 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #10 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #11 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #12 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #13 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #14 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #15 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #16 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #17 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #18 0x107bffeec in MLDB::RestRequestRouter::handleRequest(MLDB::RestConnection&, MLDB::RestRequest const&) const rest_request_router.cc:326
    #19 0x1071ac7b4 in MLDB::ProcedureCollection::handlePutWithFirstRun(MLDB::Utf8String, MLDB::PolyConfig, bool, bool) procedure_collection.cc:176
    #20 0x1071ab344 in MLDB::ProcedureCollection::handlePut(MLDB::Utf8String, MLDB::PolyConfig, bool) procedure_collection.cc:148
    #21 0x108a060b8 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::RouteManager::addPutRoute()::'lambda'(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&)::operator()(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&) const rest_collection_impl.h:1508
    #22 0x107c0109c in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:386
    #23 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #24 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #25 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #26 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #27 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #28 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #29 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #30 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #31 0x107bffeec in MLDB::RestRequestRouter::handleRequest(MLDB::RestConnection&, MLDB::RestRequest const&) const rest_request_router.cc:326
    #32 0x1115a4cb8 in MLDB::MldbPythonContext::perform(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&) python_plugin_context.cc:730
    #33 0x11156a8ec in _object* boost::python::detail::invoke<boost::python::to_python_value<Json::Value const&>, Json::Value (MLDB::MldbPythonContext::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&), boost::python::arg_from_python<MLDB::MldbPythonContext&>, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>, boost::python::arg_from_python<MLDB::RestParams const&>, boost::python::arg_from_python<Json::Value>, boost::python::arg_from_python<MLDB::RestParams const&> >(boost::python::detail::invoke_tag_<false, true>, boost::python::to_python_value<Json::Value const&> const&, Json::Value (MLDB::MldbPythonContext::*&)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&), boost::python::arg_from_python<MLDB::MldbPythonContext&>&, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>&, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>&, boost::python::arg_from_python<MLDB::RestParams const&>&, boost::python::arg_from_python<Json::Value>&, boost::python::arg_from_python<MLDB::RestParams const&>&) invoke.hpp:86
    #34 0x111569ff8 in boost::python::detail::caller_arity<6u>::impl<Json::Value (MLDB::MldbPythonContext::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&), boost::python::default_call_policies, boost::mpl::vector7<Json::Value, MLDB::MldbPythonContext&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&> >::operator()(_object*, _object*) caller.hpp:233
    #35 0x11392eebc in boost::python::objects::function::call(_object*, _object*) const+0x2b8 (libboost_python39.dylib:arm64+0xeebc)
    #36 0x113931718 in boost::detail::function::void_function_ref_invoker0<boost::python::objects::(anonymous namespace)::bind_return, void>::invoke(boost::detail::function::function_buffer&)+0x18 (libboost_python39.dylib:arm64+0x11718)
    #37 0x1139363a8 in boost::python::handle_exception_impl(boost::function0<void>)+0x5c (libboost_python39.dylib:arm64+0x163a8)
    #38 0x113931200 in boost::python::objects::function_call(_object*, _object*, _object*)+0x44 (libboost_python39.dylib:arm64+0x11200)
    #39 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #40 0x1135ef35c in method_vectorcall+0x278 (Python:arm64+0x6735c)
    #41 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #42 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #43 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #44 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #45 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #46 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #47 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #48 0x1135ef15c in method_vectorcall+0x78 (Python:arm64+0x6715c)
    #49 0x1135ec304 in PyVectorcall_Call+0xb4 (Python:arm64+0x64304)
    #50 0x113780aa0 in partial_call+0x64 (Python:arm64+0x1f8aa0)
    #51 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #52 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #53 0x1136e3f80 in _PyEval_EvalFrameDefault+0x9b90 (Python:arm64+0x15bf80)
    #54 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #55 0x1135ef15c in method_vectorcall+0x78 (Python:arm64+0x6715c)
    #56 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #57 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #58 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #59 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #60 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #61 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #62 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #63 0x1135ef200 in method_vectorcall+0x11c (Python:arm64+0x67200)
    #64 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #65 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #66 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #67 0x1135ebd00 in _PyObject_FastCallDictTstate+0x5c (Python:arm64+0x63d00)
    #68 0x11365ea38 in slot_tp_call+0xb8 (Python:arm64+0xd6a38)
    #69 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #70 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #71 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #72 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #73 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #74 0x1135ef200 in method_vectorcall+0x11c (Python:arm64+0x67200)
    #75 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #76 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #77 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #78 0x1135ebd00 in _PyObject_FastCallDictTstate+0x5c (Python:arm64+0x63d00)
    #79 0x11365ea38 in slot_tp_call+0xb8 (Python:arm64+0xd6a38)
    #80 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #81 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #82 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #83 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #84 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #85 0x1135ef200 in method_vectorcall+0x11c (Python:arm64+0x67200)
    #86 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #87 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #88 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #89 0x1135ebd00 in _PyObject_FastCallDictTstate+0x5c (Python:arm64+0x63d00)
    #90 0x11365ea38 in slot_tp_call+0xb8 (Python:arm64+0xd6a38)
    #91 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #92 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #93 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #94 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #95 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #96 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #97 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #98 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #99 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #100 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #101 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #102 0x1135ebd70 in _PyObject_FastCallDictTstate+0xcc (Python:arm64+0x63d70)
    #103 0x11366005c in slot_tp_init+0xb8 (Python:arm64+0xd805c)
    #104 0x113665cb8 in type_call+0x128 (Python:arm64+0xddcb8)
    #105 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #106 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #107 0x1136e4028 in _PyEval_EvalFrameDefault+0x9c38 (Python:arm64+0x15c028)
    #108 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #109 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #110 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #111 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #112 0x113730ce0 in run_eval_code_obj+0x84 (Python:arm64+0x1a8ce0)
    #113 0x113730bf4 in run_mod+0x6c (Python:arm64+0x1a8bf4)
    #114 0x11372e430 in pyrun_file+0xa4 (Python:arm64+0x1a6430)
    #115 0x113731d28 in PyRun_FileExFlags+0x6c (Python:arm64+0x1a9d28)
    #116 0x11398f418 in MLDB::PythonThread::exec(MLDB::EnterThreadToken const&, MLDB::Utf8String const&, MLDB::Utf8String const&, boost::python::api::object, boost::python::api::object) python_interpreter.cc:318
    #117 0x11159aadc in MLDB::MldbPythonInterpreter::runPythonScript(MLDB::EnterThreadToken const&, MLDB::Utf8String, MLDB::Utf8String, boost::python::api::object, boost::python::api::object) python_plugin_context.cc:360
    #118 0x111547f64 in MLDB::PythonPlugin::handleTypeRoute(MLDB::RestDirectory*, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) python_plugin.cc:356
    #119 0x1074ce55c in MLDB::PolyCollection<MLDB::Plugin>::Registry::handleCustomRequest(MLDB::RestDirectory*, MLDB::Utf8String const&, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&) poly_collection_impl.h:311
    #120 0x1074a04c8 in MLDB::RestRequestMatchResult std::__1::__invoke_void_return_wrapper<MLDB::RestRequestMatchResult>::__call<MLDB::TypeCollection<MLDB::Plugin>::initRoutes(MLDB::RestCollection<MLDB::Utf8String, MLDB::TypeEntry>::RouteManager&)::'lambda1'(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&)&, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&>(MLDB::Plugin&&...) __functional_base:317
    #121 0x107c0109c in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:386
    #122 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #123 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #124 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #125 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #126 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #127 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #128 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #129 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #130 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #131 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #132 0x107bffeec in MLDB::RestRequestRouter::handleRequest(MLDB::RestConnection&, MLDB::RestRequest const&) const rest_request_router.cc:326
    #133 0x107c7b338 in std::__1::__function::__func<MLDB::HttpRestService::init()::$_1, std::__1::allocator<MLDB::HttpRestService::init()::$_1>, void (std::__1::shared_ptr<MLDB::HttpRestEndpoint::RestConnectionHandler>, MLDB::HttpHeader const&, MLDB::Utf8String const&)>::operator()(std::__1::shared_ptr<MLDB::HttpRestEndpoint::RestConnectionHandler>&&, MLDB::HttpHeader const&, MLDB::Utf8String const&) functional:1727
    #134 0x107c54658 in MLDB::HttpRestEndpoint::RestConnectionHandler::handleHttpPayload(MLDB::HttpHeader const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) http_rest_endpoint.cc:151
    #135 0x109801610 in MLDB::HttpLegacySocketHandler::onDone(bool) http_socket_handler.cc:269
    #136 0x10983a8a0 in MLDB::HttpParser::feed(char const*, unsigned long) http_parsers.cc:204
    #137 0x1097fe388 in MLDB::HttpSocketHandler::onReceivedData(char const*, unsigned long) http_socket_handler.cc:106
    #138 0x10999cc7c in boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, std::__1::function<void (boost::system::error_code const&, unsigned long)>, boost::asio::any_io_executor>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) reactive_socket_recv_op.hpp:145
    #139 0x10990f16c in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) scheduler.ipp:486
    #140 0x10990e1cc in boost::asio::detail::scheduler::run(boost::system::error_code&) scheduler.ipp:204
    #141 0x10990da7c in MLDB::AsioThreadPool::Impl::run(int) asio_thread_pool.cc:84
    #142 0x109911e88 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, MLDB::AsioThreadPool::Impl::ensureThreads(int)::'lambda'()> >(void*) thread:291
    #143 0x18e887874 in _pthread_start+0x13c (libsystem_pthread.dylib:arm64e+0x7874)
    #144 0x18e8825dc in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x25dc)

Thread T7 created by T0 here:
    #0 0x109b055b0 in wrap_pthread_create+0x54 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x395b0)
    #1 0x109911d34 in std::__1::thread::thread<MLDB::AsioThreadPool::Impl::ensureThreads(int)::'lambda'(), void>(MLDB::AsioThreadPool::Impl::ensureThreads(int)::'lambda'()&&) thread:307
    #2 0x1099054b8 in MLDB::AsioThreadPool::Impl::ensureThreads(int) asio_thread_pool.cc:66
    #3 0x104961a44 in main mldb_runner.cc:348
    #4 0x18e8a544c in start+0x0 (libdyld.dylib:arm64e+0x1844c)

Thread T33 created by T7 here:
    #0 0x109b055b0 in wrap_pthread_create+0x54 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x395b0)
    #1 0x1089f1560 in std::__1::thread::thread<MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::PolyEntity> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::PolyEntity>)> const&, bool, MLDB::Any)::'lambda'()&, void>(MLDB::Utf8String&&, MLDB::PolyEntity&&...) thread:307
    #2 0x10894d008 in MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::PolyEntity> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::PolyEntity>)> const&, bool, MLDB::Any) rest_collection_impl.h:621
    #3 0x10896aaa8 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::handlePutItl(MLDB::Utf8String, MLDB::PolyConfig, std::__1::function<void (std::__1::shared_ptr<MLDB::PolyEntity>)> const&, bool) rest_collection_impl.h:1767
    #4 0x10896c76c in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::handlePutSync(MLDB::Utf8String, MLDB::PolyConfig, bool) rest_collection_impl.h:1807
    #5 0x1071ac128 in MLDB::ProcedureCollection::handlePutWithFirstRun(MLDB::Utf8String, MLDB::PolyConfig, bool, bool) procedure_collection.cc:164
    #6 0x1071ab344 in MLDB::ProcedureCollection::handlePut(MLDB::Utf8String, MLDB::PolyConfig, bool) procedure_collection.cc:148
    #7 0x108a060b8 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::RouteManager::addPutRoute()::'lambda'(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&)::operator()(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&) const rest_collection_impl.h:1508
    #8 0x107c0109c in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:386
    #9 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #10 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #11 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #12 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #13 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #14 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #15 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #16 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #17 0x107bffeec in MLDB::RestRequestRouter::handleRequest(MLDB::RestConnection&, MLDB::RestRequest const&) const rest_request_router.cc:326
    #18 0x1115a4cb8 in MLDB::MldbPythonContext::perform(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&) python_plugin_context.cc:730
    #19 0x11156a8ec in _object* boost::python::detail::invoke<boost::python::to_python_value<Json::Value const&>, Json::Value (MLDB::MldbPythonContext::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&), boost::python::arg_from_python<MLDB::MldbPythonContext&>, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>, boost::python::arg_from_python<MLDB::RestParams const&>, boost::python::arg_from_python<Json::Value>, boost::python::arg_from_python<MLDB::RestParams const&> >(boost::python::detail::invoke_tag_<false, true>, boost::python::to_python_value<Json::Value const&> const&, Json::Value (MLDB::MldbPythonContext::*&)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&), boost::python::arg_from_python<MLDB::MldbPythonContext&>&, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>&, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>&, boost::python::arg_from_python<MLDB::RestParams const&>&, boost::python::arg_from_python<Json::Value>&, boost::python::arg_from_python<MLDB::RestParams const&>&) invoke.hpp:86
    #20 0x111569ff8 in boost::python::detail::caller_arity<6u>::impl<Json::Value (MLDB::MldbPythonContext::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&), boost::python::default_call_policies, boost::mpl::vector7<Json::Value, MLDB::MldbPythonContext&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&> >::operator()(_object*, _object*) caller.hpp:233
    #21 0x11392eebc in boost::python::objects::function::call(_object*, _object*) const+0x2b8 (libboost_python39.dylib:arm64+0xeebc)
    #22 0x113931718 in boost::detail::function::void_function_ref_invoker0<boost::python::objects::(anonymous namespace)::bind_return, void>::invoke(boost::detail::function::function_buffer&)+0x18 (libboost_python39.dylib:arm64+0x11718)
    #23 0x1139363a8 in boost::python::handle_exception_impl(boost::function0<void>)+0x5c (libboost_python39.dylib:arm64+0x163a8)
    #24 0x113931200 in boost::python::objects::function_call(_object*, _object*, _object*)+0x44 (libboost_python39.dylib:arm64+0x11200)
    #25 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #26 0x1135ef35c in method_vectorcall+0x278 (Python:arm64+0x6735c)
    #27 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #28 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #29 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #30 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #31 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #32 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #33 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #34 0x1135ef15c in method_vectorcall+0x78 (Python:arm64+0x6715c)
    #35 0x1135ec304 in PyVectorcall_Call+0xb4 (Python:arm64+0x64304)
    #36 0x113780aa0 in partial_call+0x64 (Python:arm64+0x1f8aa0)
    #37 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #38 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #39 0x1136e3f80 in _PyEval_EvalFrameDefault+0x9b90 (Python:arm64+0x15bf80)
    #40 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #41 0x1135ef15c in method_vectorcall+0x78 (Python:arm64+0x6715c)
    #42 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #43 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #44 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #45 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #46 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #47 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #48 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #49 0x1135ef200 in method_vectorcall+0x11c (Python:arm64+0x67200)
    #50 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #51 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #52 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #53 0x1135ebd00 in _PyObject_FastCallDictTstate+0x5c (Python:arm64+0x63d00)
    #54 0x11365ea38 in slot_tp_call+0xb8 (Python:arm64+0xd6a38)
    #55 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #56 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #57 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #58 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #59 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #60 0x1135ef200 in method_vectorcall+0x11c (Python:arm64+0x67200)
    #61 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #62 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #63 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #64 0x1135ebd00 in _PyObject_FastCallDictTstate+0x5c (Python:arm64+0x63d00)
    #65 0x11365ea38 in slot_tp_call+0xb8 (Python:arm64+0xd6a38)
    #66 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #67 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #68 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #69 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #70 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #71 0x1135ef200 in method_vectorcall+0x11c (Python:arm64+0x67200)
    #72 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #73 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #74 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #75 0x1135ebd00 in _PyObject_FastCallDictTstate+0x5c (Python:arm64+0x63d00)
    #76 0x11365ea38 in slot_tp_call+0xb8 (Python:arm64+0xd6a38)
    #77 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #78 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #79 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #80 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #81 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #82 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #83 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #84 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #85 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #86 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #87 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #88 0x1135ebd70 in _PyObject_FastCallDictTstate+0xcc (Python:arm64+0x63d70)
    #89 0x11366005c in slot_tp_init+0xb8 (Python:arm64+0xd805c)
    #90 0x113665cb8 in type_call+0x128 (Python:arm64+0xddcb8)
    #91 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #92 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #93 0x1136e4028 in _PyEval_EvalFrameDefault+0x9c38 (Python:arm64+0x15c028)
    #94 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #95 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #96 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #97 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #98 0x113730ce0 in run_eval_code_obj+0x84 (Python:arm64+0x1a8ce0)
    #99 0x113730bf4 in run_mod+0x6c (Python:arm64+0x1a8bf4)
    #100 0x11372e430 in pyrun_file+0xa4 (Python:arm64+0x1a6430)
    #101 0x113731d28 in PyRun_FileExFlags+0x6c (Python:arm64+0x1a9d28)
    #102 0x11398f418 in MLDB::PythonThread::exec(MLDB::EnterThreadToken const&, MLDB::Utf8String const&, MLDB::Utf8String const&, boost::python::api::object, boost::python::api::object) python_interpreter.cc:318
    #103 0x11159aadc in MLDB::MldbPythonInterpreter::runPythonScript(MLDB::EnterThreadToken const&, MLDB::Utf8String, MLDB::Utf8String, boost::python::api::object, boost::python::api::object) python_plugin_context.cc:360
    #104 0x111547f64 in MLDB::PythonPlugin::handleTypeRoute(MLDB::RestDirectory*, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) python_plugin.cc:356
    #105 0x1074ce55c in MLDB::PolyCollection<MLDB::Plugin>::Registry::handleCustomRequest(MLDB::RestDirectory*, MLDB::Utf8String const&, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&) poly_collection_impl.h:311
    #106 0x1074a04c8 in MLDB::RestRequestMatchResult std::__1::__invoke_void_return_wrapper<MLDB::RestRequestMatchResult>::__call<MLDB::TypeCollection<MLDB::Plugin>::initRoutes(MLDB::RestCollection<MLDB::Utf8String, MLDB::TypeEntry>::RouteManager&)::'lambda1'(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&)&, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&>(MLDB::Plugin&&...) __functional_base:317
    #107 0x107c0109c in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:386
    #108 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #109 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #110 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #111 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #112 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #113 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #114 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #115 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #116 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #117 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #118 0x107bffeec in MLDB::RestRequestRouter::handleRequest(MLDB::RestConnection&, MLDB::RestRequest const&) const rest_request_router.cc:326
    #119 0x107c7b338 in std::__1::__function::__func<MLDB::HttpRestService::init()::$_1, std::__1::allocator<MLDB::HttpRestService::init()::$_1>, void (std::__1::shared_ptr<MLDB::HttpRestEndpoint::RestConnectionHandler>, MLDB::HttpHeader const&, MLDB::Utf8String const&)>::operator()(std::__1::shared_ptr<MLDB::HttpRestEndpoint::RestConnectionHandler>&&, MLDB::HttpHeader const&, MLDB::Utf8String const&) functional:1727
    #120 0x107c54658 in MLDB::HttpRestEndpoint::RestConnectionHandler::handleHttpPayload(MLDB::HttpHeader const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) http_rest_endpoint.cc:151
    #121 0x109801610 in MLDB::HttpLegacySocketHandler::onDone(bool) http_socket_handler.cc:269
    #122 0x10983a8a0 in MLDB::HttpParser::feed(char const*, unsigned long) http_parsers.cc:204
    #123 0x1097fe388 in MLDB::HttpSocketHandler::onReceivedData(char const*, unsigned long) http_socket_handler.cc:106
    #124 0x10999cc7c in boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, std::__1::function<void (boost::system::error_code const&, unsigned long)>, boost::asio::any_io_executor>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) reactive_socket_recv_op.hpp:145
    #125 0x10990f16c in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) scheduler.ipp:486
    #126 0x10990e1cc in boost::asio::detail::scheduler::run(boost::system::error_code&) scheduler.ipp:204
    #127 0x10990da7c in MLDB::AsioThreadPool::Impl::run(int) asio_thread_pool.cc:84
    #128 0x109911e88 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, MLDB::AsioThreadPool::Impl::ensureThreads(int)::'lambda'()> >(void*) thread:291
    #129 0x18e887874 in _pthread_start+0x13c (libsystem_pthread.dylib:arm64e+0x7874)
    #130 0x18e8825dc in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x25dc)

SUMMARY: AddressSanitizer: heap-use-after-free sql_expression.cc:3616 in MLDB::SelectExpressionDescription::printJsonTyped(MLDB::SelectExpression const*, MLDB::JsonPrintingContext&) const
Shadow bytes around the buggy address:
  0x0070230894f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x007023089500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x007023089510: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x007023089520: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x007023089530: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x007023089540: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x007023089550: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x007023089560: fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa
  0x007023089570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x007023089580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x007023089590: 00 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==67440==ABORTING
                       MLDBFB-724_classifier_exp_segfault_test.py FAILED