jemalloc_nodump_allocator should not call malloc_usable_size on memory allocated from arenas

[tudor]

Expected behavior

ASAN doesn't complain

Actual behavior

ASAN complains with something like

==661==ERROR: AddressSanitizer: attempting to call malloc_usable_size() for pointer which is not owned: 0x7f6a99501000
    #0 0x38b3ed in malloc_usable_size (/opt/rockset/rs/cpp/out/debug-clang-sanitize-address/leaf/leaf_server+0x38b3ed) (BuildId: 059497e5183f7676)
    #1 0x30b57f in __asan::asan_malloc_usable_size(void const*, unsigned long, unsigned long) crtstuff.c
    #2 0x7f6b19625b6c in rocksdb::BlockContents::usable_size() const /src/rocksdb/./table/format.h:294:37
    #3 0x7f6b19625b6c in rocksdb::Block::usable_size() const /src/rocksdb/./table/block_based/block.h:166:60
    #4 0x7f6b19625b6c in rocksdb::Block::ApproximateMemoryUsage() const /src/rocksdb/table/block_based/block.cc:1117:29
    #5 0x7f6b19639c5c in rocksdb::Status rocksdb::BlockBasedTableBuilder::InsertBlockInCache<rocksdb::Block>(rocksdb::Slice const&, rocksdb::BlockHandle const*, rocksdb::BlockType) /src/rocksdb/table/block_based/block_based_table_builder.cc:1545:57
    #6 0x7f6b1963199e in rocksdb::BlockBasedTableBuilder::InsertBlockInCacheHelper(rocksdb::Slice const&, rocksdb::BlockHandle const*, rocksdb::BlockType, bool) /src/rocksdb/table/block_based/block_based_table_builder.cc:1497:34
    #7 0x7f6b19632326 in rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, rocksdb::BlockType, rocksdb::Slice const*, bool) /src/rocksdb/table/block_based/block_based_table_builder.cc:1314:37
    #8 0x7f6b19632bd9 in rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, rocksdb::BlockType) /src/rocksdb/table/block_based/block_based_table_builder.cc:1094:16
    #9 0x7f6b1963534b in rocksdb::BlockBasedTableBuilder::EnterUnbuffered() /src/rocksdb/table/block_based/block_based_table_builder.cc:1975:17
    #10 0x7f6b19635e57 in rocksdb::BlockBasedTableBuilder::Finish() /src/rocksdb/table/block_based/block_based_table_builder.cc:2007:20
    #11 0x7f6b1933c6be in rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned long*, unsigned long*, unsigned long*) /src/rocksdb/db/builder.cc:261:26
    #12 0x7f6b19479cd2 in rocksdb::FlushJob::WriteLevel0Table() /src/rocksdb/db/flush_job.cc:930:21
    #13 0x7f6b1947c14e in rocksdb::FlushJob::Run(rocksdb::LogsWithPrepTracker*, rocksdb::FileMetaData*, bool*) /src/rocksdb/db/flush_job.cc:279:25
    #14 0x7f6b193f8837 in rocksdb::DBImpl::AtomicFlushMemTablesToOutputFiles(rocksdb::autovector<rocksdb::DBImpl::BGFlushArg, 8ul> const&, bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::Env::Priority) /src/rocksdb/db/db_impl/db_impl_compaction_flush.cc:527:41
    #15 0x7f6b193f9f82 in rocksdb::DBImpl::FlushMemTablesToOutputFiles(rocksdb::autovector<rocksdb::DBImpl::BGFlushArg, 8ul> const&, bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::Env::Priority) /src/rocksdb/db/db_impl/db_impl_compaction_flush.cc:364:74
    #16 0x7f6b193fb5fa in rocksdb::DBImpl::BackgroundFlush(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::FlushReason*, rocksdb::Env::Priority) /src/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2807:41
    #17 0x7f6b194004a0 in rocksdb::DBImpl::BackgroundCallFlush(rocksdb::Env::Priority) /src/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2847:51

Steps to reproduce the behavior

Build with ASAN, do a background flush

The fix would be to use nallocx(p, flags) instead of malloc_usable_size(p), with the same flags as passed to the mallocx call (MALLOCX_ARENA(arena_index_) | GetThreadSpecificCache(size))

[siying]

We usually run ASAN with jemalloc disabled. If it's just for ASAN, can we add a option or compiler flag that switches to the behavior you would like? You are welcome to send a PR.