Crash on main branch for "invalid record pointer"

[TheraNinjaCat]

When running the current main 7f79f14737cb0782997d5355ddc833fea55d8947 branch against my dataset I get the following new crash: RUST_BACKTRACE=full target/debug/ntdsextract2 /working/NTDS.dit user -D -F json

thread 'main' panicked at src/object_tree.rs:70:32:
invalid record pointer: id=14270/row=10907
stack backtrace:
   0:     0xaaaabf13203c - std::backtrace_rs::backtrace::libunwind::trace::hd36fbcb47e754031
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:     0xaaaabf13203c - std::backtrace_rs::backtrace::trace_unsynchronized::hf1edef0679fd1fd7
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0xaaaabf13203c - std::sys_common::backtrace::_print_fmt::hfcc2839667ddb682
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/sys_common/backtrace.rs:68:5
   3:     0xaaaabf13203c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h18d7a2571106fbf4
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/sys_common/backtrace.rs:44:22
   4:     0xaaaabf1514e0 - core::fmt::rt::Argument::fmt::h6694c2ecb3cadfef
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/fmt/rt.rs:165:63
   5:     0xaaaabf1514e0 - core::fmt::write::hbfa73a7a5186286a
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/fmt/mod.rs:1168:21
   6:     0xaaaabf12fb38 - std::io::Write::write_fmt::h5bae5422b72e7351
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/io/mod.rs:1835:15
   7:     0xaaaabf131e84 - std::sys_common::backtrace::_print::he42d75514d09ae8c
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/sys_common/backtrace.rs:47:5
   8:     0xaaaabf131e84 - std::sys_common::backtrace::print::h2a658d0148ab667e
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/sys_common/backtrace.rs:34:9
   9:     0xaaaabf133084 - std::panicking::default_hook::{{closure}}::h1495412aefb2bb36
  10:     0xaaaabf132cec - std::panicking::default_hook::he773715f5fd59c50
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:298:9
  11:     0xaaaabf1334b4 - std::panicking::rust_panic_with_hook::h32dd7f185783ae19
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:795:13
  12:     0xaaaabf133368 - std::panicking::begin_panic_handler::{{closure}}::h6e3724fa5e927a4e
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:664:13
  13:     0xaaaabf13251c - std::sys_common::backtrace::__rust_end_short_backtrace::h877ff9009acc9a78
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/sys_common/backtrace.rs:171:18
  14:     0xaaaabf1330dc - rust_begin_unwind
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:652:5
  15:     0xaaaabec31490 - core::panicking::panic_fmt::hcb4d02f688afee88
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/panicking.rs:72:14
  16:     0xaaaabecbe93c - libntdsextract2::object_tree::ObjectTree::dn_of::{{closure}}::h8a5d1b635d77d1a8
                               at /ntdsextract2/src/object_tree.rs:70:32
  17:     0xaaaabecad414 - core::option::Option<T>::unwrap_or_else::hc6582d6cb2c391a3
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/option.rs:980:21
  18:     0xaaaabed1dda4 - libntdsextract2::object_tree::ObjectTree::dn_of::hf59768ed45b2a333
                               at /ntdsextract2/src/object_tree.rs:68:9
  19:     0xaaaabec4a08c - libntdsextract2::ntds::data_table::DataTable::show_typed_objects::h4dcf07f499331ecd
                               at /ntdsextract2/src/ntds/data_table.rs:314:39
  20:     0xaaaabec795f8 - libntdsextract2::c_database::CDatabase::show_typed_objects::h08f2efe95d2c59b9
                               at /ntdsextract2/src/c_database.rs:81:9
  21:     0xaaaabec794ac - libntdsextract2::c_database::CDatabase::show_users::hbc45aae24490769b
                               at /ntdsextract2/src/c_database.rs:62:9
  22:     0xaaaabec4e070 - ntdsextract2::main::h89d6bf8c02b28172
                               at /ntdsextract2/src/main.rs:20:13
  23:     0xaaaabec443bc - core::ops::function::FnOnce::call_once::hfb8470a01d5fa166
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/ops/function.rs:250:5
  24:     0xaaaabec320cc - std::sys_common::backtrace::__rust_begin_short_backtrace::h8521348bf508effe
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/sys_common/backtrace.rs:155:18
  25:     0xaaaabec34f6c - std::rt::lang_start::{{closure}}::h6e056653c7792a42
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/rt.rs:159:18
  26:     0xaaaabf12a2b8 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h0ea9c1019722c54d
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/ops/function.rs:284:13
  27:     0xaaaabf12a2b8 - std::panicking::try::do_call::hbb3b1afed470bda3
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:559:40
  28:     0xaaaabf12a2b8 - std::panicking::try::h0129c5d9eb3a8b93
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:523:19
  29:     0xaaaabf12a2b8 - std::panic::catch_unwind::he5a8b1f9de3df436
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panic.rs:149:14
  30:     0xaaaabf12a2b8 - std::rt::lang_start_internal::{{closure}}::h91db0a62cd50de33
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/rt.rs:141:48
  31:     0xaaaabf12a2b8 - std::panicking::try::do_call::hc120f530921da74e
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:559:40
  32:     0xaaaabf12a2b8 - std::panicking::try::hf07ae5daa8b437dd
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:523:19
  33:     0xaaaabf12a2b8 - std::panic::catch_unwind::h27788fb573347317
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panic.rs:149:14
  34:     0xaaaabf12a2b8 - std::rt::lang_start_internal::hd1567374aba5b64a
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/rt.rs:141:20
  35:     0xaaaabec34f3c - std::rt::lang_start::h3403b7a824cc1b88
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/rt.rs:158:17
  36:     0xaaaabec4e6d4 - main
  37:     0xffffa9ab3e18 - __libc_start_main
  38:     0xaaaabec31d0c - <unknown>

[janstarke]

Interesting. I implemented an aggressive error handling

self.record_index
            .get(ptr)
            .unwrap_or_else(|| panic!("invalid record pointer: {ptr}"))

because I assumed the record_index contains all entries. Obviously this assumption was wrong.

This situation might happen if there is an inconsistency in your link table. I'll add a better error handling for this

[TheraNinjaCat]

Yeah, I'm not surprised by inconsistencies in the link table, I've done analysis on quite a number of NTDS files at this point, and I think just about every single one of them has had some... weirdness. I would assume it's some mix of long-lived domains having some errors compounded over time, upgrades to the underlying Windows server, or errors caused during the export of the file.

[TheraNinjaCat]

I have tested this commit 889841aed1d05bdf207ce628835429044137ca5f and now get this error:

Error: invalid value detected: '"Long"'; expected type was String (one of (text, largetext or binary)

Stack backtrace:
   0: std::backtrace_rs::backtrace::libunwind::trace
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1: std::backtrace_rs::backtrace::trace_unsynchronized
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2: std::backtrace::Backtrace::create
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/backtrace.rs:331:13
   3: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.86/src/backtrace.rs:27:14
   4: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/result.rs:1989:27
   5: libntdsextract2::value::from_value::FromValue::from_record_opt
             at ./src/value/from_value.rs:21:16
   6: <libntdsextract2::cache::meta_data_cache::MetaDataCache as core::convert::TryFrom<&libntdsextract2::esedbinfo::EsedbInfo>>::try_from
             at ./src/cache/meta_data_cache.rs:99:29
   7: libntdsextract2::c_database::CDatabase::new
             at ./src/c_database.rs:19:30
   8: ntdsextract2::main
             at ./src/main.rs:47:20
   9: core::ops::function::FnOnce::call_once
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/ops/function.rs:250:5
  10: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/sys_common/backtrace.rs:155:18
  11: std::rt::lang_start::{{closure}}
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/rt.rs:159:18
  12: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/ops/function.rs:284:13
  13: std::panicking::try::do_call
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:559:40
  14: std::panicking::try
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:523:19
  15: std::panic::catch_unwind
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panic.rs:149:14
  16: std::rt::lang_start_internal::{{closure}}
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/rt.rs:141:48
  17: std::panicking::try::do_call
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:559:40
  18: std::panicking::try
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:523:19
  19: std::panic::catch_unwind
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panic.rs:149:14
  20: std::rt::lang_start_internal
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/rt.rs:141:20
  21: std::rt::lang_start
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/rt.rs:158:17
  22: main
  23: __libc_start_main
  24: <unknown>

[janstarke]

Funny. The line that causes this issue is https://github.com/janstarke/ntdsextract2/blob/88616a18f5c23df93629d492e090e5504afee17d/src/cache/meta_data_cache.rs#L98-L99

Obviously there is a number stored in the column for samAccountName :laughing:

I'll also change this into a log message

[janstarke]

I updated the branch, but I cannot test it. Would you please give it a try?

[TheraNinjaCat]

I've just tested this out, it now runs to completion, there's a whole bunch of missing entries because of an inconsistency in the link_table, and a single error reading samAccountName because it's a long, besides those however it more or less seems to run correctly.

I've run users normally, with --member-of sam, --member-of dn, and --member-of sid, and they all seem to work correctly. It seems that it outputs null in the member_of list for sam and sid, however for dn it seems to be putting entries like "id=21607/row=16996". I've run group with --member-of dn and this acts in the same way. I suspect this is because the dn couldn't be interpreted, and in these objects the distinguished_name field is null, which ironically makes the id/row numbers useless as well, as they can't be directly linked to any other object ha ha ha. As it is, however, it is in a state where I can run the tool multiple times with different member_of fields, and parse all the data to come up with unique links to objects. Thank you heaps for your efforts in implementing these changes and fixes.

[janstarke]

Yes, I know that problem of missing entries and the id/row as replacement. We could instead print something like MISSING ENTRY FOR ID 21607, which could more descriptive

[TheraNinjaCat]

I think for the json output null is probably the cleanest value, it’s already present in the other member_of formats, and it seems to be how it’s handled for the main distinguished _name field as well. It shows that there is missing data, but it’s possible to use the other member_of formats to fill in the gaps.