Include deleted entries in timeline

[Darkrael]

Would it be possible to also include deleted entries in the timeline like ntdsxtract does?:

"=""2023-04-17 02:32:46+00:00""";"Created";"138244";"Exampleuser\nDEL:ed10da0c-3b8e-4718-98ac-183e9c49f235 (ed10da0c-3b8e-4718-98ac-183e9c49f235)";""
"=""2023-04-19 02:17:34+00:00""";"Modified";"138244";"Exampleuser\nDEL:ed10da0c-3b8e-4718-98ac-183e9c49f235 (ed10da0c-3b8e-4718-98ac-183e9c49f235)";""

[janstarke]

I'm not sure. I'm currently using a library which does not cope with deleted entries. I need to do some research about this...

[janstarke]

I did some refactoring which was needed to implement this, so this required a lot of time. However, ntdsextract2 displays deleted entries in the timeline. Such entries are marked with (DELETED) at the moment. The changes are in the branch 10-include-deleted-entries-in-timeline.

Could you please give it a try?

[Darkrael]

Hey, first of all, thank you for putting all the work in to implement this. I've just tried it on 2 NTDS databases i had and it crashed on both with different errors. The first one i've tried is a sample NTDS by Didier Stevens that i've found online: https://blog.didierstevens.com/2016/07/12/practice-ntds-dit-file-part-1/. There it crashes with:

> cargo run -r -- ntds.dit timeline --include-deleted -v
thread 'main' panicked at src/cache/meta_data_cache.rs:97:17:
more than one root object found

And on one production NTDS (that i cannot provide for obvious reasons) it crashes with another error (i've also replaced the UUID with a random one just in case):

> cargo run -r -- ntds.dit timeline --include-deleted -v
Error: invalid value detected: '"LargeText(PublicFolderMailboxMigrationRequests\nCNF:6dc93e03-f756-45df-bf81-7df1be6f84e6)"'

[janstarke]

I fixed the first problem and (hopefully) also the second one. I'm not really sure, because I don't have a database with conflicting records (which is what CNF means) available. Can you please check if it is working now?

Regards, Jan

[Darkrael]

The first issue is definitely fixed and the NTDS is successfully parsed. The second one still crashes with the same error. Through a bit of debugging, i've found the source. I did a bit of digging and your fix is actually working, but you misspelled CNF with CFN :D. Unfortunately after fixing, it now crashes with the following message:

thread 'main' panicked at src/cache/meta_data_cache.rs:191:26:
no entry found for key
stack backtrace:
   0:     0x560a4a58481c - std::backtrace_rs::backtrace::libunwind::trace::ha69d38c49f1bf263
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x560a4a58481c - std::backtrace_rs::backtrace::trace_unsynchronized::h93125d0b85fd543c
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x560a4a58481c - std::sys_common::backtrace::_print_fmt::h8d65f438e8343444
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x560a4a58481c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h41751d2af6c8033a
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x560a4a5ab25c - core::fmt::rt::Argument::fmt::h5db2f552d8a28f63
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/fmt/rt.rs:138:9
   5:     0x560a4a5ab25c - core::fmt::write::h99465148a27e4883
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/fmt/mod.rs:1114:21
   6:     0x560a4a58224e - std::io::Write::write_fmt::hee8dfd57bd179ab2
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/io/mod.rs:1763:15
   7:     0x560a4a584604 - std::sys_common::backtrace::_print::h019a3cee3e814da4
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x560a4a584604 - std::sys_common::backtrace::print::h55694121c2ddf918
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x560a4a585a03 - std::panicking::default_hook::{{closure}}::h29cbe3da3891b0b0
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:272:22
  10:     0x560a4a585724 - std::panicking::default_hook::h881e76b2b8c74280
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:292:9
  11:     0x560a4a585f85 - std::panicking::rust_panic_with_hook::hcc36e25b6e33969c
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:731:13
  12:     0x560a4a585e81 - std::panicking::begin_panic_handler::{{closure}}::ha415efb0f69f41f9
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:609:13
  13:     0x560a4a584d46 - std::sys_common::backtrace::__rust_end_short_backtrace::h395fe90f99451e4e
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/sys_common/backtrace.rs:170:18
  14:     0x560a4a585bd2 - rust_begin_unwind
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:597:5
  15:     0x560a4a3648f5 - core::panicking::panic_fmt::h452a83e54ecd764e
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/panicking.rs:72:14
  16:     0x560a4a3648b3 - core::panicking::panic_display::h559b2f73c68a58f2
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/panicking.rs:168:5
  17:     0x560a4a3648b3 - core::panicking::panic_str::h9521b3ee4fd35ecd
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/panicking.rs:152:5
  18:     0x560a4a3648b3 - core::option::expect_failed::h92d9ca41185c3cd6
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/option.rs:1988:5
  19:     0x560a4a3c4e78 - <libntdsextract2::cache::record::Record as libntdsextract2::cache::record::with_value::WithValue<libntdsextract2::cache::column_index::ColumnIndex>>::with_value::h492220066931d67c
  20:     0x560a4a3b9570 - libntdsextract2::ntds::link_table::LinkTable::new::h125f39eb9ae42aa7
  21:     0x560a4a3a8f3d - libntdsextract2::c_database::CDatabase::new::hb1553bfd069dde76
  22:     0x560a4a38d58c - ntdsextract2::main::h634f5dc3a1b07aff
  23:     0x560a4a3862c3 - std::sys_common::backtrace::__rust_begin_short_backtrace::hcba50cc8c4310310
  24:     0x560a4a38bcdd - std::rt::lang_start::{{closure}}::h493d3428b3e78c39
  25:     0x560a4a57d38b - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h14c5f6d1cd70a60f
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/ops/function.rs:284:13
  26:     0x560a4a57d38b - std::panicking::try::do_call::h2d02374ca451446a
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:504:40
  27:     0x560a4a57d38b - std::panicking::try::h9f7922394bf57392
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:468:19
  28:     0x560a4a57d38b - std::panic::catch_unwind::ha1600f9dd4ee7270
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panic.rs:142:14
  29:     0x560a4a57d38b - std::rt::lang_start_internal::{{closure}}::hfbd80e7d681b21a1
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/rt.rs:148:48
  30:     0x560a4a57d38b - std::panicking::try::do_call::heacaa33dbdaa16e0
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:504:40
  31:     0x560a4a57d38b - std::panicking::try::h637875f7c9db85ea
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:468:19
  32:     0x560a4a57d38b - std::panic::catch_unwind::h4caa9c0c78cb4c19
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panic.rs:142:14
  33:     0x560a4a57d38b - std::rt::lang_start_internal::h2d6a60ec944b523d
                               at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/rt.rs:148:20
  34:     0x560a4a38df45 - main
  35:     0x7ff7543e96ca - <unknown>
  36:     0x7ff7543e9785 - __libc_start_main
  37:     0x560a4a365091 - _start
  38:                0x0 - <unknown>

[janstarke]

I fixed the typo and found some some invalid assumption of mine: In your case there seems to be a link in the link_table which has no corresponding record in the data_table (I assumed that both tables was consistent). I added some error handling code for such a case. Now, I print a warning and ignore that link

[Darkrael]

Nice, all errors are now gone, but there are a lot of missing deleted entries. For example, ntdsxtract gives me the following deleted entries for a user (censored of cause):

"Timestamp;""Event"";""Record ID"";""Object name"";""Object type"""
"=""2023-02-27 08:39:52+00:00"";""Created"";""23010"";""Example User\nDEL:3095fd27-f231-4561-b4f2-a8dc6837b252 (3095fd27-f231-4561-b4f2-a8dc6837b252)"";"""""
"=""2023-08-26 09:17:38+00:00"";""Modified"";""23010"";""Example User\nDEL:3095fd27-f231-4561-b4f2-a8dc6837b252 (3095fd27-f231-4561-b4f2-a8dc6837b252)"";"""""

ntdsextract2 is completely missing these entries and this user. There were 3148 warning messages about backward links and 550 warnings about missing forwards links. None of those warning had the Record ID of the entry that ntdsxtract found (23010)

[janstarke]

When objects are being deleted, some of their attributes are being deleted as well. ntdsextract2 requires some attributes to be present in order to handle those objects correctly. Can you see some differences between the objects that are shown and those that are missing?

[Darkrael]

I'm not sure about the differences, but i've managed to get a sample ntds.dit from a testing environment from a colleague, where he deleted the user "attacker" and some more users are deleted. I can share this one with you: sample_ntds.dit.zip.

ntdsextract2 only manages to show 1 deleted object:

0|CRUpdate (DELETED) (object, record creation time)|0||0|0|0|-1|-1|-1|1693072034
0|CRUpdate (DELETED) (object, object created)|0||0|0|0|-1|-1|-1|1693072034
0|CRUpdate (DELETED) (object, object changed)|0||0|0|0|-1|-1|-1|1693072537

ntdsxtract shows 100 deleted objects, for example the user "attacker":

"=""2024-01-10 09:44:41+00:00""";"Created";"4385";"attacker\nDEL:05206002-b011-421e-93a5-df478e1493ae (05206002-b011-421e-93a5-df478e1493ae)";""
"=""2024-01-10 09:44:54+00:00""";"Modified";"4385";"attacker\nDEL:05206002-b011-421e-93a5-df478e1493ae (05206002-b011-421e-93a5-df478e1493ae)";""

[janstarke]

I used the wrong Deleted Objects container. For some reasons, there might be two of them; one at the top level and one below Configuration. However, I now join the set of objects in the toplevel Deleted Objects and all objects with a deletion information, which includes the ones below Configuration. I now also can see the attacker:

2024-01-10T09:44:28+00:00,0,...b,,0,0,4384,"attacker (Deleted Object, record creation time)"
2024-01-10T09:44:28+00:00,0,...b,,0,0,4384,"attacker (Deleted Object, object created)"
2024-01-10T09:44:28+00:00,0,...b,,0,0,4384,"attacker (Deleted Object, object changed)"
2024-01-10T09:44:41+00:00,0,...b,,0,0,4385,"attacker (Deleted Object, object created)"
2024-01-10T09:44:54+00:00,0,...b,,0,0,4385,"attacker (Deleted Object, record creation time)"
2024-01-10T09:44:54+00:00,0,...b,,0,0,4385,"attacker (Deleted Object, object changed)"