Large users file, only SID visible in bloodhound.

[jbfuzier]

Hello,

Describe the bug We have a 730Mo users json generated by sharphound 2.0.0. In the file, the information related to the user is populated

Example (redated) : {"Properties":{"domain":"XXXX.XXXX.NET","name":"USERXXXX@XXXX.XXXX.NET","distinguishedname":"CN=USERXXXX,DC=XXXX,DC=XXXX,DC=NET","domainsid":"S-1-5-21-XXXXXXXXXXXXXXXXXXXXXXXXXXX","highvalue":false,"samaccountname":"USERXXXX","description":null,"whencreated":1037706616,"sensitive":false,"dontreqpreauth":false,"passwordnotreqd":false,"unconstraineddelegation":false,"pwdneverexpires":false,"enabled":true,"trustedtoauth":false,"lastlogon":1692544133,"lastlogontimestamp":1692377878,"pwdlastset":1685978889,"serviceprincipalnames":[],"hasspn":false,"displayname":"USERXXXX","email":"USERXXXX@email.com","title":"USERXXXX","homedirectory":null,"userpassword":null,"unixpassword":null,"unicodepassword":null,"sfupassword":null,"logonscript":null,"admincount":false,"sidhistory":[]},"AllowedToDelegate":[],"PrimaryGroupSID":"S-1-5-21-XXXXXXX-XXXXXXXXXXXXXXXXXX","HasSIDHistory":[],"SPNTargets":[],"Aces":[{"PrincipalSID":"S-1-5-21-XXXXXXXXXXXXXXXXXX","PrincipalType":"User","RightName":"Owns","IsInherited":false},{"PrincipalSID":"XXXX.XXXX.NET-S-1-5-32-548","PrincipalType":"Group","RightName":"GenericAll","IsInherited":false},{"PrincipalSID":"S-1-5-21-XXXXXX-XXXXX-XXXXX-XXXX","PrincipalType":"Group","RightName":"GenericAll","IsInherited":false}],"ObjectIdentifier":"S-1-5-21-XXXXXXXXXXXXXXXXXXXXXXX","IsDeleted":false,"IsACLProtected":false,"ContainedBy":null}

Screenshots

Screenshot showing the issue (the user exists in the AD, the SID is resolvable and attributes such as samaccountname are available in the user json file

On the same bloodhound install, with the same sharphound flags acquired from the same PC with the same account but for a different domain we got the expected result :

Data quality show a coherent number of users :

Ingestion appears as complete :

BUT after sometime it changes to :

Ingestion logs :

bloodhound-bloodhound-1  | {"level":"debug","elapsed":3.294654,"time":"2023-09-05T12:38:04.663248988Z","message":"Starting new file upload job"}
bloodhound-bloodhound-1  | {"level":"debug","elapsed":1.765631,"time":"2023-09-05T12:38:08.028523913Z","message":"Finished file upload job"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":0.0044,"time":"2023-09-05T12:38:38.909690373Z","message":"Starting analysis"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":10419.644269,"time":"2023-09-05T12:38:49.329354983Z","message":"Fix well known node types"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":2430.994416,"time":"2023-09-05T12:38:51.760367348Z","message":"Domain Associations"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":11.772098,"time":"2023-09-05T12:38:51.772155376Z","message":"Link well known groups"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":1068.548809,"time":"2023-09-05T12:38:52.840723983Z","message":"ClearSystemTagsIncludeMeta"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":1082.749249,"time":"2023-09-05T12:38:52.854921721Z","message":"Updated asset group isolation tags"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:38:52.860734655Z","message":"Fetching tier zero nodes for domain 666519"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:38:52.860868455Z","message":"Fetching tier zero nodes for domain 666520"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:38:52.860944835Z","message":"Fetching tier zero nodes for domain 666522"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:38:52.861020825Z","message":"Fetching tier zero nodes for domain 666524"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:38:52.861090491Z","message":"Fetching tier zero nodes for domain 666526"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:38:52.861158906Z","message":"Fetching tier zero nodes for domain 666530"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":306.58747,"time":"2023-09-05T12:38:53.16775915Z","message":"Finished fetching tier zero nodes for domain 666530"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:38:53.167777096Z","message":"Fetching tier zero nodes for domain 666518"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":451.06731,"time":"2023-09-05T12:38:53.312168933Z","message":"Finished fetching tier zero nodes for domain 666526"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:38:53.312184449Z","message":"Fetching tier zero nodes for domain 666523"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":486.903584,"time":"2023-09-05T12:38:53.347936072Z","message":"Finished fetching tier zero nodes for domain 666524"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:38:53.347952816Z","message":"Fetching tier zero nodes for domain 666525"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":617.191685,"time":"2023-09-05T12:38:53.478071547Z","message":"Finished fetching tier zero nodes for domain 666520"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:38:53.478089871Z","message":"Fetching tier zero nodes for domain 666527"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":673.087708,"time":"2023-09-05T12:38:53.534042552Z","message":"Finished fetching tier zero nodes for domain 666522"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:38:53.534060274Z","message":"Fetching tier zero nodes for domain 666528"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":677.881019,"time":"2023-09-05T12:38:53.5386312Z","message":"Finished fetching tier zero nodes for domain 666519"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:38:53.538644245Z","message":"Fetching tier zero nodes for domain 666529"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":262.530373,"time":"2023-09-05T12:38:53.574723508Z","message":"Finished fetching tier zero nodes for domain 666523"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:38:53.574738492Z","message":"Fetching tier zero nodes for domain 666521"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":251.378548,"time":"2023-09-05T12:38:53.599339641Z","message":"Finished fetching tier zero nodes for domain 666525"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":253.588974,"time":"2023-09-05T12:38:53.828337902Z","message":"Finished fetching tier zero nodes for domain 666521"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":461.377938,"time":"2023-09-05T12:38:53.939478106Z","message":"Finished fetching tier zero nodes for domain 666527"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":425.074423,"time":"2023-09-05T12:38:53.963728541Z","message":"Finished fetching tier zero nodes for domain 666529"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":931.781643,"time":"2023-09-05T12:38:54.099568645Z","message":"Finished fetching tier zero nodes for domain 666518"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":779.348279,"time":"2023-09-05T12:38:54.313418571Z","message":"Finished fetching tier zero nodes for domain 666528"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":2.274806,"time":"2023-09-05T12:38:54.363770054Z","message":"Finished tagging Azure Tier Zero"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":19.519361,"time":"2023-09-05T12:38:54.383311689Z","message":"Finished deleting transit edges"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":161759.412327,"time":"2023-09-05T12:41:36.146508918Z","message":"DCSync Post Processing"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":11.012606,"time":"2023-09-05T12:41:36.157583499Z","message":"Finished deleting transit edges"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":0.010505,"time":"2023-09-05T12:41:36.161028514Z","message":"Azure User Role Assignments Post Processing"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":0.005382,"time":"2023-09-05T12:41:36.164790549Z","message":"AZAddSecret Post Processing"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":0.042426,"time":"2023-09-05T12:41:36.168586434Z","message":"AZExecuteCommand Post Processing"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":0.003614,"time":"2023-09-05T12:41:36.17159373Z","message":"Azure App Role Assignments Post Processing"}
bloodhound-bloodhound-1  | {"level":"debug","time":"2023-09-05T12:41:36.171624562Z","message":"Relationships deleted before post-processing:"}
bloodhound-bloodhound-1  | {"level":"debug","time":"2023-09-05T12:41:36.171635827Z","message":"Relationships created after post-processing:"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":32.336913,"time":"2023-09-05T12:41:36.203981833Z","message":"Asset Group Isolation Collections"}
bloodhound-bloodhound-1  | {"level":"info","time":"2023-09-05T12:41:36.203992047Z","message":"Started Data Quality Stats Collection"}
bloodhound-bloodhound-1  | {"level":"info","elapsed":14936.07487,"time":"2023-09-05T12:41:51.140074994Z","message":"Successfully Completed Data Quality Stats Collection"}
bloodhound-bloodhound-1  | {"level":"error","time":"2023-09-05T12:41:51.140093705Z","message":"Analysis failed: Collected errors:\n\tError 0: error during ad post: traversal required more memory than allowed - Limit: 1024.00 MB - Memory In-Use: 1112.03 MB\n"}

Thanks

[jbfuzier]

Sorry, This issue is related to the new bloodhound CE.

[executionByFork]

This seems to still be an issue in BH 4.3.1 (for posterity I am running it on Kali and using the bloodhound-python ingestor).

In the node graph, many green user nodes show the SID instead of the account name, and the "Node Info" tab does not contain the data from the JSON file. In fact, when you click on a user node which does have info correctly populated, any of the incomplete "SID" nodes you click on after this will show the previous user node's info. Likely some sort of caching issues, because old values aren't erased?

For example, I click on a green node labeled John@Domain.com, and then if I go and click on a user node labeled SID-1-5-21-###, in the Node Info tab, it will still show John@Domain.com, along with John's SID and other information. This is very confusing.

If I grep the JSON user file I uploaded to bloodhound for SID-1-5-21-### all of the proper data is there. It's just not being loaded into Neo properly, or maybe BH GUI isn't reading from Neo properly.

I'm not sure why this issue is closed. From comments above it doesn't look like it was addressed last year. Can we leave this open since it is an ongoing bug? Or, if this is being tracked in another issue can we link to that issue here before closing please?

Silly me. This repo is getting archived soon. I will try installing Bloodhound CE from the repo linked in the readme