Disable query timeout and memory protections from Cypher not working

[rphlwnk]

Description:

I tried running the default cipher Shortest paths to systems trusted for unconstrained delegation against my data set but it did not finish, in the log i can see errors regarding dbms.timeout and api error - i tried increasing it in Neo4j but it still does not finish the cipher. I also tried with Docker option bhe_disable_cypher_qc=true but same outcome.

Component(s) Affected:

• [ ] UI
• [x] API
• [x] Neo4j
• [ ] PostgreSQL
• [ ] Data Collector (SharpHound, AzureHound)
• [ ] Other (tooling, documentation, etc.)

Actual Behavior:

Cipher ends with 'An error occured' and the following error log lines:

bloodhound  | {"level":"info","query":"match p = shortestPath((n)-[:Owns|GenericAll|GenericWrite|WriteOwner|WriteDacl|MemberOf|ForceChangePassword|AllExtendedRights|AddMember|HasSession|Contains|GPLink|AllowedToDelegate|TrustedBy|AllowedToAct|AdminTo|CanPSRemote|CanRDP|ExecuteDCOM|HasSIDHistory|AddSelf|DCSync|ReadLAPSPassword|ReadGMSAPassword|DumpSMSAPassword|SQLAdmin|AddAllowedToAct|WriteSPN|AddKeyCredentialLink|SyncLAPSPassword|WriteAccountRestrictions*1..]->(m:Computer)) where m.unconstraineddelegation = $STRIPPED and n <> m return p","time":"2023-09-18T15:05:32.439571861Z","message":"Executing user cypher query"}
bloodhound  | {"level":"warn","time":"2023-09-18T15:06:04.268603391Z","message":"Writing API Error. Status: 500. Message: [{ driver error: Neo4jError: Neo.ClientError.Transaction.TransactionTimedOut (The transaction has been terminated. Retry your operation in a new transaction, and you should see a successful result. The transaction has not completed within the specified timeout (dbms.transaction.timeout). You may want to retry with a longer timeout. ) - query: match p = shortestPath((n)-[:Owns|GenericAll|GenericWrite|WriteOwner|WriteDacl|MemberOf|ForceChangePassword|AllExtendedRights|AddMember|HasSession|Contains|GPLink|AllowedToDelegate|TrustedBy|AllowedToAct|AdminTo|CanPSRemote|CanRDP|ExecuteDCOM|HasSIDHistory|AddSelf|DCSync|ReadLAPSPassword|ReadGMSAPassword|DumpSMSAPassword|SQLAdmin|AddAllowedToAct|WriteSPN|AddKeyCredentialLink|SyncLAPSPassword|WriteAccountRestrictions*1..]->(m:Computer)) where m.unconstraineddelegation = true and n <> m return p}]"}
bloodhound  | {"level":"warn","time":"2023-09-18T15:06:04.268622873Z","message":"Writing API Error. Context Deadline Exceeded while writing JSON response."}

Screenshots/Code Snippets/Sample Files:

Current dataset volume

Users | 4 741 Groups | 8 082 Computers | 2 958 OUs | 451 GPOs | 167 Containers | 69 Domains | 5 Sessions | 1 690 ACLs | 224 903 Relationships | 351 346

Environment Information:

BloodHound: Bloodhound Docker image with tag latest

Collector: [SharpHound version / AzureHound version]

OS: Ubuntu Server LTS 22.04

Database (if persistence related): Neo4j version 4.4

Docker (if using Docker): 24.0.6, build ed223bc

Additional Information:

I also tried increasing the dbms.timeout with /config overwrite of Neo4J (mounted config file to /conf of Neo4J Container)

dbms.transaction.timeout=2m
dbms.lock.acquisition.timeout=2m

Potential Solution (Optional):

If you have any ideas about what might be causing the issue or how it could be fixed, you can share them here.

Related Issues:

If you've found related issues in the project's issue tracker, mention them here.

Contributor Checklist:

• [x] I have searched the issue tracker to ensure this bug hasn't been reported before or is not already being addressed.
• [ ] I have provided clear steps to reproduce the issue.
• [x] I have included relevant environment information details.
• [x] I have attached necessary supporting documents.
• [x] I have checked that any JSON files I am attempting to upload to BloodHound are valid.

[CatzCc]

Same issue. Verified that this is not neo4j problem - the query which times out in BH will respect timeout setting in neo4j and finish successfully if run directly on the database. @StephenHinck Could you please advise a workaround for now?

[ag-michael]

I have confirmed this as well. Best I can tell, BH is getting a transaction timeout message in the Neo4j response:

https://github.com/SpecterOps/BloodHound/blob/331c298d3086c8ff05fb5f38910464dbfb6a1f69/packages/go/dawgs/util/errors.go#L60

I made sure in Neo4j that dbms.transaction.timeout is set to 0s (using CALL dbms.listConfig()). And when running the same query in the Neo4j user interface I get results just fine with the message Started streaming 125 records after 21 ms and completed after 204409 ms. for the test query I've been using.

The same query using the Cypher query search box in the BH-CE UI, gets me this in BH container log:

{"level":"info","query":"match p = shortestPath((m:AZUser)-[r:AZAvereContributor|AZContains|AZContributor|AZGetCertificates|AZGetKeys|AZGetSecrets|AZHasRole|AZMemberOf|AZOwner|AZRunsAs|AZVMContributor|AZAutomationContributor|AZKeyVaultContributor|AZVMAdminLogin|AZAddMembers|AZAddSecret|AZExecuteCommand|AZGlobalAdmin|AZPrivilegedAuthAdmin|AZGrant|AZGrantSelf|AZPrivilegedRoleAdmin|AZResetPassword|AZUserAccessAdministrator|AZOwns|AZCloudAppAdmin|AZAppAdmin|AZAddOwner|AZManagedIdentity|AZAKSContributor|AZNodeResourceGroup|AZWebsiteContributor|AZLogicAppContributor|AZMGAddMember|AZMGAddOwner|AZMGAddSecret|AZMGGrantAppRolesAZMGGrantRole*1..]->(n)) where n.system_tags = $STRIPPED and n.name =~ $STRIPPED and m <> n return p","time":"2024-01-16T18:44:00.540762455Z","message":"Executing user cypher query"}
{"level":"warn","time":"2024-01-16T18:44:30.929439377Z","message":"Writing API Error. Status: 500. Message: [{ driver error: Neo4jError: Neo.ClientError.Transaction.TransactionTimedOut (The transaction has been terminated. Retry your operation in a new transaction, and you should see a successful result. The transaction has not completed within the specified timeout (dbms.transaction.timeout). You may want to retry with a longer timeout. ) - query: match p = shortestPath((m:AZUser)-[r:AZAvereContributor|AZContains|AZContributor|AZGetCertificates|AZGetKeys|AZGetSecrets|AZHasRole|AZMemberOf|AZOwner|AZRunsAs|AZVMContributor|AZAutomationContributor|AZKeyVaultContributor|AZVMAdminLogin|AZAddMembers|AZAddSecret|AZExecuteCommand|AZGlobalAdmin|AZPrivilegedAuthAdmin|AZGrant|AZGrantSelf|AZPrivilegedRoleAdmin|AZResetPassword|AZUserAccessAdministrator|AZOwns|AZCloudAppAdmin|AZAppAdmin|AZAddOwner|AZManagedIdentity|AZAKSContributor|AZNodeResourceGroup|AZWebsiteContributor|AZLogicAppContributor|AZMGAddMember|AZMGAddOwner|AZMGAddSecret|AZMGGrantAppRolesAZMGGrantRole*1..]->(n)) where n.system_tags = \"admin_tier_0\" and n.name =~ '(?i)Global Administrator.*' and m <> n return p}]"}
{"level":"warn","time":"2024-01-16T18:44:30.929471278Z","message":"Writing API Error. Context Deadline Exceeded while writing JSON response."}
{"level":"info","remote_addr":"172.18.0.1:33676","proto":"HTTP/1.1","referer":"https://bloodhound/ui/explore","user_agent":"","request_id":"","request_bytes":736,"response_bytes":23,"status":200,"elapsed":9223372036854.775,"time":"2024-01-16T18:44:30.92991058Z","message":"POST /api/v2/graphs/cypher"}

My conclusion at this point is that BH is somehow overriding the dbms.transaction.timeout when making queries.

Please help me resolve this.

[jrlane]

I'm also experiencing this. Anyone figured out a work around yet?

[StephenHinck]

The BloodHound Engineering team has this issue on our plate for implementation soon!