Open Albert-W opened 1 year ago
Welcome to HashiCorp Vault, and thanks for filing this issue. Would you please provide us with the Vault configuration and and steps to reproduce the issue. Also, I am wondering if you could verify that Cassandra was reachable at the time of the issue?
Hi @hghaf099, Here is my user case. I have a Cassandra cluster running in an AWS account with a public load-balancer. And I have a vault running in a different AWS account under an api-gateway. we get this error when we are trying to execute a similar command like:
vault write database/config/my-cassandra-database \
plugin_name="cassandra-database-plugin" \
hosts=<host> \
port=<port> \
protocol_version=4 \
username=<username> \
password=<password>\
allowed_roles=* \
consistency=LOCAL_QUORUM \
connect_timeout=30s
It works well with vault 1.6.0, but failed with vault 1.12.2 and 1.13.0
Can you change the connect timeout to 60 or 90s to see if that helps at all?
Sure, I update it into 3000s, but still failed.
Error writing data to database/config/yichang-cassandra-database: context deadline exceeded
using vault monitor -log-level=debug, get the following message
2023-05-10T15:34:45.327Z [DEBUG] secrets.database.database_f4c1337b: got database plugin instance: type=cassandra
only one line related.
Try adding verify_connection=false
to your
vault write database/config/my-cassandra-database \
If that makes it "work", you will have proved that the problem is that Vault is unable to successfully connect to your Cassandra database, and you will have to debug your network setup and Cassandra to determine why that is the case.
the connection is definitely working, because I can read the creds.
Key Value
--- -----
lease_id database/creds/my-role/GeSBplzOxjuknJzvNo3J0srr
lease_duration 8m
lease_renewable true
password oWToydMwaZc1qD-9rrTn
username v_root_my_role_8oxsqvg6ytjlknq1khrq_xxxxx
but the error persists when I update the configuration.
Add "verify_connection=false" will improve the configuration, but I will get an error when read roles.
sh-4.2$ vault read database/creds/my-role
Error reading database/creds/my-role: context deadline exceeded
OK, so, the connection was working using the configuration that had previously been set in Vault:
the connection is definitely working, because I can read the creds.
Key Value --- ----- lease_id database/creds/my-role/GeSBplzOxjuknJzvNo3J0srr lease_duration 8m lease_renewable true password oWToydMwaZc1qD-9rrTn username v_root_my_role_8oxsqvg6ytjlknq1khrq_xxxxx
but the error persists when I update the configuration.
But the new configuration you were trying to apply is in some way broken, such that Vault times out connecting to Cassandra when using it.
Add "verify_connection=false" will improve the configuration, but I will get an error when read roles.
By adding verify_connection=false
you were able to update the configuration...
sh-4.2$ vault read database/creds/my-role Error reading database/creds/my-role: context deadline exceeded
... and so now, generating new credentials no longer works, because Vault times out connecting to Cassandra.
Everything you've shared is pointing at the configuration you're trying to set in Vault being incorrect. You need to investigate that. It doesn't look like an issue with Vault itself.
Thank you, @maxb - I agree completely. This may be a question better suited to our Discuss forum. @Albert-W, please consider closing this issue and posting it there. Thanks! :)
Let me put it in a sequence, so that it's easy to understand.
vault write database/config/yichang-cassandra-database \
plugin_name="cassandra-database-plugin" \
hosts="cassandra-us-east-1.cerberus.io" \
port=9042 \
protocol_version=4 \
username="vault" \
password="ioWN9mdYcDbh9y39EviEriX/JmO7rOAqxxxxxxxxxx" \
allowed_roles=my-role \
consistency="LOCAL_QUORUM" \
connect_timeout="60s" \
tls=true \
pem_json="{\"ca_chain\":$ca_chain}" \
Error writing data to database/config/yichang-cassandra-database: context deadline exceeded
, but the configuration is created. vault read database/roles/my-role
vault write database/config/yichang-cassandra-database \
plugin_name="cassandra-database-plugin" \
hosts="cassandra-us-east-1.cerberus.io" \
port=9042 \
protocol_version=4 \
username="vault" \
password="ioWN9mdYcDbh9y39EviEriX/JmO7rOAqxxxxxxxxxx" \
allowed_roles=my-role \
consistency="LOCAL_QUORUM" \
connect_timeout="70s" \
tls=true \
pem_json="{\"ca_chain\":$ca_chain}" \
same thing happens.
The point is that the configuration is working, but the command returns an error.
The best guess I can make with this information is that the connection to cassandra is working, but the return trip to report that to Vault is not. I would start by doing some packet tracing and other network troubleshooting for connections to and from your Vault to your cassandra instance.
The same code has being running for a while (more than a year). When I am using vault 1.6.0, all is fine, when I upgrade vault to 1.12.2 or 1.13.0, it begins to fail. I tested this for a couple times.
Okay, thanks for that info. I'll check with some folks who are more experienced with cassandra. :)
There's something really odd going on here... If you're still seeing the error when writing the config with verify_connection=false, then that operation isn't connecting to Cassandra at all. It should be just a simple write to Vault storage.
With that extra information, my understanding of the problem completely changes... The problem is, it's now firmly into "this shouldn't be possible" territory.
The only things I can think of to suggest are general "something weird is happening" debugging options:
Hi @maxb , thanks, set verify_connection=false
will succeed.
without the setting, I dump the stacks, I get the traces
2023-05-11T08:08:07.383Z [DEBUG] secrets.database.database_0e42fb14: created database object: name=yichang-cassandra-database plugin_name=cassandra-database-plugin
2023-05-11T08:08:42.321Z [DEBUG] secrets.database.database_0e42fb14: got database plugin instance: type=cassandra
2023-05-11T08:09:00.055Z [DEBUG] core.cluster-listener: performing server cert lookup
2023-05-11T08:09:00.122Z [DEBUG] core.request-forward: got request forwarding connection
the creds generated by it is working.
When reading the creds, it will first fail with "context deadline exceeded", but it can by fixed by vault write -force /sys/leases/revoke-force/database/creds/my-role
. Here are the full commands.
sh-4.2$ vault read database/creds/my-role
Error reading database/creds/my-role: context deadline exceeded
sh-4.2$ vault write -force /sys/leases/revoke-force/database/creds/my-role
Success! Data written to: sys/leases/revoke-force/database/creds/my-role
sh-4.2$ vault read database/creds/my-role
Key Value
--- -----
lease_id database/creds/my-role/TBYEhJ0Wm6O20jXbCxTksik8
lease_duration 8h
lease_renewable true
password PkZjRaHRylIB47-UCtWL
username v_root_my_role_tocakq3ip9ris7kzlbbs_1683794198
Hi @maxb , thanks, set
verify_connection=false
will succeed. without the setting, I dump the stacks, I get the traces
This seems to contradict what you said earlier.
I am sorry, but due to too much conflicting information given, I no longer have any idea what the actual problem is, and don't expect to be able to help further.
sorry for the confusion. When I paste the commands, I accidentally paste the wrong command, here is the updated version: https://github.com/hashicorp/vault/issues/20169#issuecomment-1542706527
Describe the bug A clear and concise description of what the bug is.
Vault is logging the below error:
Vault configuration
Expected behavior Expected the action to be success.
Environment: vault 1.13.0 Failed. vault 1.12.2 Failed. vault 1.6.0 Succeed.