godocean
commented
1 year ago I met the same issue as you, not sure how to fix it.
Closed LilMonk closed 9 months ago
godocean
commented
1 year ago I met the same issue as you, not sure how to fix it.
cccadet
commented
1 year ago Same here.
shicholas
commented
1 year ago Same here
github-actions[bot]
commented
1 year ago This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io
Gerrit-K
commented
1 year ago Not stale, same issue here with similar setup.
For the record, because I thought it just didn't pick up my environment variables, I tried these variants:
ELASTICSEARCH_SSL_TRUSTSTORE_FILE (as above, from application.yml)ELASTICSEARCH_SSLCONTEXT_TRUSTSTOREFILE (hoping that we could set spring properties such as elasticsearch.sslContext.trustStoreFile (see here) using environment variables by replacing dots with _ and transforming everything to uppercase)ELASTIC_CLIENT_SSL_TRUST_STORE_FILE (from frontend docker.env/application.conf, probably not applicable to the upgrade job)I could confirm by trial & error that the first two variable variants are indeed picked up, but the client doesn't accept the self-signed certificate from elasticsearch.
wei-jiang-dns53
commented
1 year ago I run a Nginx HTTP proxy in front of elasticsearch to mitigate this issue
ozmoze
commented
12 months ago I had the same issue using datahub-gms:v0.12.0 along with elasticsearch 8.10.4.
After having tried most of elasticsearch environment variables combinations suggested by @LilMonk and @Gerrit-K, I finally got it to work by setting JAVA_OPTS env variable in datahub-gms section.
datahub-gms:
enabled: true
/// truncated ///
extraEnvs:
- name: ELASTICSEARCH_SSL_PROTOCOL
value: SSL
- name: ELASTICSEARCH_SSL_TRUSTSTORE_TYPE
value: PKCS12
- name: ELASTICSEARCH_SSL_TRUSTSTORE_FILE
value: /elastic-certificates/truststore-elastic.p12
- name: ELASTICSEARCH_SSL_TRUSTSTORE_PASSWORD
valueFrom:
secretKeyRef:
name: elasticsearch-certs
key: truststore.password
- name: JAVA_OPTS
value: -Djavax.net.ssl.trustStore=$(ELASTICSEARCH_SSL_TRUSTSTORE_FILE) -Djavax.net.ssl.trustStoreType=$(ELASTICSEARCH_SSL_TRUSTSTORE_TYPE) -Djavax.net.ssl.trustStorePassword=$(ELASTICSEARCH_SSL_TRUSTSTORE_PASSWORD)Definitively something that should be addressed indatahub helm chart.
Gerrit-K
commented
11 months ago Thanks @ozmoze, that was a good hint. Although, it didn't work for me right away, since the JAVA_OPTS variable (contrary to my prior belief) isn't a standardized option picked up by the JVM (see this comment). It's just often used in scripts, including the GMS startup script, but unfortunately not the datahub-system-update-job that I was testing the connection with.
However, the variable JDK_JAVA_OPTIONS is a standardized option and is automatically picked up by the JVM. And with this, I was finally able to get it to work with these values:
Needless to say, this is waaay too much logic for a Helm "values" file and should be integrated in the Helm chart.
LilMonk
commented
11 months ago I have ended up using Istio service mesh with mtls. So even though all the components are using HTTP but using the envoy proxy the communication between the services is done on HTTPS.
github-actions[bot]
commented
10 months ago This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io
github-actions[bot]
commented
9 months ago This issue was closed because it has been inactive for 30 days since being marked as stale.
fcecagno
commented
3 months ago Just used instructions on https://github.com/datahub-project/datahub/issues/8433#issuecomment-1843072352 to be able to use Elasticsearch, it should definitely be part of the chart to avoid so many configurations just to allow using a custom CA.
Describe the bug I'm trying to set up datahub on Kubernetes. My environment contains Postgres, Kafka (Strimzi), and ElasticSearch (Using the official elastic search operator). I have enabled TLS/SSL on Kafka and ElasticSearch. I'm able to connect to Kafka but not to ElasticSeach.
This is my YAML:
The error log:
To Reproduce Steps to reproduce the behavior:
ES_HOME=
pwdES_CRTS=$ES_HOME/crts ES_CLUSTER_NAMESPACE=elasticsearch ES_PASSWORD=es_password TARGET_NAMESPACE=data-hubcd $ES_HOME
echo "Current working dir:
pwd"makeCrtsDir () { [ -d foo ] || mkdir -p $ES_CRTS }
clearCrts () { rm -r $ES_CRTS makeCrtsDir }
getCrts () { kubectl get secret elasticsearch-es-http-certs-public \ --namespace=$ES_CLUSTER_NAMESPACE \ --output=go-template='{{index .data "ca.crt" | base64decode }}' \
}
createCrtsStore () { openssl pkcs12 -export \ -in $ES_CRTS/tls.crt \ -inkey $ES_CRTS/tls.key \ -out $ES_CRTS/keystore.p12 \ -name elasticsearch \ -CAfile $ES_CRTS/ca.crt \ -caname elasticsearch \ -password pass:$ES_PASSWORD
}
createSecretAtTarget() { kubectl create secret generic elasticsearch-user-certs \ --from-file=$ES_CRTS/keystore.jks \ --from-file=$ES_CRTS/truststore.jks \ --from-literal=keystore.password=$ES_PASSWORD \ --from-literal=truststore.password=$ES_PASSWORD \ --namespace=$TARGET_NAMESPACE }
makeCrtsDir clearCrts getCrts createCrtsStore createSecretAtTarget