Open chriswue opened 1 year ago
So, since I'm running an Azure cloud instance, I managed to get it to work by creating a hosts.conf
with the following content:
elastic {
events {
hosts: [{host: 'localhost', port: 9200}]
args {
cloud_id: '<cloud id of deployment>'
}
}
workers {
hosts: [{host: 'localhost', port: 9200}]
args {
cloud_id: '<cloud id of deployment>'
}
}
}
One needs to make sure to escape =
as \=
- the cloud_id
contains a base64 encoded string that ends in =
and the config file parser treats those as special characters.
While this solved my immediate problem, it doesn't fix the more general case that may need an actual host name configured.
Hi @chriswue, when using the environment variables, did you see Using override elastic host...
in the apiserver log?
Yes, all those messages appeared. I also patched the server init python file to dump out extra information and the correct ES host was being passed in.
The error you attched for that case says Name or service not known
which basically indicated the apiserver could not reach that address...?
No, what I wrote is that using this constructor doesn't work (which is what ClearML is doing):
es = Elasticsearch(hosts=[{"host":'https://<myinstance>.azure.elastic-cloud.com', "port": 443}]
While this works:
es = Elasticsearch(hosts=['https://<myinstance>.azure.elastic-cloud.com:443']
Tested within the same ClearML container
OK, but since we're doing it always in the same way, you're saying it works for the default values we use but not when you override them?
Looks like it, here is the cleanest reproducer I can come up with:
Possibly a bug in the elastic SDK? 7.13.3 is now more than 2 years old, might be time upgrade to 8.8?
Describe the bug
I've tried to connect ClearML to a standalone elasticsearch instance but it's refusing to connect.
To reproduce
Run the
clearml
container withapiserver
command and configure ES via the environment variablesCLEARML_ELASTIC_SERVICE_HOST
,CLEARML_ELASTIC_SERVICE_PORT
,CLEARML_ELASTIC_SERVICE_USERNAME
,CLEARML_ELASTIC_SERVICE_PASSWORD
.The result is always:
So to debug this I started the
clearml
container with a shell and ranpython3
Now this works (below is the Python REPL output):However after looking how ES is configured in clearml, it seems like it's passing an array of dictionaries rather than an array of strings and this fails:
Trying to set
hosts
config to['myhost:port']
yields in the following error:Looking at the ES python SDK documentation doesn't really reveal much aside from the fact that none of the examples I've seen anywhere passes a structure like this
[{"host": "<host>", "port": 123}]
. All examples only ever pass lists of strings like['host1:port', 'host2:port']
. So I don't know where the problem really is but the current implementation doesn't work.Expected behaviour
I should be able to connect to a non-standard (i.e. not localhost) ES instance.
Environment