search

NREL / api-umbrella

Open source API management platform
MIT License
2.03k stars 325 forks source link

Elasticsearch index name #325

Open mafiu opened 7 years ago

mafiu commented 7 years ago

Dear @GUI ,

We're using API-Umbrella in 2 separate environments (test/prod) but with the same ES server,

Is there any way to change the index name according to a variable, I tried changed in $API_HOME/embedded/apps/core/current/config/elasticsearch_templates.json $API_HOME/embedded/apps/core/current/src/api-umbrella/proxy/jobs/elasticsearch_setup.lua $API_HOME/embedded/apps/core/current/src/api-umbrella/http-api/health.lua $API_HOME/embedded/apps/core/current/src/api-umbrella/web-app/app/models/log_search/elastic_search.rb $API_HOME/embedded/apps/core/current/templates/etc/rsyslog.conf.mustache

But I found some mappings problems on some variables that I can't find. Do you know if there is other file or binary that uses the index name api-umbrella-logs?

Thanks

GUI commented 7 years ago

Hm, it seems like you've found all the correct files where this is referenced. Do you have a specific error you're getting?

But more generally, I do like the idea of making this configurable. As you've seen, currently, there's not a single variable where this is controlled. However, it seems like as long was we track down all the places where this is referenced (which you seem to have at least gotten a great start on), then this should be pretty easy to turn into a configurable variable.

Since you seem to have found all the right files, my one thought without seeing the specific errors you're getting is that this might be related to the config/elasticsearch_templates.json file. In addition to modifying the template matcher (eg, changing "api-umbrella-logs-v1-*" to something like "api-umbrella-logs-prod-v1-*"), I think you may also need to change the "id" of the template so it's unique for each environment (so changing "api-umbrella-log-v1-template" to something like "api-umbrella-log-prod-v1-template"). Then you'll also need to make sure your naming convention in the template file matches all the aliases defined in src/api-umbrella/proxy/jobs/elasticsearch_setup.lua.

mafiu commented 7 years ago

I'm using Kibana for best analytics viewing and some months I can't look at events correctly.

I found that some months the request_at field is with a different type that the correct one in the template:

"request_at": {
   "type": "date",
   "format": "strict_date_optional_time||epoch_millis"
}

But I can't figure out why sometimes is creating with this type: data:

"request_at": {
   "type": "long"
}