Closed philippkahr closed 4 years ago
Pinging @elastic/apm-ui (Team:apm)
@Bargs shouldn't this be labeled with Team:KibanaApp
? Apart from being an APM user, this looks like an issue with index patterns or the Index Management app . I'm not sure how the APM app can fix this?
@philippkahr I've reproduced your issue, and now understand the confusion.
Afaict you are inserting fields like http.request.cookies.LASTUSERNAMEINPUT
and are expecting it to become searchable after refreshing the index patterns.
However, since http.request
is mapped as dynamic: false
descendant fields will not get indexed in elasticsearch. Refreshing the index patterns won't change this.
The UI in Discover can be confusing since a warning give the user the impression that clicking the refresh button will solve the problem. It won't.
Solution A Index your additional data as labels. These are dynamic so everything will become indexed and be searchable and aggregatable.
Solution B Update the mapping manually. I see you already suggested that here. I'm curious why that didn't work?
PUT apm*/_mapping/
{
"properties": {
"http.request.cookies.LASTSKIN": {
"type": "keyword",
"index": true
},
"http.request.cookies.LASTUSERNAME": {
"type": "keyword",
"index": true
}
}
}
Hi @sqren
thanks for looking into my matter! Regarding the PUT apm*...
that worked without any issue, so there is no trouble there. I just opened up this issue, cause I wasn't quite sure if this is intended behavior. Maybe the intended behavior is that the Kibana warning bang messages states field is dynamically mapped, change it to true in the fields.yml
. However, I do not like to touch things like a fields.yml since I would then have to update it everytime I install an update. Sure I can template that with ansible, but I would have to ensure that my template fits the version.
The idea with labels is nice if you have access to the source code and can modify it properly as I saw that you need to add it as a dependency to your application and call the labels API https://www.elastic.co/guide/en/apm/agent/java/1.x/public-api.html#api-transaction-add-tag. However, I am more a DevOps / SysOps for bought products, like Jira, Bitbucket, Confluence. We are using APM basically to ensure that a user's claim about something being slow is valid or not.
I am not sure if we should close this issue and generate a new issue with a title like better index mapping warning bang
or keep this one and hook up the right team to it :).
The idea with labels is nice if you have access to the source code and can modify it properly as I saw that you need to add it as a dependency to your application and call the labels API
How did you add http.request.cookies
to the ingested docs? Can't you do the same but add it as labels.cookies
?
I am not sure if we should close this issue and generate a new issue with a title like better index mapping warning bang or keep this one and hook up the right team to it :).
It sounds like there are two different problems:
How did you add
http.request.cookies
to the ingested docs? Can't you do the same but add it aslabels.cookies
?
to be honest, I have no idea where those fields originate from. The java application is running with a tomcat and I have added the apm jar as an -javaagent
to the CATALINA_OPTS
, as shown here https://www.elastic.co/guide/en/apm/agent/java/master/setup-javaagent.html#setup-tomcat-unix , so I guess it must be coming from there? Maybe @felixbarny could explain this. I have apm-server and the java agent up to date. I am using a basic apm-server config, just like it is out of the box.
- Discover shows a misleading warning. That deserves a separate issue
I will open one if that is ok with you?
- It is difficult to search for custom fields in APM docs
That seems to be outta my league. I would gladly help, but I do not know how.
I will open one if that is ok with you?
Please do! :)
That seems to be outta my league. I would gladly help, but I do not know how.
Totally fair. Didn't mean to imply you should do it. For now I'll wait for @felixbarny to chime in on this.
Pinging @elastic/kibana-app (Team:KibanaApp)
@sqren my bad, I saw APM in the title and pulled the trigger without thinking 😬
@felixbarny here you go for a bit more information.
```yaml apm-server: host: "0.0.0.0:8200" rum: enabled: true output.elasticsearch: hosts: ["elastic1:9200"] protocol: "https" username: "fancyuser" password: "fancypassword" ssl.certificate_authorities: ["/etc/ssl/certs/cachain.pem"] pipelines: - pipeline: "apm_match_user_id" monitoring: enabled: true elasticsearch: username: fancyuser password: fancypassword tags: ["itsm-test"] ```
```json PUT _ingest/pipeline/apm_match_user_id { "description" : "map weird user.id to user.id", "processors" : [ { "set": { "field": "user.id", "value": "{{http.request.headers.Vm_user.0}}", "ignore_failure": true } } ] } ```
I had to blackbox some stuff, but I guess it is still better than nothing.
![Discover - Kibana copy](https://user-images.githubusercontent.com/12175559/67335466-08656000-f524-11e9-9135-cecdae1a8025.jpg)
![Discover - Kibana copy 2](https://user-images.githubusercontent.com/12175559/67335467-08fdf680-f524-11e9-8a9d-94502f2be5c5.jpg)
![Discover - Kibana copy 3](https://user-images.githubusercontent.com/12175559/67335468-08fdf680-f524-11e9-9904-e84d54b0d781.jpg)
![Discover - Kibana copy 4](https://user-images.githubusercontent.com/12175559/67335470-08fdf680-f524-11e9-9f4f-61d5918cecf9.jpg)
![Discover - Kibana copy 5](https://user-images.githubusercontent.com/12175559/67335471-08fdf680-f524-11e9-9c83-437a0c527e76.jpg)
![Discover - Kibana copy 6](https://user-images.githubusercontent.com/12175559/67335472-09968d00-f524-11e9-9dc0-d3e493b0944a.jpg)
![Discover - Kibana copy 7](https://user-images.githubusercontent.com/12175559/67335474-09968d00-f524-11e9-8bc8-9037ca2983d5.jpg)
Not sure what the actual question is. I assume it's this?
How did you add
http.request.cookies
to the ingested docs?
Our agents add them automatically by default (controlled by the capture_headers
config option).
Hi @felixbarny,
yes, that was the primary question. Maybe we should change http.request
from dynamic: false
to true
if capture_headers are enabled? I am not quite sure, how this should play out in the end. What is your opinion on that matter?
@philippkahr I've been made aware of an experimental feature append_fields
that might solve your problem.
Adding the following to apm-server.yml
will enable dynamic indexing for http.request.cookies
and it should work across updates:
setup.template.enabled: true
setup.template.overwrite: true
setup.template.append_fields:
- name: http.request.cookies
type: object
dynamic: true
@sqren I deleted my index mapping and appended that to my 7.3.0 and 7.4.0 config and it worked! Thanks for the help. Maybe that should be pointed out in the documentation somewhere?
Should we close this issue now?
I deleted my index mapping and appended that to my 7.3.0 and 7.4.0 config and it worked!
Great to hear!
Maybe that should be pointed out in the documentation somewhere?
I agree. @bmorelli25 can you figure out if this makes sense to document somewhere?
tldr: The Java agent (and possibly other agents) ingests documents with cookie values located in http.request.cookies
. Since http.request
has disabled dynamic indexing, and http.request.cookies
is not declared in the mappings the values in http.request.cookies
are not indexed and thus not searchable. One solution is outlined in https://github.com/elastic/kibana/issues/48674#issuecomment-545342220
Interesting. I could probably add something to our troubleshooting doc with a quick walkthrough. Opened a new issue to track on the docs board: https://github.com/elastic/kibana/issues/49070
Kibana version: 7.3.0 & 7.4.0 Elasticsearch version: 7.3.0 & 7.4.0 Server OS version: Centos 7 Browser version: Chrome Dev 79.0.3941.4 Browser OS version: MacOS Original install method (e.g. download page, yum, from source, etc.): yum repo Describe the bug: I am using the apm-server 7.3.0 and 7.4.0 with the java apm-agent. There is this issue that the index mapping does not get updated when hitting the refresh index pattern for
apm-*
. Steps to reproduce:Expected behavior: Just refresh the fields and make them aggregate and searchable.
Any additional context: This originated at my question here, which is way more detailed: https://discuss.elastic.co/t/apm-index-not-refreshing/195884