3.0.0-beta.1: Error when creating a new user

[funkyfuture]

i get the following error logged when a user is created, yet there's no error message in the client.

the first messages that log a 404 are weird, as i get a 200 when using curl:

# curl http://elasticsearch:9200/kitodo
{"kitodo":{"aliases":{},"mappings":{"client":{"properties":{"clientName":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"user":{"properties":{"active":{"type":"boolean"},"clients":{"properties":{"id":{"type":"long"}}},"ldapLogin":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"location":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"login":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"metadataLanguage":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"surname":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}},"settings":{"index":{"creation_date":"1542026126782","number_of_shards":"5","number_of_replicas":"1","uuid":"GN9t_WtYRfSQtRCChQXm-w","version":{"created":"5061199"},"provided_name":"kitodo"}}}}

these are the logged messages:

[ERROR] 2018-11-12 12:34:51.110 [http-nio-8080-exec-10] LazyDTOModel - GET http://elasticsearch:9200/kitodo: HTTP/1.1 404 Not Found
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"kitodo","index_uuid":"_na_","index":"kitodo"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"kitodo","index_uuid":"_na_","index":"kitodo"},"status":404}
[ERROR] 2018-11-12 12:34:54.564 [http-nio-8080-exec-8] LazyDTOModel - GET http://elasticsearch:9200/kitodo: HTTP/1.1 404 Not Found
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"kitodo","index_uuid":"_na_","index":"kitodo"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"kitodo","index_uuid":"_na_","index":"kitodo"},"status":404}
[ERROR] 2018-11-12 12:34:54.609 [http-nio-8080-exec-8] LazyDTOModel - GET http://elasticsearch:9200/kitodo: HTTP/1.1 404 Not Found
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"kitodo","index_uuid":"_na_","index":"kitodo"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"kitodo","index_uuid":"_na_","index":"kitodo"},"status":404}
[ERROR] 2018-11-12 12:34:54.620 [http-nio-8080-exec-8] LazyDTOModel - GET http://elasticsearch:9200/kitodo: HTTP/1.1 404 Not Found
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"kitodo","index_uuid":"_na_","index":"kitodo"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"kitodo","index_uuid":"_na_","index":"kitodo"},"status":404}
elasticsearch_1  | [2018-11-12T12:35:26,786][INFO ][o.e.c.m.MetaDataCreateIndexService] [f50ytHB] [kitodo] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
elasticsearch_1  | [2018-11-12T12:35:27,721][INFO ][o.e.c.m.MetaDataMappingService] [f50ytHB] [kitodo/GN9t_WtYRfSQtRCChQXm-w] create_mapping [client]
[ERROR] 2018-11-12 12:35:28.365 [http-nio-8080-exec-6] LazyDTOModel - org.elasticsearch.client.ResponseException: GET http://elasticsearch:9200/kitodo/user/_search?pretty=true&from=0&size=10: HTTP/1.1 400 Bad Request
{
  "error" : {
    "root_cause" : [
      {
        "type" : "query_shard_exception",
        "reason" : "No mapping found for [login] in order to sort on",
        "index_uuid" : "GN9t_WtYRfSQtRCChQXm-w",
        "index" : "kitodo"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "kitodo",
        "node" : "f50ytHBmT3CCrtmxTYK8EA",
        "reason" : {
          "type" : "query_shard_exception",
          "reason" : "No mapping found for [login] in order to sort on",
          "index_uuid" : "GN9t_WtYRfSQtRCChQXm-w",
          "index" : "kitodo"
        }
      }
    ]
  },
  "status" : 400
}
mproduction_1     |
elasticsearch_1  | [2018-11-12T12:36:28,755][INFO ][o.e.c.m.MetaDataMappingService] [f50ytHB] [kitodo/GN9t_WtYRfSQtRCChQXm-w] create_mapping [user]
[ERROR] 2018-11-12 12:36:29.076 [http-nio-8080-exec-7] LazyDTOModel - org.elasticsearch.client.ResponseException: GET http://elasticsearch:9200/kitodo/user/_search?pretty=true&from=0&size=10: HTTP/1.1 400 Bad Request
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Fielddata is disabled on text fields by default. Set fielddata=true on [login] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "kitodo",
        "node" : "f50ytHBmT3CCrtmxTYK8EA",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "Fielddata is disabled on text fields by default. Set fielddata=true on [login] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
        }
      }
    ]
  },
  "status" : 400
}

[beatrycze-volk]

This one is already addressed. Mistake comes from creating own index by CURL command instead of creating by click Create Mapping button in the application. You can read more here: https://github.com/kitodo/kitodo-production/pull/1834

[funkyfuture]

neither did i create an index w/ curl, nor would i know something about that button. is the patch you linked to included in the beta 1?

[beatrycze-volk]

Yes, it is in the beta 1.

How did you create index if not by the button on the indexing page nor by CURL command?

[funkyfuture]

How did you create index […] ?

not at all. it seems that the software did it, which is reasonable.

1) start up a fresh stack of the components 2) curl http://elasticsearch:9200/kitodo yields an index_not_found_exception 3) request the top level page 4) as 2) 5) login as testAdmin 6) as 2) 7) create a 'Mandant' 8) curl … yields {"kitodo":{"aliases":{},"mappings":{"client":{"properties":{"clientName":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}},"settings":{"index":{"creation_date":"1542032508035","number_of_shards":"5","number_of_replicas":"1","uuid":"NPERN3oCQsWGU8t4Ys1zDA","version":{"created":"5061199"},"provided_name":"kitodo"}}}} 9) try to create a user 10) curl … yields {"kitodo":{"aliases":{},"mappings":{"client":{"properties":{"clientName":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"user":{"properties":{"active":{"type":"boolean"},"clients":{"properties":{"id":{"type":"long"}}},"ldapLogin":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"location":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"login":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"metadataLanguage":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"surname":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}},"settings":{"index":{"creation_date":"1542032508035","number_of_shards":"5","number_of_replicas":"1","uuid":"NPERN3oCQsWGU8t4Ys1zDA","version":{"created":"5061199"},"provided_name":"kitodo"}}}}

at this point, the button to create the ES mapping is disabled.

now that i learned about that button, and apply it as very first action, i come to wonder why this action has to be invoked manually. (user creation works by this path.)

[beatrycze-volk]

So here is the answer. You use automatic creation of index, what in the most of the cases is wrong. It is necessary first to create correct mapping. We can't depend on automatic creation, as you can see, it causes errors.

[funkyfuture]

the software could prevent this by setting up the mapping itself.

[Erikmitk]

We had a similar discussion on https://github.com/kitodo/kitodo-production/pull/1834. I agree that the user should not worry about mapping creation at all. If you want to add data and there's no mapping in the index yet it should be created automatically as Kitodo.Production sees it fit.

In case we don't want that we should at least create an explicit documentation like First Steps with a new Installation or something to highlight all the things necessary to get it up and running the way we suggest.

What do you think @Kathrin-Huber?

[Kathrin-Huber]

As this issue is no longer related to the initial problem I am going to close it. If you have any concerns about the handling of the index, please feel free to open a new issue.