search

FusionAuth / fusionauth-issues

FusionAuth issue submission project
https://fusionauth.io
90 stars 12 forks source link

[Bug]: Very large amounts of data in the user.data field will cause indexing failures in elasticsearch #2675

Closed lyleschemmerling closed 3 months ago

lyleschemmerling commented 3 months ago

What happened?

Seen in a system where a bug caused an excessive amount of data to be loaded into the user data field. If too many fields are loaded into the object Elasticsearch will reject the document as it is unable to index it with the following error:

java.lang.IllegalArgumentException: Limit of total fields [1000] in index [fusionauth_user] has been exceeded
    at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:614) ~[elasticsearch-7.6.1.jar:7.6.1]

(from fusionauth-search.log)

If any users get into this state attempting to reindex all users will fail. We should add validation to prevent users with large amounts of data from being loaded to prevent this.

Version

1.47.0

Affects Versions

all

mooreds commented 3 months ago

how much data @lyleschemmerling ?

lyleschemmerling commented 3 months ago

It's in the error. The ES limit is 1000 total fields

lyleschemmerling commented 3 months ago

closing in favor of https://github.com/FusionAuth/fusionauth-issues/issues/2676