solving the total fields limit

[palica]

Dear coders!

I have another issue to discuss. This would help to solve the field limit error.

root@robor ~ # ❯❯❯ funtoo-report -s -d
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [fundev-3.0.0-beta-2018.04.09] has been exceeded"}],"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [fundev-3.0.0-beta-2018.04.09] has been exceeded"},"status":400}
Funtoo::Report::send_report: Failed submission: 400 Bad Request at line 77
 at /usr/bin/funtoo-report line 89.
root@robor ~ # ❯❯❯ curl -XPUT "https://elk.liguros.net:9200/fundev-3.0.0-beta-2018.04.09/_settings" -H 'Content-Type: application/json' -d'
{
  "index.mapping.total_fields.limit": 5000
}'
{"acknowledged":true}#
root@robor ~ # ❯❯❯ funtoo-report -s -d
{"_index":"fundev-3.0.0-beta-2018.04.09","_type":"report","_id":"zALSq2IBo3plbojXleYl","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":1,"_primary_term":1}
your report can be seen at: https://elk.liguros.net:9200/fundev-3.0.0-beta-2018.04.09/report/zALSq2IBo3plbojXleYl

And now the question. Would it make sense to solve this on the application level? Would it be possible to parse the error from ES and if it is "total fields limit reached", call the json payload to update the total_fields.limit and retry with submission of the report?

Does it make sense? Comments?

[haxmeister]

and what exactly does the command to "update total_fields.limit" look like?

[palica]

curl -XPUT "https://elk.liguros.net:9200/fundev-3.0.0-beta-2018.04.09/_settings" -H 'Content-Type: application/json' -d'
{
  "index.mapping.total_fields.limit": 5000
}'
{"acknowledged":true}#

[haxmeister]

It may be possible to calculate the number of fields in a report before it's sent, or after. Is there any information that clarifies what a "field" is. This could help because we could avoid setting the field limit higher than necessary.

Perhaps we can send some reports without certain structures and see how many fields ES claims it to be using?

[palica]

the problem is that the fields are also fields from other reports so if other reports have for example other packages installed they get counted as unique fields and may reach the limit.

On Thu, Apr 12, 2018 at 3:46 AM Joshua S. Day notifications@github.com wrote:

It may be possible to calculate the number of fields in a report before it's sent, or after. Is there any information that clarifies what a "field" is. This could help because we could avoid setting the field limit higher than necessary.

Perhaps we can send some reports without certain structures and see how many fields ES claims it to be using?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/haxmeister/funtoo-reporter/issues/107#issuecomment-380648391, or mute the thread https://github.com/notifications/unsubscribe-auth/ADB9OlVR2gNv49vweDOi-IYFJXCN5KWWks5tnrH_gaJpZM4TNEvQ .

[haxmeister]

Needs POD added to reflect new function. Also unnecessary lines need removed, --debug output possibly, --verbose output possible also. Consideration on the max allowable field limit.

[palica]

Could you turn some kind of debug output when the limit gets bumped, please? with -d|--debug option

Fields limit increased from 1000 => 2000
Fields limit increased from 2000 => 3000
 ...

I would like to see how it is bumping and maybe even show the response from ES like we are doing with --debug for report Thank you.

[palica]

so this, would kind of be the last thing I would like to have in 3.0.0, let's polish it and release :)

[haxmeister]

@palica the associated branch has been merged to develop now. It doesn't hurt to test it more by moving the limit down and trying to make it resubmit. Perhaps move it down really low to see that it will indeed attempt to raise the limit more than once. If everything seems to be working ok, you can close this issue as needed.

[palica]

worked in test, good to merge

[gtlt]

Is it fixed in 3.0.0?

# /usr/bin/funtoo-report -s
field limit error but field limit is already at max 5000 or more at /usr/bin/funtoo-report line 90.

# eix funtoo-report
[I] dev-perl/Funtoo-Report [1]
     Available versions:  2.0.1 3.0.0 **9999 {test}
     Installed versions:  3.0.0(10:17:20 PM 04/28/2018)
     Homepage:            https://github.com/haxmeister/funtoo-reporter
     Description:         Anonymous reporting tool for Funtoo Linux

[palica]

well, yes and no. limits get bumped up to 5000 now automatically. but since we hit those, we might bump that number even higher.

[palica]

let's bump the limit to 10k and see if that works for a while.