use of unitialized value

[palica]

funtoo-report send
Use of uninitialized value in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.26.1/Funtoo/Report.pm line 49.
your report can be seen at: http://elk2.liguros.net:9200

this is with the latest master code.

[palica]

and it doesn't submit anything to ES

[tejr]

The reason for the uninitialized variable is that there is no location header in the response hash, because the data submission failed.

Having added a little HTTP error-checking to Funtoo::Reporter::send_report() in commits b010379 and 1ed7dae, I get the same response as @palica to form submissions: HTTP/1.1 400 Bad Request, with the following application/json content as the body:

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse [installed-pkgs.pkgs]"}],"type":"mapper_parsing_exception","reason":"failed to parse [installed-pkgs.pkgs]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 5:16"}},"status":400}

The line and column number varies, but always points to the same structural byte of the report:

   ...
   "installed-pkgs" : {
      "pkg-count-total" : 300,
      "pkg-count-world" : 7,
      "pkgs" : {<- this character here
         "other" : {
            "sys-libs/timezonedata" : {
               "2017b" : "0"
            },
            ...

So it looks very much like the server is upset about the opening of the pkgs member object under installed-pkgs, maybe because it is a structural element that it did not expect. Is this something @palica fixes?

[tejr]

OK, now we're getting:

eels funtoo-reporter # perl -Ilib funtoo-report -d send
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [funtoo-2018.10] has been exceeded"}],"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [funtoo-2018.10] has been exceeded"},"status":400}
Failed submission: 400 Bad Request

[haxmeister]

we can break off some part of the report as a separate index if needed. Would just have to create a second %hash and modify the send_report() function to send 2 reports to different indexes

[palica]

in elk. there are systems with >1000 packages so that would probably not help that much. I'll have to look if we can setup ES to have this limit higher by default for all newly created indices.

[haxmeister]

Note that this issue relates to the get_all_pkg structure in the report. Per @palica we need to preserve the work on that structure so we don't lose it, and get the strait array list that we had before into the main branch so we can make a release. We will likely have to re-write it but in it's original form it was a trivial function just to list those dirs and push to array. The release model we want will have the get_all_pkg (in array) structure as well as the world file list (in array) and the original get_pkg_version of those key softwares like before (we can extract this from the last release if it is otherwise lost). The continued work on this issue will need to fall into the new development branch we are intending to implement.

[tejr]

I think we should close this issue (since the original problem for which it was raised is fixed) and raise a new general one, something like "Decide on how to deal with report field limits" or similar. Is this OK?