Check for error in requests with ENV parameter

[tdipisa]

During last checks with Kibana we found some requests to GeoServer referencing the ENV parameter were generating errors. To check if this requests are still present to be able to proceed with an investigation to verify if these requests are due to a bug in the MS application or simply to a configuration problem (on GeoServer or MapStore).

We need first of all a report of failing requests if any: if the problem is related to a miss configuration in GeoServer we have to fix it or notify the client.

[lpasquali]

@tdipisa @randomorder did you save some output about such requests for me to identify what we are talking about in a quicker way when I will work on it?

[randomorder]

nope

[tdipisa]

@lpasquali do you mean some sample request with the ENV param? If yes, below an example:

https://sit.comune.bolzano.it/geoserver/wms?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=dbt_pub%3AVW_EDIFICI&STYLES=&SRS=EPSG%3A3857&CRS=EPSG%3A3857&TILED=true&ENV=mapstore_language%3Ait&ms2-authkey=f12293fe-52b5-418c-b145-467e3eb15746&WIDTH=256&HEIGHT=256&BBOX=1262128.2110448293%2C5860579.832681034%2C1264574.195949955%2C5863025.817586159

the query param we are talking about is indeed the following one:

&ENV=mapstore_language%3Ait

Therefore when saying:

During last checks with Kibana we found some requests to GeoServer referencing the ENV parameter were generating errors.

We have to check from Kibana if there are failing requests like the above with the ENV param and see why these are failing to be able to evaluate if the reason is related to a bug in MS or to a miss configuration in geoserver.

[lpasquali]

Last two months errors with layer: http://kibana:5601/app/kibana#/visualize/edit/Bolzano-GeoServer-Error-messages?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-2M,to:now))&_a=(filters:!(),linked:!f,query:(language:lucene,query:'type:geoserver-audit%20AND%20Error:true'),uiState:(vis:(params:(sort:(columnIndex:2,direction:desc)))),vis:(aggs:!((enabled:!t,id:'1',params:(),schema:metric,type:count),(enabled:!t,id:'2',params:(field:ErrorMessage.keyword,missingBucket:!f,missingBucketLabel:Missing,order:desc,orderBy:_key,otherBucket:!f,otherBucketLabel:Other,size:30),schema:bucket,type:terms),(enabled:!t,id:'3',params:(field:LAYERS.keyword,missingBucket:!f,missingBucketLabel:Missing,order:desc,orderBy:_key,otherBucket:!f,otherBucketLabel:Other,size:30),schema:bucket,type:terms)),params:(dimensions:(buckets:!((accessor:0,aggType:terms,format:(id:terms,params:(id:string,missingBucketLabel:Missing,otherBucketLabel:Other,parsedUrl:(basePath:'',origin:'http:%2F%2Fkibana:5601',pathname:%2Fapp%2Fkibana))),label:'ErrorMessage.keyword:%20Descending',params:())),metrics:!((accessor:1,aggType:count,format:(id:number),label:Count,params:()))),perPage:10,percentageCol:'',showMetricsAtAllLevels:!f,showPartialRows:!f,showTotal:!f,sort:(columnIndex:!n,direction:!n),totalFunc:sum),title:'Bolzano%20-%20GeoServer%20-%20Error%20messages',type:table))

errors of that single layer in this issue:

http://kibana:5601/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-5M,to:now))&_a=(columns:!(_source),index:c6e76760-95fd-11ea-8830-df562728e334,interval:auto,query:(language:kuery,query:'type:geoserver-audit%20AND%20Error:true%20AND%20LAYERS:dbt_pub*EDIFICI'),sort:!(!('@timestamp',desc)))

in this latter case error is always "Could not find layer dbt_pub:VW_EDIFICI", more over it is no more present since end of march.

a thing that I think is not correct is the fact that logged entries for audits have strage doubled values, see this screenshot for example

[randomorder]

that's likely a bug in the parsing / ingestion pipeline we need to fix it moving back to Ready to reschedule the work FYI @tdipisa

[randomorder]

can you please add an estimate @lpasquali ?

[lpasquali]

@randomorder I put a "political" 2 because being a bug which might be related to small typos in the log filters, it could take some amount of time to be spotted.

[lpasquali]

problem was there since May 14 2020, after a work on logstash it was created a backup file in the pipelines directory, logstash takes no consideration about extensions removing the file resolved @randomorder

[randomorder]

ok Can you let us know about the failing request that include the "ENV" parameter in the query string? Are they still there?

[lpasquali]

ok Can you let us know about the failing request that include the "ENV" parameter in the query string? Are they still there?

sure but I'd suggest to let the system ingest at least 48 hours of clean data

[lpasquali]

@tdipisa @randomorder

I was able to analyze log after my fixes I exported latest 10 days in a sheet here MapStore2-C028#184 and put an unique count of layers with error in a second sheet. https://docs.google.com/spreadsheets/d/1P7IMAIBYSdlb6hmMUcfLAZYghAfoaju54rem0_TYi7Y/edit#gid=1815483739

roughly half of errors (3900 out of 9200) are with ENV parameter in a 10 days period analyzed

[tdipisa]

@lpasquali

roughly half of errors (3900 out of 9200) are with ENV parameter in a 10 days period analyzed

But, as far as I can see by looking at the ErrorMessage, none of these requests is failing due to a problem with the ENV parameter and the errors seems correct. Therefore, this issue can be closed in my opinion.