search

telefonicaid / fiware-cygnus

A connector in charge of persisting context data sources into other third-party databases and storage systems, creating a historical view of the context
https://fiware-cygnus.rtfd.io/
GNU Affero General Public License v3.0
64 stars 105 forks source link

[NGSIRestHandler] Bad HTTP notification (application/json; charset=UTF-8 content type not supported) #1609

Closed pmo-sdr closed 5 years ago

pmo-sdr commented 5 years ago

When Apache Flume is updated to v1.9 unexpected "400 Bad request" was returned while sending data to Cygnus.

How to

Sending like (working ok with v1.4)

curl $URL -v -s -S --header 'Content-Type: application/json; charset=utf-8' --header 'Accept: application/json' --header 'User-Agent: orion/0.10.0' --header "Fiware-Service: $SERVICE" --header "Fiware-ServicePath: $SERVICE_PATH" -d @- <<EOF

You get:

lvl=WARN | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnusagent | op=getEvents | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[193] : [NGSIRestHandler] Bad HTTP notification (application/json; charset=UTF-8 content type not supported)

Why

Tip: Sending:

--header 'Content-Type: application/json; charset=utf-8'

You get (notice "UTF-8" uppercase received)

--->400 Bad Request
com.telefonica.iot.cygnus.handlers.NGSIRestHandler[185] : [NGSIRestHandler] Header content-type received with value application/json; charset=UTF-8

Sending whitout space before "application":

--header 'Content-Type:application/json; charset=utf-8'

You get: (notice "utf-8" received as is)

--->200 OK
com.telefonica.iot.cygnus.handlers.NGSIRestHandler[185] : [NGSIRestHandler] Header content-type received with value application/json; charset=utf-8

It looks like version 3.1 of javax.servlet, introduced by Flume 1.9, formats header values and causes wrongContentType() method to fail validating value.

https://github.com/telefonicaid/fiware-cygnus/blob/c43bdf845a6ea890b5ed856370b314abf199fc9b/cygnus-ngsi/src/main/java/com/telefonica/iot/cygnus/handlers/NGSIRestHandler.java#L395-L397

Solution

We think toLowerCase is needed at NGSIRestHandler line 396 (or 184) to solve this issue.

https://github.com/telefonicaid/fiware-cygnus/blob/c43bdf845a6ea890b5ed856370b314abf199fc9b/cygnus-ngsi/src/main/java/com/telefonica/iot/cygnus/handlers/NGSIRestHandler.java#L183-L194

fgalan commented 5 years ago

Thank you for so detailed analysis. The bug reports that include solution hints are the best ones! :)

We haven't detected this bug in our e2e CI system after upgrading Flume to 1.9 but probably we are not covering this specific case.

Maybe you could do a PR with the fix? Or prefer us to do so?

pmo-sdr commented 5 years ago

We are launching local tests right now, we will do a PR when finished.

fgalan commented 5 years ago

We haven't detected this bug in our e2e CI system after upgrading Flume to 1.9 but probably we are not covering this specific case.

This explain why we haven't seen fails in e2e CI after the change in pom.xml files: https://github.com/telefonicaid/fiware-cygnus/pull/1604#issuecomment-476571136

fgalan commented 5 years ago

Fixed by PR https://github.com/telefonicaid/fiware-cygnus/pull/1610