Publishing a message to a non-existent topic without auto topic creation returns GATEWAY_TIMEOUT

[fangliuwh]

In my setup of the kafka restproxy, the auto topic creation is switched off; And topic 'test' is not manually created either. Sending the following message gets gateway timeout. I think there are two issues with it: (1) the response code GatewayTimeout is wrong. Would BadRequest be better? (2) it takes long time before we get the gateway timeout (kind of expected in case of gateway timeout), but it should take much less time if we change it to return other error code.

curl -X POST -H "Content-Type: application/vnd.kafka.binary.v1+json" --data "{\"records\":[{\"value\":\"S2Fma2E=\"}]}" "http://localhost:8082/topics/test" -v

• About to connect() to localhost port 8082 (#0)
• Trying 10.160.11.250... connected
• Connected to localhost port 8082 (#0)

  POST /topics/test HTTP/1.1 User-Agent: curl/7.21.1 (i686-pc-mingw32) libcurl/7.21.1 OpenSSL/0.9.8r zlib/1.2.3 Host: localhost:8082 Accept: / Content-Type: application/vnd.kafka.binary.v1+json Content-Length: 34

  < HTTP/1.1 504 GATEWAY_TIMEOUT < Content-Length: 0 < Connection: keep-alive <

• Connection #0 to host localhost left intact
• Closing connection #0

[jsmaupin]

Would it be possible to return 404 Not Found here given that the resource does not exist?

[ewencp]

Unfortunately it's difficult to satisfy the needs of both clusters w/ auto topic creation and those without. Returning a 404 immediately might be feasible, though we'd then have to invalidate the cached metadata immediately so that any subsequent produce request wouldn't also immediately fail if auto topic creation was enabled. Also, to make it work with auto topic creation, we need to actually try to produce the data, but the reason for the timeout is that when we try to produce it the failure doesn't occur until after this timeout.

If you would prefer getting something other than a timeout error, I'd recommend adjusting the max.block.ms setting in your REST proxy configuration file. It defaults to 60s which is longer than the timeout imposed for each request. If you reduce it to less than the 30s timeout, if a topic isn't available after the specified amount of time you'll get a 500 error with error coded 50003 indicating a timeout. Your app could then decide what to do with that (retry if you expect auto topic creation to be enabled, fail/log an error if you don't).