Redesigning Configs in Heron

[jerrypeng]

Typing for configs, whether on a cluster, topology, or component level, is not clear. Some configs are manually converted into Strings while others are left as its original type. This can be very frustrating to developers when setting or reading configs since the type of the config is not clear. This may also cause unexpected bugs in the code.

[jerrypeng]

My thoughts:

For every config we need to have single source of truth of what type the config is so as to be consistent across the whole project. I also don't think we should manually convert the configs to strings. If we need to do so across languages for compatibility, we can always create a layer of converting certain configs to strings and converting them back to the correct types where appropriate

[jerrypeng]

Perhaps we can use annotations like in Storm to specify what types configs are and also validate config types accordingly e.g.

@isString Public static String SOME.HERON.CONFIG

[huijunw]

There are two configs, one from APi and the other from Spi. Does this issue target at APi.config or SPI.config, or both? For annotations, does annotation apply to c++ and python?

[srkukarni]

This is basically API configs if I understand correctly.

[jerrypeng]

Correct I am talking about API.Config. For C++ and python we could have similar approach ( though not sure what at this moment). We just need to be clear what is the type of the config that is being passed around in the code so all developers are on the same page