search

ballerina-platform / ballerina-spec

Ballerina Language and Platform Specifications
Other
167 stars 53 forks source link

Support configurable values to be read from env #723

Open anuruddhal opened 3 years ago

anuruddhal commented 3 years ago

Description: We need to support reading configurable values from environment variables as many Cloud applications depend on configurations via env.

Please refer https://12factor.net/

Suggested Labels:

Code sample that shows issue:

Related Issues: https://github.com/ballerina-platform/ballerina-lang/issues/28327

sameerajayasoma commented 3 years ago

The platform does not yet support this.

In the meantime, can we store the complete Config.toml file in a ConfigMap resource and mount it as a volume on to the pod? In addition to this, you need to give the path of the Config.toml via the BALCONFIGFILE env variable.

jclark commented 3 years ago

You can have a environment variable whose value is TOML. At least in Linux, you can have 128k of environment variables.

isurulucky commented 3 years ago

@sameerajayasoma @jclark This feature was supported previously, however seems to be removed from a recent version on. IMO in the cloud native perspective, using env variables is the more intuitive than using toml, at least for the simple usecases. If cannot be done immediately, do we have a timeline to support this? Can we use the getEnv('xxx') method from the os package as a substitute for this functionality, if this will not be supported?

jclark commented 3 years ago

We should allow them to be specified on the command-line also: an option would specify a value for a single configurable variable, and could be repeated to specify multiple variables.

pubudu91 commented 3 years ago

In the previous config API, configs were considered from multiple sources. So whether the configs were defined in a file or as env vars, they could be looked up through the config API. And there was a precedence defined to resolve conflicts which could arise with the same config key defined in multiple sources. The precedence (in order of increasing priority) was conf-file < env vars < CLI args e.g., if there was a config key named foo.bar in the config file, if needed the user could override it by providing a new value to it either through an env var or a CLI arg. Use of the config API was then like a lookup on this collated source of configs.

Will it be possible to support the same or a similar behaviour with the configurable feature as well?

sameerajayasoma commented 3 years ago

Refer to this specification that defines how values can be supplied to configurable variables via TOML syntax and command-line arguments.

However, we are not planning to support supplying individual values via environment variables. Use the following environment variables to supply the toml file or the contents of the toml file. Read this section in the spec for more details.

xlight05 commented 2 years ago

Reopening the issue as this topic came up in a discussion of implementation of micro service samples.

Communicating from one miroservice to another is a common use case. You can see an example here. Other languages usually uses env variables for this purpose. https://github.com/GoogleCloudPlatform/microservices-demo/blob/521602c7f6e097b095dba4b899714134b3005362/kubernetes-manifests/checkoutservice.yaml#L41

If we use confiugrables we have to keep everything in one env variable. https://github.com/xlight05/gcp-microservices-demo/blob/79976216b188b0b7336546ec54337db614ed9dd7/src/docker-compose.yml#L11 This can be complex when we have to replace the values in CICD pipelines and when we use tools kustomize to replace the value.

sameerajayasoma commented 1 year ago

We already have two ways to supply values to configurable variables:

Here is an approach to supply values via environment variables:

The toString() representation of the value can be provided with an environment variable.

Type of value Representation of the value Parse
boolean toString() boolean:fromString()
string toString() use as-is
int toString() int:fromString()
float toString() float:fromString()
decimal toString() decimal:fromString()
xml toString() xml:fromString()

Examples: With command-line arguments: 

bal run -- -Cballerina.log.level="DEBUG"

Environment variables:

BAL_CONFIG_BALLERINA_LOG_LEVEL="DEBUG" bal run
xlight05 commented 1 year ago

We already have two ways to supply values to configurable variables:

  • Config.toml file.

  • Command-line args

    • Only for configurable variables of type boolean | int | float | decimal | string | xml

Here is an approach to supply values via environment variables:

  • Allowed only for configurable variables of type S.

    • type S boolean | int | float | decimal | string | xml
  • Each configurable variable of S with the structured identifier p.q.r.s that conforms to [[org .] module .] variable, can be configured via an environment variable of the form BAL_CONFIG_P_Q_R_S.
  • BAL_CONFIG_ prefix is used to identify environment variables defined for Ballerina programs.
  • Each part in the structured identifier is converted to uppercase, and dots are converted to underscores.

The toString() representation of the value can be provided with an environment variable.

Type of value Representation of the value Parse boolean toString() boolean:fromString() string toString() use as-is int toString() int:fromString() float toString() float:fromString() decimal toString() decimal:fromString() xml toString() xml:fromString() Examples: With command-line arguments:

bal run -- -Cballerina.log.level="DEBUG"

Environment variables:

BAL_CONFIG_BALLERINA_LOG_LEVEL="DEBUG" bal run

This should be perfect for code to cloud related use cases. The supported toml features will be same as Command-line args right?

Just to understand, whats stopping us from supporting records as well? Is it the toml table arrays?

HindujaB commented 1 year ago

@xlight05 AFAIK, we use string processing to support the command-line configuration. And configuring types like structures, and lists can make the logic more complex as we need to handle underlying types such as fields and elements. Plus the fromString() of such values can contain special characters which need to be handled.

HindujaB commented 1 year ago

From the implementation perspective,

warunalakshitha commented 1 year ago

One of the limitation of using environment variables is, configurable variables with quoted identifiers.

According to [1] "Environment variable names used by the utilities in the Shell and Utilities volume of IEEE Std 1003.1-2001 consist solely of uppercase letters, digits, and the '_' (underscore) from the characters defined in Portable Character Set and do not begin with a digit."

[1] https://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html

yuriylesyuk commented 1 year ago

@sameerajayasoma @jclark This feature was supported previously, however seems to be removed from a recent version on. IMO in the cloud native perspective, using env variables is the more intuitive than using toml, at least for the simple usecases. If cannot be done immediately, do we have a timeline to support this? Can we use the getEnv('xxx') method from the os package as a substitute for this functionality, if this will not be supported?

This a legitimate requirement. Use case I'm implementing is:

https://cloud.google.com/run/docs/configuring/services/containers#configure-port

"For Cloud Run services, Cloud Run injects the PORT environment variable into the container. The container should listen on the port defined by the PORT environment variable rather than a specific hardcoded port. "

With help of @pcnfernando I can confirm that getEnv() is a good start for a substitute.

` int PORT = check int:fromString(os:getEnv("PORT"));

service http:Service /foo on new http:Listener(PORT) { `

For a better solution, it would be good to have

os:getEnvOrDefault( <envvar>, <default-value-if-not-defined> )

function.

sanjiva commented 9 months ago

I'm +1 for going ahead with @sameerajayasoma proposal above. I don't see any major negative comments above .. if there is can someone restate it please? We can also introduce an annotation so users can override the (potentially ugly) default computed environment variable name.

HindujaB commented 8 months ago

As per the offline discussion with @warunalakshitha and @sameerajayasoma, the following design decisions were taken.