search

canonical / edgex-snap-testing

4 stars 2 forks source link

Remove config override tests after service startup when using config provider #172

Closed farshidtz closed 1 year ago

farshidtz commented 1 year ago

As expected by #4448, the snap tests have started failing since this change because the overrides no longer apply on top of configurations coming from config provider: Common configuration loaded from the Configuration Provider. No overrides applied

Need to change the snap tests to no longer test that, or make the services run without config provider.

Failing test result:

=== RUN   TestChangeStartupMsg_app
    config_test.go:26: Set and verify new startup message: snap-testing (app)
    exec.go:19: [exec] sudo snap set edgexfoundry apps.support-scheduler.config.service-startupmsg='snap-testing (app)'
    exec.go:19: [exec] sudo snap restart edgexfoundry.support-scheduler
    exec.go:101: [stdout] 2023-03-16T14:24:06Z INFO Waiting for "snap.edgexfoundry.support-scheduler.service" to stop.
    exec.go:101: [stdout] Restarted.
    net.go:132: Retry 1/180: Waiting for ports: 59880 (core-data), 59881 (core-metadata), 59882 (core-command), 8200 (vault), 8500 (consul), [63](https://github.com/edgexfoundry/edgex-go/actions/runs/4431591455/jobs/7788709103?pr=4449#step:2:67)79 (redis)
    config_test.go:109: Retry 1/10: Waiting for startup message: snap-testing (app)
    exec.go:19: [exec] sudo journalctl --since "2023-03-16 14:24:05" --no-pager | grep "edgexfoundry.support-scheduler"|| true
    config_test.go:109: Retry 2/10: Waiting for startup message: snap-testing (app)
    exec.go:19: [exec] sudo journalctl --since "2023-03-16 14:24:05" --no-pager | grep "edgexfoundry.support-scheduler"|| true
    config_test.go:109: Retry 3/10: Waiting for startup message: snap-testing (app)
    exec.go:19: [exec] sudo journalctl --since "2023-03-16 14:24:05" --no-pager | grep "edgexfoundry.support-scheduler"|| true
    config_test.go:109: Retry 4/10: Waiting for startup message: snap-testing (app)
    exec.go:19: [exec] sudo journalctl --since "2023-03-16 14:24:05" --no-pager | grep "edgexfoundry.support-scheduler"|| true
    config_test.go:109: Retry 5/10: Waiting for startup message: snap-testing (app)
    exec.go:19: [exec] sudo journalctl --since "2023-03-16 14:24:05" --no-pager | grep "edgexfoundry.support-scheduler"|| true
    config_test.go:109: Retry 6/10: Waiting for startup message: snap-testing (app)
    exec.go:19: [exec] sudo journalctl --since "2023-03-16 14:24:05" --no-pager | grep "edgexfoundry.support-scheduler"|| true
    config_test.go:109: Retry 7/10: Waiting for startup message: snap-testing (app)
    exec.go:19: [exec] sudo journalctl --since "2023-03-16 14:24:05" --no-pager | grep "edgexfoundry.support-scheduler"|| true
    config_test.go:109: Retry 8/10: Waiting for startup message: snap-testing (app)
    exec.go:19: [exec] sudo journalctl --since "2023-03-16 14:24:05" --no-pager | grep "edgexfoundry.support-scheduler"|| true
    config_test.go:109: Retry 9/10: Waiting for startup message: snap-testing (app)
    exec.go:19: [exec] sudo journalctl --since "2023-03-16 14:24:05" --no-pager | grep "edgexfoundry.support-scheduler"|| true
    config_test.go:109: Retry 10/10: Waiting for startup message: snap-testing (app)
    exec.go:19: [exec] sudo journalctl --since "2023-03-16 14:24:05" --no-pager | grep "edgexfoundry.support-scheduler"|| true
    config_test.go:118: Time out: reached max 10 retries.
    config_test.go:31: 
            Error Trace:    /home/runner/work/_actions/canonical/edgex-snap-testing/v3/test/suites/edgexfoundry/config_test.go:31
            Error:          Should be true
            Test:           TestChangeStartupMsg_app
            Messages:       new startup message = snap-testing (app)
MonicaisHer commented 1 year ago

For the services listed, environment variable overrides will not be applied:

Here is an example:

snap install edgexfoundry --edge
sudo snap set edgexfoundry apps.support-scheduler.config.service-startupmsg='manul-snap-testing-A'
sleep 10
sudo snap restart edgexfoundry.support-scheduler
app=support-scheduler source=config.go:143 msg="Common configuration loaded from the Configuration Provider. No overrides applied"
farshidtz commented 1 year ago

In order to run the services without the config provider, the services should have the full set of configurations (local and common) available to them. The default service configuration files only include the local config. We should either inject the common config parts into the same file, or point to the common config file via the respective common config environment variable override. The latter is currently being implemented; see https://github.com/edgexfoundry/go-mod-bootstrap/pull/487.

The common config file is available at https://github.com/edgexfoundry/edgex-go/blob/main/cmd/core-common-config-bootstrapper/res/configuration.yaml. It can be sourced from the edgexfoundry snap or for quick/manual testing directly from github.

MonicaisHer commented 1 year ago

An example of making service run without config provider:

# Use no configuration provider
sudo snap set edgexfoundry apps.core-data.config.edgex-config-provider='none'

# Pull common configuration
sudo snap set edgexfoundry apps.core-data.config.edgex-common-config=./config/core-common-config-bootstrapper/res/configuration.yaml

# Set environment variables
sudo snap set edgexfoundry apps.core-data.config.service-startupmsg='manual-snap-testing'

snap restart edgexfoundry.core-data
MonicaisHer commented 1 year ago

The behavior of core-data has changed compared to its previous version:

$ snap install edgexfoundry --edge
edgexfoundry (edge) 3.0.0-dev.114 from Canonical✓ installed
$ sudo snap set edgexfoundry apps.core-data.config.edgex-config-provider='none'
$ sudo snap set edgexfoundry apps.core-data.config.edgex-common-config=./config/core-common-config-bootstrapper/res/configuration.yaml
$ sudo snap set edgexfoundry apps.core-data.config.service-startupmsg='manual-snap-testing'
$ snap restart edgexfoundry.core-data
Mär 27 14:51:55 ubuntu systemd[1]: Started Service for snap application edgexfoundry.core-data.
Mär 27 14:51:55 ubuntu edgex-core-data.service-config-overrides.sh[34122]: sourcing /var/snap/edgexfoundry/4359/config/core-data/res/core-data.env
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: level=INFO ts=2023-03-27T12:51:55.473473087Z app=core-data source=secret.go:65 msg="Creating SecretClient"
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: level=INFO ts=2023-03-27T12:51:55.473585204Z app=core-data source=variables.go:457 msg="Variables override of 'SecretStore/TokenFile' by environment variable: SECRETSTORE_TOKENFILE=/var/snap/edgexfoundry/4359/secrets/core-data/secrets-token.json"
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: level=INFO ts=2023-03-27T12:51:55.473645868Z app=core-data source=secret.go:159 msg="SecretStore information created with 1 overrides applied"
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: level=INFO ts=2023-03-27T12:51:55.473653455Z app=core-data source=secret.go:75 msg="Reading secret store configuration and authentication token"
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: level=INFO ts=2023-03-27T12:51:55.473660562Z app=core-data source=secret.go:200 msg="load token from file"
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: level=INFO ts=2023-03-27T12:51:55.473732885Z app=core-data source=secret.go:93 msg="Attempting to create secret client"
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: level=INFO ts=2023-03-27T12:51:55.474765087Z app=core-data source=secret.go:104 msg="Created SecretClient"
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: level=INFO ts=2023-03-27T12:51:55.474775605Z app=core-data source=secret.go:109 msg="SecretsFile not set, skipping seeding of service secrets."
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: level=INFO ts=2023-03-27T12:51:55.474787185Z app=core-data source=variables.go:457 msg="Variables override of 'Configuration Provider Information' by environment variable: EDGEX_CONFIG_PROVIDER=none"
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: level=INFO ts=2023-03-27T12:51:55.474795573Z app=core-data source=variables.go:457 msg="Variables override of '-cc/--commonConfig' by environment variable: EDGEX_COMMON_CONFIG=./config/core-common-config-bootstrapper/res/configuration.yaml"
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: level=INFO ts=2023-03-27T12:51:55.474801088Z app=core-data source=config.go:565 msg="Loading configuration file from ./config/core-common-config-bootstrapper/res/configuration.yaml"
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: level=INFO ts=2023-03-27T12:51:55.475239929Z app=core-data source=secrets.go:277 msg="kick off token renewal with interval: 30m0s"
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: panic: runtime error: invalid memory address or nil pointer dereference
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x98 pc=0x7553df]
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: goroutine 1 [running]:
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: github.com/edgexfoundry/go-mod-bootstrap/v3/bootstrap/environment.(*Variables).buildPaths(0xc0004b0f38?, 0x5?)
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]:         github.com/edgexfoundry/go-mod-bootstrap/v3@v3.0.0-dev.56/bootstrap/environment/variables.go:236 +0x11f
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: github.com/edgexfoundry/go-mod-bootstrap/v3/bootstrap/environment.(*Variables).buildPaths(0xc0004b10a8?, 0x5?)
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]:         github.com/edgexfoundry/go-mod-bootstrap/v3@v3.0.0-dev.56/bootstrap/environment/variables.go:243 +0x1d4
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: github.com/edgexfoundry/go-mod-bootstrap/v3/bootstrap/environment.(*Variables).buildPaths(0x852dc0?, 0xc0004423e0?)
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]:         github.com/edgexfoundry/go-mod-bootstrap/v3@v3.0.0-dev.56/bootstrap/environment/variables.go:243 +0x1d4
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: github.com/edgexfoundry/go-mod-bootstrap/v3/bootstrap/environment.(*Variables).OverrideConfigMapValues(0xc000012378, 0x61c?)
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]:         github.com/edgexfoundry/go-mod-bootstrap/v3@v3.0.0-dev.56/bootstrap/environment/variables.go:149 +0x3e
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: github.com/edgexfoundry/go-mod-bootstrap/v3/bootstrap/environment.(*Variables).OverrideConfiguration(0xc0000b69a0?, {0x8ccd00, 0xc0003c5e00})
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]:         github.com/edgexfoundry/go-mod-bootstrap/v3@v3.0.0-dev.56/bootstrap/environment/variables.go:130 +0x14f
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: github.com/edgexfoundry/go-mod-bootstrap/v3/bootstrap/config.(*Processor).Process(0xc0000b69a0, {0x9132b7, 0x9}, {0x90f6dc, 0x5}, {0x912c27, 0x8}, {0x9ce3b8, 0xc0003c5e00}, {0x9ceea0, ...})
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]:         github.com/edgexfoundry/go-mod-bootstrap/v3@v3.0.0-dev.56/bootstrap/config/config.go:188 +0x765
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: github.com/edgexfoundry/go-mod-bootstrap/v3/bootstrap.RunAndReturnWaitGroup({0x9cb2d0, 0xc0003eeaa0}, 0xc0003fb7b0, {0x9ce418?, 0xc00042a770}, {0x9132b7, 0x9}, {0x912c27, 0x8}, {0x9ce3b8, ...}, ...)
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]:         github.com/edgexfoundry/go-mod-bootstrap/v3@v3.0.0-dev.56/bootstrap/bootstrap.go:126 +0x4ac
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: github.com/edgexfoundry/go-mod-bootstrap/v3/bootstrap.Run({0x9cb2d0?, 0xc0003eeaa0?}, 0xc0003fb7b0, {0x9ce418?, 0xc00042a770?}, {0x9132b7?, 0x0?}, {0x912c27?, 0x0?}, {0x9ce3b8, ...}, ...)
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]:         github.com/edgexfoundry/go-mod-bootstrap/v3@v3.0.0-dev.56/bootstrap/bootstrap.go:215 +0x11d
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: github.com/edgexfoundry/edgex-go/internal/core/data.Main({0x9cb2d0, 0xc0003eeaa0}, 0x405391?, 0xc000001d40)
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]:         github.com/edgexfoundry/edgex-go/internal/core/data/main.go:62 +0x705
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]: main.main()
Mär 27 14:51:55 ubuntu edgexfoundry.core-data[34092]:         github.com/edgexfoundry/edgex-go/cmd/core-data/main.go:27 +0x89
Mär 27 14:51:55 ubuntu systemd[1]: snap.edgexfoundry.core-data.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mär 27 14:51:55 ubuntu systemd[1]: snap.edgexfoundry.core-data.service: Failed with result 'exit-code'.
farshidtz commented 1 year ago

@MonicaisHer please report the bug at https://github.com/edgexfoundry/go-mod-bootstrap

MonicaisHer commented 1 year ago

@farshidtz sure, reported: https://github.com/edgexfoundry/go-mod-bootstrap/issues/494