tomuben
commented
3 years ago The root cause of this is the fact that we always assign a variable to TestContainer, here for external_exasol_db_port.
If this variable is not set by the caller, it is None, and will remain None for Luigi. Even if we declare a default value for the parameter, it will be overwritten with None.
(If you for example remove the line in the link above, Luigi will take the default value then).
I can think of following solutions:
1.Clients need to take care what they assign to the integration test docker environment. If external_exasol_db_port is not used by integration-test-docker-environment, there is no issue today. If it would be used, integration-test-docker-environment would raise an exception. We can change click options to help clients to avoid assign None for those parameters: here
- We create a new Parameter class in integration-test-docker-environment which always takes a default parameter, and always returns this if a clients passes None: here
- More generic, and more effort: All tasks in integration-test-docker-environmnet need to verify if arguments are valid.
Background
It turned out that Luigi treats dynamic dependencies differently from static dependencies, and parameters which are not set can cause unexpected errors when transforming a dependency to a dynamic one. This was the case of
which are forwarded in cli.commands.run_db_test() to the TestContainer, but actually not set, and then eventually used as an Luigi IntParameter.
Acceptance Criteria
Not setting these parameter not correctly must not result in any unexpected behavior. Best way is to initialize these parameter correctly in cli.commands.run_db_test()