search

balena-os / leviathan

A distributed hardware testing framework
https://balena-os.github.io/leviathan
Apache License 2.0
22 stars 6 forks source link

Add Config.js Validator #1079

Closed vipulgupta2048 closed 5 months ago

vipulgupta2048 commented 8 months ago

After the feedback from customers, it seems having a validator utility that actually checks inputs provided on runtime in the config.js could be beneficial. This provides helpful error that users can understand and help in debugging what the actual issue could be.

Signed-off-by: Vipul Gupta (@vipulgupta2048) vipulgupta2048@gmail.com

rcooke-warwick commented 5 months ago

other than that it looks good, and I can see the output of the e2e test on this PR and I like the formatting - it would be interesting to see what happens if an invalid configuration was given, do you have an example?

rcooke-warwick commented 5 months ago

as we mentioned, there is still the issue that if the image is not present, the worker will not be torn down so you have to wait or manually tear it down - but as you say that can be a separate PR.

vipulgupta2048 commented 5 months ago

@rcooke-warwick Output for invalid configuration, more context: https://balena.zulipchat.com/#narrow/stream/345889-balena-io.2Fos/topic/.5Bos-testing.5D.20Leviathan.20UX.20improvements/near/403224017

Screenshot-from-2023-11-21-00-13-14

Indeed will be creating a sub issue for this and another pull request to explicitly tear worker down when validation fails

rcooke-warwick commented 5 months ago

last question @vipulgupta2048 - just on the off chance there's something jank, has this been tested with jenkins? So a job triggered to use this leviathan branch.

vipulgupta2048 commented 5 months ago

last question @vipulgupta2048 - just on the off chance there's something jank, has this been tested with jenkins? So a job triggered to use this leviathan branch.

No worries, keep the questions running. Great question, Executed this job https://jenkins.product-os.io/job/leviathan-v2-template/38852/console with a near latest generix-x86-ext job and the commits from the PR. The logs are below:

jenkins-leviathan-v2-template-38852-client-1  | Validation config 1 from config.js file ...
jenkins-leviathan-v2-template-38852-client-1  | Device is valid. Targeting tests on Generic x86_64 (legacy MBR) DUT ✅
jenkins-leviathan-v2-template-38852-client-1  | Using test suite from path: /usr/src/app/workspace/../suites/hup ✅
jenkins-leviathan-v2-template-38852-client-1  | Testing using OS image: #/usr/src/app/workspace/balena-image.docker ✅
jenkins-leviathan-v2-template-38852-client-1  | 
jenkins-leviathan-v2-template-38852-client-1  | Validation config 2 from config.js file ...
jenkins-leviathan-v2-template-38852-client-1  | Device is valid. Targeting tests on Generic x86_64 (legacy MBR) DUT ✅
jenkins-leviathan-v2-template-38852-client-1  | Using test suite from path: /usr/src/app/workspace/../suites/cloud ✅
jenkins-leviathan-v2-template-38852-client-1  | Testing using OS image: #/usr/src/app/workspace/balena.img.gz ✅
jenkins-leviathan-v2-template-38852-client-1  | 
jenkins-leviathan-v2-template-38852-client-1  | Validation config 3 from config.js file ...
jenkins-leviathan-v2-template-38852-client-1  | Device is valid. Targeting tests on Generic x86_64 (legacy MBR) DUT ✅
jenkins-leviathan-v2-template-38852-client-1  | Using test suite from path: /usr/src/app/workspace/../suites/os ✅
jenkins-leviathan-v2-template-38852-client-1  | Testing using OS image: #/usr/src/app/workspace/balena.img.gz ✅
jenkins-leviathan-v2-template-38852-client-1  | 
jenkins-leviathan-v2-template-38852-client-1  | [2024-02-05T14:44:27.354Z] INFO: Configuration Validated ✔️