search

clicon / clixon

YANG-based toolchain including NETCONF and RESTCONF interfaces and an interactive CLI
http://www.clicon.org/
Other
216 stars 72 forks source link

Tests should continue gracefully #451

Closed pprindeville closed 1 year ago

pprindeville commented 1 year ago

I'm trying to run all.sh to test my changes, and I'm hitting some failures because $IETFRFC ($YANG_STANDARD_DIR/ietf/RFC) haven't been installed. Not sure where that gets installed, but it's not part of clixon. Yet clixon requires it in the tests.

Anyway, what would be better than having the tests run and then fail due to a missing component would be to skip the tests that it's not able to complete for environmental/setup reasons and test what it can.

Otherwise, I'm having to run the tests over and over with an increasingly long $SKIPLIST.

pprindeville commented 1 year ago

Maybe add skiptest() to test/lib.sh

olofhagsand commented 1 year ago

No, Clixon error-handling and testing uses "fail early and loud" as a primary principle, not skipping tests if they fail. The info on downloading the yang is cleary given in the README, please read it: https://github.com/clicon/clixon/blob/master/test/README.md#openconfig-and-yang

  cd /usr/local/share
  git clone https://github.com/YangModels/yang
pprindeville commented 1 year ago

Not entirely true:

test_autocli_listkey_compress.sh:    echo "...skipped: OPENCONFIG not set"
test_autocli_obsolete.sh:# - obsolete - skipped
test_http_data.sh:    echo "...skipped: Must run with --with-restconf=native"
test_netconf_monitoring_location.sh:    echo "...skipped: Must run with --with-restconf=native"
test_netconf_ssh_callhome.sh:    echo "...skipped "
test_openconfig_interfaces.sh:    echo "...skipped: OPENCONFIG not set"
test_openconfig_network_instance.sh:    echo "...skipped: OPENCONFIG not set"
test_openconfig.sh:    echo "...skipped: OPENCONFIG not set"
test_pagination_config.sh:    echo "...skipped: Expect not installed"
test_pagination_config.sh:    echo "...skipped for valgrind "
test_privileges.sh:    echo "...skipped "
test_restconf_basic_auth.sh:    echo "...skipped: Must run with http/1"
test_restconf_callhome.sh:    echo "...skipped: Must run with http/1"
test_restconf_continue.sh:    echo "...skipped: Must run with http/1"
test_restconf_err.sh:    echo "...skipped: must run with http/1"
test_restconf_http_upgrade.sh:    echo "...skipped: must run with http/1"
test_restconf_internal.sh:    echo "...skipped: Must run with http/1"
test_restconf_internal_usecases.sh:    echo "...skipped: Must run with http/1"
test_restconf_netns.sh:    echo "...skipped: HAVE_HTTP1 is false, must run with http/1"
test_restconf_notifications.sh:    echo "...skipped "
test_restconf_yang_patch_json.sh:    echo "...skipped: YANG_PATCH JSON NYI"
test_restconf_yang_patch_xml.sh:    echo "...skipped: YANG_PATCH XML NYI"
olofhagsand commented 1 year ago

Hm depends. Several of those are compile-time options, http/1, -with-restconf=native, which means they must be recompiled (done in backoffice tests) which means they cannot succedd in that binary. OPENCONFIG skip should probably be removed valgrind is a special case, should possibly say NYI