louberger commented 6 years ago

Added lib/autogen which builds on lib/lutil to generate pytests for each luInclud'ed script file.
To use, each test now has it: the a top level test that generates the real test file a customize.py file that has in it topology, verion check and an other routines to be used (including tests) the normal config directories (daemon run if config file present)

NetDEF-CI commented 6 years ago

Continuous Integration Result: SUCCESSFUL

Congratulations, this patch passed basic tests

Tested-by: NetDEF / OpenSourceRouting.org CI System

CI System Testrun URL: https://ci1.netdef.org/browse/FRR-TOPOPR-69/

This is a comment from an EXPERIMENTAL automated CI system. For questions and feedback in regards to this CI system, please feel free to email Martin Winter - mwinter (at) opensourcerouting.org.

louberger commented 6 years ago

I followed https://github.com/FRRouting/frr/blob/master/COMMUNITY.md#source-file-header

On 12/20/2017 9:24 PM, Martin Winter wrote:

@mwinter-osr requested changes on this pull request.

The topotest (so far) was under a open BSD style (ISC) license and you changed it in your file to GPL.

Can you fix (if error) or explain the reason (if intended)?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/FRRouting/topotests/pull/52#pullrequestreview-84953822, or mute the thread https://github.com/notifications/unsubscribe-auth/AGRWyFYrKHArWKs1C6lviaVomsHatfJjks5tCcFUgaJpZM4RIq6J.

mwinter-osr commented 6 years ago

That community.md is part of the FRR repo - not the topotests. We decided earlier some time back to build (and maintain) the topotests on a more open license as it's built from scratch.

louberger commented 6 years ago

Okay, I'll take a look.

On 12/21/2017 5:38 PM, Martin Winter wrote:

That community.md is part of the FRR repo - not the topotests. We decided earlier some time back to build (and maintain) the topotests on a more open license as it's built from scratch.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/FRRouting/topotests/pull/52#issuecomment-353475541, or mute the thread https://github.com/notifications/unsubscribe-auth/AGRWyKIh7vhM634qiewjNsSXK3jbD-sOks5tCt3UgaJpZM4RIq6J.

NetDEF-CI commented 6 years ago

Continuous Integration Result: SUCCESSFUL

Congratulations, this patch passed basic tests

Tested-by: NetDEF / OpenSourceRouting.org CI System

CI System Testrun URL: https://ci1.netdef.org/browse/FRR-TOPOPR-74/

This is a comment from an EXPERIMENTAL automated CI system. For questions and feedback in regards to this CI system, please feel free to email Martin Winter - mwinter (at) opensourcerouting.org.

louberger commented 6 years ago

@mwinter-osr do you want to see any other changes?

louberger commented 6 years ago

@mwinter-osr ????

NetDEF-CI commented 6 years ago

Continuous Integration Result: SUCCESSFUL

Congratulations, this patch passed basic tests

Tested-by: NetDEF / OpenSourceRouting.org CI System

CI System Testrun URL: https://ci1.netdef.org/browse/FRR-TOPOPR-75/

This is a comment from an EXPERIMENTAL automated CI system. For questions and feedback in regards to this CI system, please feel free to email Martin Winter - mwinter (at) opensourcerouting.org.

NetDEF-CI commented 6 years ago

Continuous Integration Result: SUCCESSFUL

Congratulations, this patch passed basic tests

Tested-by: NetDEF / OpenSourceRouting.org CI System

CI System Testrun URL: https://ci1.netdef.org/browse/FRR-TOPOPR-76/

This is a comment from an EXPERIMENTAL automated CI system. For questions and feedback in regards to this CI system, please feel free to email Martin Winter - mwinter (at) opensourcerouting.org.

mwinter-osr commented 6 years ago

More issues found: The new framework changes the file structures, so the tests are NOT discovered. If you look at the log of the CI above, you'll notice that none of your bgp_direct_to_bgp_vpn got executed. See https://ci1.netdef.org/browse/FRR-TOPOPR-TMB-76/log

The reason for this is that the bgp_direct_to_bgp_vpn/test_bgp_direct_to_bgp_vpn.py doesn't match the required format for pytest and the discovery skips it. See https://docs.pytest.org/en/latest/goodpractices.html#conventions-for-python-test-discovery

Basically, all test functions need to be prefixed with test_ and there needs to be a setup_module() and teardown_module() as well

if you run (in the main topotest directory) the discovery, you'll see the issue:

mkdir: cannot create directory ‘/tmp/topotests’: File exists
======================================================== test session starts =========================================================
platform linux2 -- Python 2.7.12, pytest-3.3.1, py-1.5.2, pluggy-0.6.0
rootdir: /root/topotests, inifile: pytest.ini
collected 75 items                                                                                                                   
<Module 'all-protocol-startup/test_all_protocol_startup.py'>
  <Function 'test_router_running'>
  <Function 'test_error_messages_vtysh'>
  <Function 'test_error_messages_daemons'>
  <Function 'test_converge_protocols'>
  <Function 'test_rip_status'>
  <Function 'test_ripng_status'>
  <Function 'test_ospfv2_interfaces'>
  <Function 'test_isis_interfaces'>
  <Function 'test_bgp_summary'>
  <Function 'test_bgp_ipv6_summary'>
  <Function 'test_bgp_ipv4'>
  <Function 'test_bgp_ipv6'>
  <Function 'test_mpls_interfaces'>
  <Function 'test_shutdown_check_stderr'>
  <Function 'test_shutdown_check_memleak'>
<Module 'bgp-ecmp-topo1/test_bgp_ecmp_topo1.py'>
  <Function 'test_bgp_convergence'>
  <Function 'test_bgp_ecmp'>
<Module 'bgp_multiview_topo1/test_bgp_multiview_topo1.py'>
  <Function 'test_router_running'>
  <Function 'test_bgp_converge'>
  <Function 'test_bgp_routingTable'>
  <Function 'test_shutdown_check_stderr'>
  <Function 'test_shutdown_check_memleak'>
<Module 'eigrp-topo1/test_eigrp_topo1.py'>
  <Function 'test_converge_protocols'>
  <Function 'test_eigrp_routes'>
  <Function 'test_zebra_ipv4_routingTable'>
  <Function 'test_shutdown_check_stderr'>
<Module 'ldp-topo1/test_ldp_topo1.py'>
  <Function 'test_router_running'>
  <Function 'test_mpls_interfaces'>
  <Function 'test_mpls_ldp_neighbor_establish'>
  <Function 'test_mpls_ldp_discovery'>
  <Function 'test_mpls_ldp_neighbor'>
  <Function 'test_mpls_ldp_binding'>
  <Function 'test_zebra_ipv4_routingTable'>
  <Function 'test_mpls_table'>
  <Function 'test_linux_mpls_routes'>
  <Function 'test_shutdown_check_stderr'>
  <Function 'test_shutdown_check_memleak'>
<Module 'ldp-vpls-topo1/test_ldp_vpls_topo1.py'>
  <Function 'test_ospf_convergence'>
  <Function 'test_rib'>
  <Function 'test_ldp_adjacencies'>
  <Function 'test_ldp_neighbors'>
  <Function 'test_ldp_bindings'>
  <Function 'test_ldp_pwid_bindings'>
  <Function 'test_ldp_pseudowires'>
  <Function 'test_ldp_pseudowires_after_link_down'>
  <Function 'test_memory_leak'>
<Module 'ospf-topo1/test_ospf_topo1.py'>
  <Function 'test_ospf_convergence'>
  <Function 'test_ospf_kernel_route'>
  <Function 'test_ospf6_convergence'>
  <Function 'test_ospf6_kernel_route'>
  <Function 'test_ospf_json'>
  <Function 'test_ospf_link_down'>
  <Function 'test_ospf_link_down_kernel_route'>
  <Function 'test_ospf6_link_down'>
  <Function 'test_ospf6_link_down_kernel_route'>
  <Function 'test_memory_leak'>
<Module 'ospf6-topo1/test_ospf6_topo1.py'>
  <Function 'test_router_running'>
  <Function 'test_ospf6_converged'>
  <Function 'test_ospfv3_routingTable'>
  <Function 'test_linux_ipv6_kernel_routingTable'>
  <Function 'test_shutdown_check_stderr'>
  <Function 'test_shutdown_check_memleak'>
<Module 'rip-topo1/test_rip_topo1.py'>
  <Function 'test_router_running'>
  <Function 'test_converge_protocols'>
  <Function 'test_rip_status'>
  <Function 'test_rip_routes'>
  <Function 'test_zebra_ipv4_routingTable'>
  <Function 'test_shutdown_check_stderr'>
<Module 'ripng-topo1/test_ripng_topo1.py'>
  <Function 'test_router_running'>
  <Function 'test_converge_protocols'>
  <Function 'test_ripng_status'>
  <Function 'test_ripng_routes'>
  <Function 'test_zebra_ipv6_routingTable'>
  <Function 'test_shutdown_check_stderr'>
  <Function 'test_shutdown_check_memleak'>

Or more specific:

~/topotests# py.test --collect-only bgp_direct_to_bgp_vpn/
mkdir: cannot create directory ‘/tmp/topotests’: File exists
======================================================== test session starts =========================================================
platform linux2 -- Python 2.7.12, pytest-3.3.1, py-1.5.2, pluggy-0.6.0
rootdir: /root/topotests, inifile: pytest.ini
collected 0 items

Sorry, needs fixing.

louberger commented 6 years ago

@mwinter-osr yeah, I see how to refactor - will be a big change...

louberger commented 6 years ago

replaced by #56

NetDEF-CI commented 6 years ago

Continuous Integration Result: FAILED

See below for issues. CI System Testrun URL: https://ci1.netdef.org/browse/FRR-TOPOPR-78/

This is a comment from an EXPERIMENTAL automated CI system. For questions and feedback in regards to this CI system, please feel free to email Martin Winter - mwinter (at) opensourcerouting.org.