Tests are broken for packaging

[danielquinn]

I wrote a PR for Gentoo to roll in the latest version of Magellan, and it was rejected because the unit tests didn't pass, which is interesting because they do pass... once the package is installed. However, if you're running the tests pre-install (as is typical for packaging systems) then a few blow up. Details are available here, but the short-version is that we're testing for the output of a command that doesn't exist in $PATH yet, so stuff is breaking.

The @idella made some suggestions on that ticket as to how this might best be addressed, so I'm going to leave it to you lot to decide which is the best option. As for his other input about Gentoo-specific stuff (stabilisation requests and removing old packages) don't worry about that as I have it covered.

[danielquinn]

Any movement on this? I tried again to package this today and everything keeps failing. See ebuild.log.txt

[astrikos]

@danielquinn I opted for something close to option 2. I don't really have a mean to test it for gentoo, so I created a new branch with proposed solution. Could you please run gentoo test stuff against this branch and see if it solves our problem. If it does I will create a PR against this branch and we can merge.

[danielquinn]

I won't be able to do this today on account of having a day job ;-) but I'll see what I can do this weekend.

[astrikos]

no worries, I just let you know that I have something, and you can work on it based on your availability.

[danielquinn]

Unfortunately, there's no win there. There's still 5 failing tests. The output is below:

======================================================================
ERROR: User passes no args, should fail with SystemExit
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/tmp/portage/net-analyzer/ripe-atlas-tools-2.0.2/work/ripe.atlas.tools-2.0.2/tests/commands/report.py", line 68, in test_with_empty_args
    self.cmd.run()
  File "/var/tmp/portage/net-analyzer/ripe-atlas-tools-2.0.2/work/ripe.atlas.tools-2.0.2-python2_7/lib/ripe/atlas/tools/commands/report.py", line 181, in run
    use_regular_file
  File "/var/tmp/portage/net-analyzer/ripe-atlas-tools-2.0.2/work/ripe.atlas.tools-2.0.2-python2_7/lib/ripe/atlas/tools/commands/report.py", line 243, in _get_results_from_file
    sample = next(self.file)
StopIteration

======================================================================
FAIL: Tests autocompletion of specific command.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/tmp/portage/net-analyzer/ripe-atlas-tools-2.0.2/work/ripe.atlas.tools-2.0.2/tests/test_bash_completion.py", line 57, in test_command_completion
    self.assertTrue("measure" in output)
AssertionError: False is not true
-------------------- >> begin captured stdout << ---------------------
(u'', u'\nNo such command.\n\n')

--------------------- >> end captured stdout << ----------------------

======================================================================
FAIL: Tests autocompletion of commands.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/tmp/portage/net-analyzer/ripe-atlas-tools-2.0.2/work/ripe.atlas.tools-2.0.2/tests/test_bash_completion.py", line 39, in test_commands_completion
    self.assertTrue("report" in output)
AssertionError: False is not true
-------------------- >> begin captured stdout << ---------------------
(u'', u'\nUsage: ripe-atlas <measurements|configure|render|probe|measurement|go|measure|report|stream|probes> [arguments]\n\n')

--------------------- >> end captured stdout << ----------------------

======================================================================
FAIL: Tests autocompletion of specific option of a command.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/tmp/portage/net-analyzer/ripe-atlas-tools-2.0.2/work/ripe.atlas.tools-2.0.2/tests/test_bash_completion.py", line 71, in test_option_completion
    self.assertEqual(output, "--help")
AssertionError: u"Usage: ripe-atlas measure [-h] [--renderer {aggregate_ping,dns,dst_asn,http,ntp,ping,raw,ssl_consistency,sslcert,traceroute,traceroute_aspath}] [--dry-run] [--auth auth] [--af {4,6}] [--description description] [--target target] [--no-report] [--interval interval]\n                          [--from-area {ww,west,north-central,south-central,north-east,south-east} | --from-country country | --from-prefix prefix | --from-asn asn | --from-probes probes | --from-measurement measurement_id] [--probes probes] [--include-tag tag]\n                          [--exclude-tag tag] [--packets packets] [--size size] [--packet-interval packet_interval]\n\nCreate a measurement and optionally wait for the results\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --renderer {aggregate_ping,dns,dst_asn,http,ntp,ping,raw,ssl_consistency,sslcert,traceroute,traceroute_aspath}\n                        The renderer you want to use. If this isn't defined, an appropriate renderer will be selected.\n  --dry-run             Do not create the measurement, only show its definition.\n  --auth AUTH           The API key you want to use to create the measurement\n  --af {4,6}            The address family, either 4 or 6\n  --description DESCRIPTION\n                        A free-form description\n  --target TARGET       The target, either a domain name or IP address.  If creating a DNS measurement, the absence of this option will imply that you wish to use the probe's resolver.\n  --no-report           Don't wait for a response from the measurement, just return the URL at which you can later get information about the measurement.\n  --interval INTERVAL   Rather than run this measurement as a one-off (the default), create this measurement as a recurring one, with an interval of n seconds between attempted measurements. This option implies --no-report.\n  --from-area {WW,West,North-Central,South-Central,North-East,South-East}\n                        The area from which you'd like to select your probes.\n  --from-country COUNTRY\n                        The two-letter ISO code for the country from which you'd like to select your probes. Example: --from-country=GR\n  --from-prefix PREFIX  The prefix from which you'd like to select your probes. Example: --from-prefix=82.92.0.0/14\n  --from-asn ASN        The ASN from which you'd like to select your probes. Example: --from-asn=3333\n  --from-probes PROBES  A comma-separated list of probe-ids you want to use in your measurement. Example: --from-probes=1,2,34,157,10006\n  --from-measurement MEASUREMENT_ID\n                        A measurement id which you want to use as the basis for probe selection in your new measurement.  This is a handy way to re-create a measurement under conditions similar to another measurement. Example: --from-measurement=1000192\n  --probes PROBES       The number of probes you want to use.  Defaults to 50,unless --from-probes is invoked, in which case the number of probes selected is used.\n  --include-tag TAG     Include only probes that are marked with these tags. Example: --include-tag=system-ipv6-works\n  --exclude-tag TAG     Exclude probes that are marked with these tags. Example: --exclude-tag=system-ipv6-works\n\nPing-specific Options:\n  --packets PACKETS     The number of packets sent\n  --size SIZE           The size of packets sent\n  --packet-interval PACKET_INTERVAL\n" != '--help'
-------------------- >> begin captured stdout << ---------------------
(u"Usage: ripe-atlas measure [-h] [--renderer {aggregate_ping,dns,dst_asn,http,ntp,ping,raw,ssl_consistency,sslcert,traceroute,traceroute_aspath}] [--dry-run] [--auth auth] [--af {4,6}] [--description description] [--target target] [--no-report] [--interval interval]\n                          [--from-area {ww,west,north-central,south-central,north-east,south-east} | --from-country country | --from-prefix prefix | --from-asn asn | --from-probes probes | --from-measurement measurement_id] [--probes probes] [--include-tag tag]\n                          [--exclude-tag tag] [--packets packets] [--size size] [--packet-interval packet_interval]\n\nCreate a measurement and optionally wait for the results\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --renderer {aggregate_ping,dns,dst_asn,http,ntp,ping,raw,ssl_consistency,sslcert,traceroute,traceroute_aspath}\n                        The renderer you want to use. If this isn't defined, an appropriate renderer will be selected.\n  --dry-run             Do not create the measurement, only show its definition.\n  --auth AUTH           The API key you want to use to create the measurement\n  --af {4,6}            The address family, either 4 or 6\n  --description DESCRIPTION\n                        A free-form description\n  --target TARGET       The target, either a domain name or IP address.  If creating a DNS measurement, the absence of this option will imply that you wish to use the probe's resolver.\n  --no-report           Don't wait for a response from the measurement, just return the URL at which you can later get information about the measurement.\n  --interval INTERVAL   Rather than run this measurement as a one-off (the default), create this measurement as a recurring one, with an interval of n seconds between attempted measurements. This option implies --no-report.\n  --from-area {WW,West,North-Central,South-Central,North-East,South-East}\n                        The area from which you'd like to select your probes.\n  --from-country COUNTRY\n                        The two-letter ISO code for the country from which you'd like to select your probes. Example: --from-country=GR\n  --from-prefix PREFIX  The prefix from which you'd like to select your probes. Example: --from-prefix=82.92.0.0/14\n  --from-asn ASN        The ASN from which you'd like to select your probes. Example: --from-asn=3333\n  --from-probes PROBES  A comma-separated list of probe-ids you want to use in your measurement. Example: --from-probes=1,2,34,157,10006\n  --from-measurement MEASUREMENT_ID\n                        A measurement id which you want to use as the basis for probe selection in your new measurement.  This is a handy way to re-create a measurement under conditions similar to another measurement. Example: --from-measurement=1000192\n  --probes PROBES       The number of probes you want to use.  Defaults to 50,unless --from-probes is invoked, in which case the number of probes selected is used.\n  --include-tag TAG     Include only probes that are marked with these tags. Example: --include-tag=system-ipv6-works\n  --exclude-tag TAG     Exclude probes that are marked with these tags. Example: --exclude-tag=system-ipv6-works\n\nPing-specific Options:\n  --packets PACKETS     The number of packets sent\n  --size SIZE           The size of packets sent\n  --packet-interval PACKET_INTERVAL\n", u'')

--------------------- >> end captured stdout << ----------------------

======================================================================
FAIL: Tests autocompletion of existing options for a command.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/tmp/portage/net-analyzer/ripe-atlas-tools-2.0.2/work/ripe.atlas.tools-2.0.2/tests/test_bash_completion.py", line 64, in test_options_completion
    self.assertEqual(output, "dns http ntp ping sslcert traceroute")
AssertionError: u'' != 'dns http ntp ping sslcert traceroute'
-------------------- >> begin captured stdout << ---------------------
(u'', u'\nFor extended options for a specific measurement type, try ripe-atlas measure <type> --help.\n\n')

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 148 tests in 4.530s

But I have good news: I've figured out how to reproduce it so you can find out what's breaking where. The following list of instructions should reproduce all of the above except for the erroring test (StopIteration) I'm not sure what's causing that one:

git clone git@github.com:RIPE-NCC/ripe-atlas-tools.git
cd ripe-atlas-tools
git checkout gentooTestsFix
virtualenv --python python2 virtualenv
. virtualenv/bin/activate
python setup.py test

The StopIteration thing is a little weirder though. the gentooTestsFix branch is missing one line in it that is in the 2.0.1 tag:

Renderer.add_arguments_for_available_renderers(self.parser)

Though I'm not sure if that's related or not. Let's just see if we can get the other stuff fixed first?

[chrisamin]

So I think this just works these days?

Ran 153 tests in 24.122s

OK camin-pro git ripe-atlas-tools gentooTestsFix /

[danielquinn]

I no longer maintain the Gentoo package (hell, I don't even use Gentoo anymore) so unless there are objections from someone else, I think it's probably safe to close this.