search

mozilla / PollBot

Robots for checking if things have been delivered
https://pollbot.services.mozilla.com/v1
Mozilla Public License 2.0
9 stars 21 forks source link

Unit tests don't mock the network requests #233

Open peterbe opened 5 years ago

peterbe commented 5 years ago

If you switch off your WiFi and run the whole test suite this happens:

Results (11.58s):
     154 passed
      38 failed
         - tests/test_views.py:390 test_nightly_archive[pyloop]
         - tests/test_views.py:399 test_release_archive[pyloop]
         - tests/test_views.py:409 test_candidate_archive[pyloop]
         - tests/test_views.py:420 test_candidate_archive_build[pyloop]
         - tests/test_views.py:431 test_beta_archive[pyloop]
         - tests/test_views.py:441 test_devedition_archive[pyloop]
         - tests/test_views.py:451 test_esr_archive[pyloop]
         - tests/test_views.py:461 test_release_partner_repacks[pyloop]
         - tests/test_views.py:470 test_candidate_partner_repacks_build[pyloop]
         - tests/test_views.py:479 test_candidate_partner_repacks[pyloop]
         - tests/test_views.py:488 test_beta_partner_repacks[pyloop]
         - tests/test_views.py:497 test_release_balrog_rules[pyloop]
         - tests/test_views.py:505 test_release_buildhub[pyloop]
         - tests/test_views.py:514 test_candidates_buildhub[pyloop]
         - tests/test_views.py:523 test_candidates_buildhub_build[pyloop]
         - tests/test_views.py:532 test_devedition_buildhub[pyloop]
         - tests/test_views.py:541 test_release_bedrock_release_notes[pyloop]
         - tests/test_views.py:549 test_devedition_bedrock_release_notes[pyloop]
         - tests/test_views.py:557 test_release_bedrock_esr_release_notes[pyloop]
         - tests/test_views.py:565 test_release_bedrock_security_advisories[pyloop]
         - tests/test_views.py:573 test_release_bedrock_download_links[pyloop]
         - tests/test_views.py:582 test_devedition_bedrock_download_links[pyloop]
         - tests/test_views.py:592 test_release_bouncer_download_links[pyloop]
         - tests/test_views.py:602 test_devedition_bouncer_download_links[pyloop]
         - tests/test_views.py:612 test_release_product_details[pyloop]
         - tests/test_views.py:620 test_devedition_product_details[pyloop]
         - tests/test_views.py:649 test_esr_balrog_rules[pyloop]
         - tests/test_views.py:657 test_beta_balrog_rules[pyloop]
         - tests/test_views.py:665 test_devedition_balrog_rules[pyloop]
         - tests/test_views.py:673 test_nightly_balrog_rules[pyloop]
         - tests/test_views.py:681 test_firefox_releases_list[pyloop]
         - tests/test_views.py:688 test_devedition_releases_list[pyloop]
         - tests/test_views.py:704 test_heartbeat[pyloop]
         - tests/test_views.py:741 test_ongoing_versions_view_firefox[pyloop]
         - tests/test_views.py:751 test_ongoing_versions_view_devedition[pyloop]
         - tests/test_views.py:757 test_endpoint_have_got_cache_control_headers[pyloop-/v1/__heartbeat__]
         - tests/test_views.py:777 test_get_buildid_for_version[pyloop]
         - tests/test_views.py:782 test_get_buildid_for_nightly_version[pyloop]

That's because those 38 tests depend on actually going out on the network and going to URLs like https://www.mozilla.org/en-US/firefox/58.0beta/releasenotes/

That means the test suite (at least those ~38 tests) aren't unit tests really. They're end-to-end tests.

I'm not worried about the traffic burden on www.mozilla.org and product-details.mozilla.org etc. I'm worried that soon those URLs will expire and tests will fail and nobody will understand how or why.

mostlygeek commented 5 years ago

This is fine. IMO we really care if we can go to mozilla.org and see if can scrape out the information we need. We don't really care if we got networking/GET requests right.

Since mozilla.org is not tightly coupled to pollbot, if they change their html or anything that affects our scraping I think the tests should just fail. That'll be a good trigger for us to go fix / change something.

peterbe commented 5 years ago

The point is that the tests are going to stop working soon. In mysterious ways. Considering that PollBot poorly swallows errors it's going to be a pain to comprehend why CI stopped working one day.

peterbe commented 5 years ago

By the way, I think an end-to-end test is quite nice. Every URL that Delivery Dashboard hits should probably be hit too. E.g.

for product in test_config.all_supported_products:
  for api_url_template in test_config.all_api_url_templates:
     url = api_url_template.format(product)
     response = requests.get(url)
     assert response.status_code == 200

Except, this wouldn't work anyway because of this ;)

willkg commented 5 years ago

Just as a thing, mozilla.org redesigned their /firefox/all/ page and the tests now fail. With the existing setup, that's the only signal we got that things changed and PollBot is probably broken.

I'm not sure what "The point is that the tests are going to stop working soon" means. What's going to cause them to stop working? Changes to mozilla.org and other things that PollBot uses? Isn't that a good thing?

I think I want to WONTFIX this.