astropy helpers does not seem to support --coverage

[aphearin]

This Issue is regarding https://github.com/aphearin/halotools/commit/5746bf9e0270c5ad2e97328feeafb3d5b682725e

There are a couple of things I don’t understand about this commit that I’m hoping either @embray or @astrofrog can shed some light on. All I’ve done is update the helpers to 1.1.1, and I notice two things:

coverage problem:

When I run python setup.py test —coverage, I get the following error message:

WARNING: The coverage option is ignored on run_tests, since it can not be made to work in that context. Use 'python setup.py test --coverage' instead. [astropy.tests.helper]

Since the suggested syntax is the exact same syntax I used, I’m confused about how to proceed.

Test suite report:

The report at the end of the test suite is very different before and after this commit. Before updating the helpers, on my local machine my test report reads 320 passed in 121.10 seconds.

The report after this commit reads 258 passed in 118.45 seconds.

The numbers will be different if you were to run the test suite on your own machines because there are many tests that are marked such that they only run on my personal machine. Why do these numbers not agree?

[aphearin]

Sorry @astrofrog and @embray , it would have been more helpful if I'd included the full traceback:

platform darwin -- Python 2.7.11 -- pytest-2.5.1

Running tests with Astropy version 1.0.4.
Running tests in halotools.

Platform: Darwin-13.4.0-x86_64-i386-64bit

Executable: /Users/aphearin/anaconda/bin/python

Full Python Version: 
2.7.11 |Anaconda 2.3.0 (x86_64)| (default, Dec  6 2015, 18:57:58) 
[GCC 4.2.1 (Apple Inc. build 5577)]

encodings: sys: ascii, locale: UTF-8, filesystem: utf-8, unicode bits: 15
byteorder: little
float info: dig: 15, mant_dig: 15

Numpy: 1.9.3
Scipy: 0.15.1
Matplotlib: 1.4.2
h5py: 2.5.0

plugins: cache, cov, pep8, xdist
collected 258 items 

halotools/empirical_models/model_helpers.py ..
halotools/empirical_models/assembias_models/tests/test_assembias.py .
halotools/empirical_models/composite_models/hod_models/hearin15.py .
halotools/empirical_models/composite_models/hod_models/leauthaud11.py .
halotools/empirical_models/composite_models/hod_models/tinker13.py .
halotools/empirical_models/composite_models/hod_models/zheng07.py .
halotools/empirical_models/composite_models/sfr_models/smhm_binary_sfr.py .
halotools/empirical_models/composite_models/smhm_models/behroozi10.py .
halotools/empirical_models/composite_models/tests/test_preloaded_models.py ..
halotools/empirical_models/factories/hod_model_factory.py .
halotools/empirical_models/factories/mock_factory_template.py ..
halotools/empirical_models/factories/model_factory_template.py ..
halotools/empirical_models/factories/prebuilt_model_factory.py .
halotools/empirical_models/factories/subhalo_model_factory.py .
halotools/empirical_models/factories/tests/test_hod_factory.py ...
halotools/empirical_models/factories/tests/test_mock_factories.py .
halotools/empirical_models/factories/tests/test_prebuilt_subhalo_model_factory.py ..
halotools/empirical_models/factories/tests/test_subhalo_model_factory.py ..
halotools/empirical_models/occupation_models/leauthaud11_components.py ...
halotools/empirical_models/occupation_models/tinker13_components.py ..
halotools/empirical_models/occupation_models/zheng07_components.py ......
halotools/empirical_models/occupation_models/tests/test_hod_components.py ..
halotools/empirical_models/occupation_models/tests/test_leauthaud11_hod.py ..
halotools/empirical_models/phase_space_models/nfw_phase_space.py ..........
halotools/empirical_models/phase_space_models/profile_models/conc_mass_models.py .
halotools/empirical_models/phase_space_models/profile_models/nfw_profile.py ...........
halotools/empirical_models/phase_space_models/profile_models/trivial_profile.py .
halotools/empirical_models/phase_space_models/profile_models/tests/test_analytic_profile_subclasses.py .
halotools/empirical_models/phase_space_models/profile_models/tests/test_conc_mass.py ..
halotools/empirical_models/phase_space_models/profile_models/tests/test_halo_catalog_nfw_consistency.py ..
halotools/empirical_models/phase_space_models/profile_models/tests/test_nfw_profile.py .....
halotools/empirical_models/phase_space_models/profile_models/tests/test_profile_helpers.py ....
halotools/empirical_models/phase_space_models/profile_models/tests/test_trivial_profile.py .
halotools/empirical_models/phase_space_models/tests/test_phase_space.py .........
halotools/empirical_models/phase_space_models/velocity_models/tests/test_nfw_isotropic_jeans.py .
halotools/empirical_models/sfr_models/sfr_components.py .
halotools/empirical_models/sfr_models/tests/test_sfr_components.py .
halotools/empirical_models/smhm_models/scatter_models.py .
halotools/empirical_models/smhm_models/tests/test_redshift_defensiveness.py .
halotools/empirical_models/smhm_models/tests/test_smhm_components.py .....
halotools/empirical_models/tests/test_model_helpers.py .
halotools/mock_observables/angular_tpcf.py .
halotools/mock_observables/delta_sigma.py .
halotools/mock_observables/error_estimation_tools.py ..
halotools/mock_observables/groups.py .
halotools/mock_observables/isolation_criteria.py ....
halotools/mock_observables/marked_tpcf.py .
halotools/mock_observables/mock_survey.py ..
halotools/mock_observables/nearest_neighbor.py .
halotools/mock_observables/pairwise_velocity_stats.py ....
halotools/mock_observables/rp_pi_tpcf.py .
halotools/mock_observables/s_mu_tpcf.py .
halotools/mock_observables/tpcf.py .
halotools/mock_observables/tpcf_jackknife.py .
halotools/mock_observables/tpcf_multipole.py .
halotools/mock_observables/tpcf_one_two_halo_decomp.py .
halotools/mock_observables/void_stats.py ..
halotools/mock_observables/wp.py .
halotools/mock_observables/pair_counters/double_tree.py .
halotools/mock_observables/pair_counters/double_tree_pair_matrix.py ....
halotools/mock_observables/pair_counters/double_tree_pairs.py ....
halotools/mock_observables/pair_counters/double_tree_per_object_pairs.py .
halotools/mock_observables/pair_counters/test_pair_counters/test_double_tree_pairs.py ........
halotools/mock_observables/pair_counters/test_pair_counters/test_marked_double_tree_pairs.py ......
halotools/mock_observables/pair_counters/test_pair_matrices/test_pair_matrix.py ....
halotools/mock_observables/pair_counters/test_trees/test_single_tree.py .
halotools/mock_observables/test_clustering/test_delta_sigma.py .
halotools/mock_observables/test_clustering/test_marked_tpcf.py ..
halotools/mock_observables/test_clustering/test_rp_pi_tpcf.py ....
halotools/mock_observables/test_clustering/test_s_mu_tpcf.py ..
halotools/mock_observables/test_clustering/test_tpcf.py ......
halotools/mock_observables/test_clustering/test_tpcf_jackknife.py ..
halotools/mock_observables/test_clustering/test_tpcf_one_two_halo.py ..
halotools/mock_observables/test_clustering/test_wp.py ....
halotools/mock_observables/test_groups/test_fof_groups.py ...
halotools/mock_observables/test_mock_survey/test_mock_survey.py ..
halotools/sim_manager/cached_halo_catalog.py .
halotools/sim_manager/download_manager.py .
halotools/sim_manager/user_supplied_halo_catalog.py .
halotools/sim_manager/tests/test_cached_halo_catalog.py ..........
halotools/sim_manager/tests/test_download_manager.py s.sssss..sss
halotools/sim_manager/tests/test_halo_table_cache.py ....
halotools/sim_manager/tests/test_halo_table_cache_log_entry.py ...........
halotools/sim_manager/tests/test_ptcl_table_cache_log_entry.py ..........
halotools/sim_manager/tests/test_supported_sims.py .
halotools/sim_manager/tests/test_tabular_ascii_reader.py ......
halotools/sim_manager/tests/test_user_supplied_halo_catalog.py ...
halotools/tests/test_amurrica.py .
halotools/utils/array_utils.py .....
halotools/utils/match.py .
halotools/utils/table_utils.py ...
halotools/utils/tests/test_array_utils.py ..
halotools/utils/tests/test_crossmatch.py .
halotools/utils/tests/test_table_utils.py ...

======================================= 249 passed, 9 skipped in 137.25 seconds ========================================
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/aphearin/anaconda/lib/python2.7/site-packages/astropy/tests/helper.py", line 320, in _save_coverage
    cov._harvest_data()
AttributeError: 'Coverage' object has no attribute '_harvest_data'
aphearin (sim_manager_overhaul) halotools $ 

[astrofrog]

@aphearin - this is due to a bug in the coverage package. The workaround is to use coverage==3.7.1. This isn't a problem in other affiliated packages because most have switched to using ci-helpers (https://github.com/astropy/ci-helpers) which explicitly specifies that version of the coverage package.

If you want to update halotools to use ci-helpers, see the latest .travis.yml file in the package template.

(ccing @bsipocz)

[aphearin]

Thanks, @astrofrog . From the github page looks like all I need to do is just overwrite my existing .travis.yml file with the one in the latest package-template. Do I have that right? (Making the appropriate packagename --> halotools changes, of course)

[bsipocz]

And to check the dependencies and set them in CONDA_DEPENDENCIES, etc.

[bsipocz]

alternatively I can open a PR in the weekend to do this.

[aphearin]

@bsipocz - a PR to the package-template or to Halotools?

[bsipocz]

@aphearin - I meant halotools. However checking it out now, it seems that you already use ci-helpers, so the coverage should work as is (and indeed it does on travis).

[aphearin]

I just checked, and it works when I downgrade to 3.7.1.

[aphearin]

Not sure what the discrepancy is about with v4.x, @bsipocz, but I'm perfectly happy with this as a workaround unless you advise otherwise.

[bsipocz]

Sure, I think I misunderstood this anyway. It was only a local issue and not something on travis. I should really stop multitasking and read through the issues first before starting commenting ;)

[aphearin]

Right, yes, local issue. When I throw --coverage, the coverage report does not build and the test report raises an exception at the end. When I locally downgrade coverage from 4.0 to 3.7.1, the issue is resolved.

[aphearin]

So I'm happy to consider the coverage report aspect of this issue resolved. But @astrofrog or @bsipocz - as for the second issue, should I worry about the significant changes to the number-counts of the tests executed? Do you know what's going on here?

[aphearin]

So I think I might have figured out the issue causing the number of performed tests to be reduced when I updated the helpers. I think it is related to https://github.com/astropy/astropy-helpers/issues/143 and https://github.com/astropy/astropy-helpers/pull/184. @bsipocz and @embray - are y'all sure this Issue got resolved? Maybe it got resolved in the latest version of the package-template, but since I started building my package over a year ago, my repo is still subject to these bugs?

The reason I think this is the issue is illustrated in the following commit: https://github.com/aphearin/halotools/commit/e99aaee3b43fd1b09b2a615993920fb5e3f2e3dd

[aphearin]

Closing this issue now. The remaining problem is now covered in its own Issue #308 .