Code Health

[U8NWXD]

Note: This depends on #22.

Code Style

• Get all code to pass lint checks

  • [x] core/composition.py
  • [x] core/control.py
  • [ ] core/emitter.py
  • [x] core/experiment.py
  • [ ] core/store.py
  • [x] core/process.py
  • [ ] core/registry.py

• Complete docstrings

  • [ ] core/composition.py
  • [ ] core/control.py
  • [ ] core/emitter.py
  • [ ] core/experiment.py
  • [ ] core/store.py
  • [x] core/process.py
  • [ ] core/registry.py

How to Use Pylint

To run pylint, just execute pylint vivarium.

The files left to fix are listed in .pylintrc after ignore=. We use an exclusion list to ensure that once we fix a file, any code added to it also passes the linter. This prevents regressions. It also ensures that any new files added have proper style.

I use pylint's default rules. You can adjust this in .pylintrc after the disable= line. You can find the list of pylint checks here.

Fixing a File

1. Delete the file from the list in .pylintrc
2. Run pylint by executing pylint vivarium
3. Fix all the errors that are shown
4. Repeat steps 2-3 until pylint finds no errors

Test Coverage

• Reach 100% code coverage
  • [x] core/composition.py
  • [x] core/control.py
  • [ ] core/emitter.py
  • [ ] core/experiment.py
  • [ ] core/store.py
  • [ ] core/process.py
  • [ ] core/registry.py

How to Calculate Coverage

I use the pytest-cov plugin to calculate test coverage. Whenever you run the tests, add the --cov=vivarium argument to calculate coverage. At the end of the test output, you'll get a table like this:

Name                                                  Stmts   Miss  Cover   Missing
-----------------------------------------------------------------------------------
vivarium/__init__.py                                     37      0   100%
vivarium/composites/__init__.py                           0      0   100%
vivarium/composites/injected_glc_phosphorylation.py      13      0   100%
vivarium/core/__init__.py                                 0      0   100%
vivarium/core/composition.py                            403    180    55%   62, 64-68, 141, 143, 156-157, 179-193, 200-221, 247-333, 347, 364-370, 375-386, 417, 445-456, 527-540, 543-544, 548-552, 559-569, 578-580, 614-643, 686-706, 746-755, 998-1000
vivarium/core/control.py                                 85     13    85%   40, 42, 59-60, 63-64, 89-90, 104-106, 112, 119, 211
vivarium/core/emitter.py                                156     80    49%   32-33, 55, 59, 65, 70-73, 86-94, 103, 116, 119, 128, 176-189, 192-196, 199, 203-212, 216-222, 226-227, 232-280, 284-298, 304-312, 320-324
vivarium/core/experiment.py                            1018    104    90%   93, 102, 171, 173, 182-184, 191, 245, 250-251, 262, 267, 271-272, 285-286, 288, 293, 311, 327-328, 381-384, 416, 437, 442, 447, 452, 460-467, 481, 485, 487, 490, 500-502, 520-523, 665-672, 827-831, 855-872, 909, 920, 922, 948-949, 990-997, 1014, 1069, 1076, 1096, 1348-1352, 1461-1462, 1494-1495, 1517, 1546, 1548, 1570-1576, 1677, 2254
vivarium/core/process.py                                277     40    86%   30, 41, 43, 51, 78-81, 100-104, 211, 227, 274, 277-279, 319, 337, 353-361, 370, 386, 392-394, 401, 457, 459, 507, 516, 564
vivarium/core/registry.py                               118     57    52%   77-78, 115-122, 146-153, 164, 182-197, 202-204, 212, 227, 238, 246, 249, 253, 256, 260-264, 271-275, 284-286, 289, 294-296, 300
vivarium/experiments/__init__.py                          0      0   100%
vivarium/experiments/glucose_phosphorylation.py          48      5    90%   91-96, 105
vivarium/library/__init__.py                              0      0   100%
vivarium/library/datum.py                                28     19    32%   2-3, 6-7, 28-41, 44, 47, 50
vivarium/library/dict_utils.py                          163     75    54%   17-20, 33-36, 38, 53, 68, 70, 74, 94, 113-117, 127-132, 136-144, 150-152, 156-175, 183-202, 206, 230-236, 259-261
vivarium/library/fasta.py                                10      8    20%   4-12
vivarium/library/filepath.py                             35     22    37%   22-25, 31-34, 51-58, 73-77, 81-82, 87-89, 93-94
vivarium/library/make_network.py                         87     79     9%   15-27, 33-40, 55-92, 96-125, 136-171
vivarium/library/path.py                                  4      2    50%   4-5
vivarium/library/pretty.py                               24      5    79%   16, 18, 20-21, 26
vivarium/library/schema.py                               39     28    28%   4, 7, 12, 18-25, 28-40, 45-52, 55, 62, 69, 77
vivarium/library/timeseries.py                           32     27    16%   27-50, 61-69
vivarium/library/topology.py                            148     17    89%   36-37, 58-61, 93-98, 107-116
vivarium/library/units.py                                38      6    84%   62-66, 90
vivarium/plots/__init__.py                                0      0   100%
vivarium/plots/agents_multigen.py                       135    127     6%   13-24, 64-226
vivarium/plots/simulation_output.py                     103     16    84%   70, 76, 78-79, 82, 84, 99-103, 122, 129, 132, 138, 172
vivarium/plots/topology.py                              105     17    84%   40, 137-140, 151, 166-180, 187, 202, 255, 268-277
vivarium/processes/__init__.py                            0      0   100%
vivarium/processes/agent_names.py                        14      6    57%   12-14, 17, 29-30
vivarium/processes/burst.py                              58      6    90%   208-212, 217
vivarium/processes/derive_concentrations.py              28     19    32%   20-31, 34, 53-69
vivarium/processes/derive_counts.py                      24      8    67%   32, 35, 55-62
vivarium/processes/divide_condition.py                   15      7    53%   10-11, 14, 17, 25-28
vivarium/processes/engulf.py                             58      6    90%   183-187, 191
vivarium/processes/exchange_a.py                         15      0   100%
vivarium/processes/glucose_phosphorylation.py            39      5    87%   129-139
vivarium/processes/injector.py                           37      9    76%   57, 103-109, 113
vivarium/processes/meta_division.py                      76      9    88%   29, 52, 65, 207-211, 215
vivarium/processes/nonspatial_environment.py             27     15    44%   22-24, 27-72, 75-87
vivarium/processes/remove.py                             47      6    87%   116-120, 124
vivarium/processes/swap_processes.py                     63      6    90%   175-179, 183
vivarium/processes/template_process.py                   38      7    82%   138-146, 151
vivarium/processes/timeline.py                           57      5    91%   43-44, 51, 56-57
vivarium/processes/tree_mass.py                          50      7    86%   69, 168-172, 176
-----------------------------------------------------------------------------------
TOTAL                                                  3752   1048    72%

Required test coverage of 72.0% reached. Total coverage: 72.07%

This lists the fraction of lines in each file that are covered by the tests. It also tells you which lines in the file weren't covered so you can add tests for them.

Notice the last line says we require a code coverage of 72%. This is just the current coverage value. If coverage dips below 72%, the tests will fail to make sure we don't regress.

How to Improve Coverage

1. Run the tests with pytest --cov=vivarium.
2. Pick a file that is not completely covered (coverage < 100%) using the coverage output.
3. Match the lines listed as not being covered to the source file. These are the lines you need to write tests for.
4. Add tests that will check that the uncovered lines are working correctly.
5. Run the tests again. You should get a coverage value of 100% for that file.
6. Update the minimum coverage in .coveragerc to your current coverage value for the entire project.

Static Typing

• Annotate all code with types and resolve all issues raised by mypy
  • [x] core/composition.py
  • [x] core/control.py
  • [ ] core/emitter.py
  • [ ] core/experiment.py
  • [x] core/process.py
  • [ ] core/registry.py

How to Use Mypy

We use type annotations as specified by PEP 484 and prefer annotations to special comments like # type: str. See the documentation for typing for more information.

Mypy is configured using the mypy.ini file, which looks like this:

[mypy]

disallow_untyped_defs = True
disallow_incomplete_defs = True

# Ignore missing types from third-party libraries.
[mypy-numpy.*]
ignore_missing_imports = True
...

# Ignore missing types from Vivarium Core files we haven't typed yet.
[mypy-vivarium.core.registry.*]
disallow_untyped_defs = False
disallow_incomplete_defs = False

[mypy-vivarium.core.control.*]
disallow_untyped_defs = False
disallow_incomplete_defs = False
...

Notice how we first set disallow_untyped_defs and disallow_incomplete_defs to True. This means that by default, mypy will raise an error whenever a function is not completely typed. Then, for every file that hasn't been typed yet, we include a section like [mypy-vivarium.core.registry.*] for vivarium/core/registry.py that sets those options to False. As we add type hints, we can remove files from this list until everything has type annotations.

Also note that we tell mypy to ignore types in our dependencies like numpy. Many third-party libraries are poorly-typed, and we want to avoid spending time trying to get mypy to work with them. We will focus on making sure the types in our code are correct instead.

How to Add Types

1. Pick a file listed in mypy.ini as not being covered yet.
2. Delete the file's section from mypy.ini.
3. Run mypy by executing mypy vivarim
4. Fix any errors raised by mypy
5. Repeat 3-4 until no errors are raised

[1fish2]

Tips:

• PEP 563 makes a small semantic change to Python annotations that will become mandatory in Python 3.10. Ideally we should make the annotations compatible with that now and keep them that way as we fill them out. We can use from __future__ import annotations to ensure they're good with it now, or it might suffice to run PyCharm inspections set to check for Python 3.10 compatibility.
  • I'll have to re-read it but IIRC it won't change any of our type hints. The point of the change is to defer processing of the annotations so that (1) they can more easily handle forward references to classes and (2) they avoid taking the time to process annotations when "just" loading code to run it. The change impacts cases like using annotations for RPC protocol definitions, database queries, etc.
• It'd be nice to make PyCharm inspections run clean on the code, even if that means letting it add some comments to ignore some inspections in some locations.

[1fish2]

PR #43 adds runscripts to run the static checkers (so we don't have to remember how) and also fixes some checks.