Unit tests fail for pip-installed gpfit

[pgkirsch]

Unit tests currently fail on a fresh pip installation for two reasons:

1. It uses some pre-release GPkit features (initsolwarnings).
   • This will be resolved by GPkit release 1.0
   • Current solution is to pip install a local version of latest GPkit master
2. 4 tests write output to a file that doesn't exist

   ======================================================================
   ERROR: test_hoburgabbeel_ex6_3_mosek_cli (gpfit.tests.t_examples.TestExamples)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
   File "/Users/philippe.kirschen/Optimization/gpkit/gpkit/tests/helpers.py", line 55, in test
   testfn(name, import_dict, path)(self)
   File "/Users/philippe.kirschen/Optimization/gpkit/gpkit/tests/helpers.py", line 86, in test
   with StdoutCaptured(logfilepath=filepath):
   File "/Users/philippe.kirschen/Optimization/gpkit/gpkit/tests/helpers.py", line 155, in __enter__
   sys.stdout = (open(self.logfilepath, mode="w")
   FileNotFoundError: [Errno 2] No such file or directory: '/opt/anaconda3/envs/test_gpfit/lib/python3.9/site-packages/docs/source/examples/hoburgabbeel_ex6_3_output.txt'

[whoburg]

Does #105 fix this?

[pgkirsch]

No, I don't think it will. Thank you for the reminder.

[pgkirsch]

@bqpd do you have suggestions for unit tests that use gpkit.tests.helpers.generate_example_tests when the example/test files live in a subdirectory of docs which is not included as a package inside GPfit. Is the only appropriate solution to move the docs dir into gpfit/gpfit/?

[bqpd]

I might just skip the docs/example tests when it's a PyPi install (see t_examples.py) :grimacing: bad practice, I know

[bqpd]

There might be the alternate option of including docs as a data directory, using the package_data attribute

[pgkirsch]

Thanks @bqpd this was the nudge I needed. I used include_package_data=True added a MANIFEST.in.

[bqpd]

ahaha glad to hear you're doing it the right way!

[pgkirsch]

Although doesn't seem to have worked... (Thanks @galbramc for retesting)

[pgkirsch]

I tested locally when I made the change and the tar-zip included the directories it was supposed to, so I'm kinda confused.

[pgkirsch]

@galbramc sorry to ask more of you, but I have a few questions about Jenkins to try to understand what's going on. My problem seems to be that the directory tests/artifacts/ doesn't exist in the version installed on Macys and Reynolds, despite being included in my locally generated dist/ tarballs.

1. What is the significance of '--omit=gpfit/tests/*' in the line below? Is it relevant to the issue I am describing?

   ++ python /Users/jenkins/workspace/CE_gpfit_PR_unit_tests/buildnode/macys/venv2_gpfit/bin/coverage run --source=gpfit '--omit=gpfit/tests/*' test.py

2. I re-tested a locally pip-installed version of the code and realized I wasn't including a package gpfit.maths in setup.py. This caused my tests to fail locally and that's why I made the change in 11013ad, but I'm confused why those tests weren't failing on Jenkins. Do you know why that would be the case?
3. Similarly, prior to that last commit (which didn't change any core code) I was failing two tests that are no longer failing. The tests should be deterministic now so I'm confused why this would be happening.
4. It looks like pylint tests were passing prior to this last commit but now I see the following in the console output:

   FileNotFoundError: [Errno 2] No such file or directory: '/Users/jenkins/Library/Caches/pylint'

[galbramc]

1. The --omit=gpfit/tests/* simply means don't include the testing files when generating coverage information. The coverage information tells you how much of your code is actually getting tested, and the testing files them selves artifcially inflate the numbers.
2. I have no idea why the testes were not failing due to the missing gpfit.maths...
3. I've noticed there are a couple of tests in gpfit that do not seem to be deterministic. I'm guessing they use random numbers and some random numbers give you passing tests, while other random numbers give you failing tests.
4. I don't see that pylint error in the console output. Could you point me to the specific build? Jenkins was previously not properly catching the pylint errors. Make sure you have this line in the gpfit pylintrc file: msg-template={path}:{module}:{line}: [{msg_id}({symbol}), {obj}] {msg}

[galbramc]

It looks like something is wrong with pylint.sh in the gpfit repo. Can you run it locally on your machine?

[galbramc]

Can you also update README.md with:

[galbramc]

Well I didn't know github would replace the text with strings. You can grab the "unprotected Markdown" code from here:

https://acdl.mit.edu/csi/view/convex%20engineering/job/CE_gpfit_Push_pylint/badge/ https://acdl.mit.edu/csi/view/convex%20engineering/job/CE_gpfit_Push_unit_tests/badge/

[pgkirsch]

Thanks @galbramc.

1. Ah I see
2. Is there something in the logs (or that I could add to gpfit) that would help me understand what is being installed on the VMs? It makes me think it was using a cached build or something, but maybe that's not possible and I'm just fundamentally misunderstanding something. This would also explain why it can't find artifacts/.
3. Hmm I thought I had made all of them deterministic now by specifying the seed.
4. This latest build: https://acdl.mit.edu/csi/job/CE_gpfit_PR_pylint/118/ Line 89 of the pylintrc is msg-template={path}:{line}: [{msg_id}({symbol}), {obj}] {msg} which seems to be missing the {module} part -- I will update this.

[pgkirsch]

pylint.sh seems to run fine for me locally:

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

[pgkirsch]

Updated the README, thanks!

[pgkirsch]

To expand upon my response to 2 above

Is there something in the logs (or that I could add to gpfit) that would help me understand what is being installed on the VMs?

I am seeing things like:

Checking out Revision 03b40d235eb9a32ccd05b8e69d038a5bcf45ebf6 (origin/pr/105/merge, origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 03b40d235eb9a32ccd05b8e69d038a5bcf45ebf6 # timeout=3
Commit message: "Merge 11013ade4147129f04d2cc5ec885cd2beb8f5cf0 into 930b2dc4bdabea0f44edcf02fc418806c41fb96a"

(where 11013 was latest commit of v1 and 930b2 is the latest commit of master) and

Successfully installed gpfit-0.2.0

that make me believe the right thing is being built but I'm still very confused why some of the behaviour seems to suggest outdated code is being tested.

[galbramc]

It turns out that pylint.sh was getting called in the "unit_tests", which it should not have been. That is fixed.

I don't know what the problem is with the artifacts directiry, but gpkit and gpfit are installed with the commands:

#Install GPkit
git clone --depth 1 https://github.com/convexengineering/gpkit.git $WORKSPACE/gpkit
python $PIP install -v --no-cache-dir -e $WORKSPACE/gpkit

# Install gpfit to run the documentation examples
python $PIP install -e $WORKSPACE/

gpfit is checked out in the $WORKSPACE directory. Does this help?

[galbramc]

I added a print statment to show you the current working directory in the script to hopefully help with the artifacts problem

[pgkirsch]

Unit tests now run on a clean pip-installed GPfit (v0.2.0)! 🎉

[20:46:59]$ pytest --pyargs gpfit
========================================== test session starts ===========================================
platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
Matplotlib: 3.4.3
Freetype: 2.6.1
rootdir: /Users/philippe.kirschen/Documents
plugins: mpl-0.13
collected 60 items                                                                                       

tests/test_constraint_set.py ..                                                                    [  3%]
tests/test_evaluate.py ............                                                                [ 23%]
tests/test_examples.py ss                                                                          [ 26%]
tests/test_fit.py ...........                                                                      [ 45%]
tests/test_initialize.py ...                                                                       [ 50%]
tests/test_least_squares.py ....                                                                   [ 56%]
tests/test_logsumexp.py ..............                                                             [ 80%]
tests/test_plot_fit.py .........                                                                   [ 95%]
tests/test_print_fit.py ...                                                                        [100%]

============================================ warnings summary ============================================
tests/test_constraint_set.py::TestFitConstraintSet::test_outside_bounds
  /Users/philippe.kirschen/opt/anaconda3/envs/gpfit020/lib/python3.9/site-packages/numpy/core/fromnumeric.py:86: UnitStrippedWarning: The unit of the quantity is stripped when downcasting to ndarray.
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=============================== 58 passed, 2 skipped, 1 warning in 41.32s ================================

The two skipped tests are the examples tests that can't be reached when docs/source/examples aren't in path.