Pull request test passes on GitHub pull request page but fails when run locally

[sujay-kazi]

I have mentioned this problem here (http://itensor.org/support/2290/questions-about-checks-for-itensors-jl-pull-request) and here (https://github.com/ITensor/ITensors.jl/pull/434), but I have been asked to put it as an issue here.

Basically, I submitted the above pull request, and all of the tests have passed. However, I have noticed that, when I run the tests locally, there is a particular test that fails--it is the line @test str[1] == "MPS" in mps.jl (line 15). Here is the error message:

MPS Basics: Test Failed at c:\Users\user\ITensors.jl\test\mps.jl:16
  Expression: str[1] == "MPS"        
   Evaluated: "ITensors.MPS" == "MPS"

For additional information, I have seen this test failure at every individual commit I have placed for that pull request. I am running this on my desktop, which is a 64-bit computer using Windows 10. When I ran the tests locally, I ran each individual file on its own in Visual Studio Code.

It is also worth mentioning that all of the files gave me this warning:

Warning: Package VSCodeDebugger does not have ITensors in its dependencies:
│ - If you have VSCodeDebugger checked out for development and have
│   added ITensors as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with VSCodeDebugger
└ Loading ITensors into VSCodeDebugger from project dependency, future warnings for VSCodeDebugger are suppressed.

Some of them ran without incident afterward, but some of them then gave me this error (perhaps with "QuadGK" replaced with some other package name):

ERROR: LoadError: LoadError: ArgumentError: Package VSCodeDebugger does not have QuadGK in its dependencies:
- If you have VSCodeDebugger checked out for development and have
  added QuadGK as a dependency but haven't updated your primary  
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with VSCodeDebugger

I believe this is just some issue I have in VSCode, but I just thought I should mention it so that you have the full information about the way I have been running the tests locally.

[emstoudenmire]

Thank you! We will try to reproduce this. It may be something to do with running each test individually instead of through runtests.jl, but I'll have to check about that. We do want the testing files to be individually runnable so there's nothing wrong with you doing that.

[sujay-kazi]

UPDATE: I ran runtests.jl as well, and I confirm that running test files individually shouldn't be the problem here. When I did this, I got 2 failed tests and 6 errors during tests.

The 2 failed tests were just like the one mentioned above--one was the aforementioned one (line 15 of mps.jl), and the other one said @test str[1] == "MPO" (line 26 of mpo.jl) and generated an identical error message but with "MPS" replaced by "MPO."

The 6 broken tests are due to the error with the package VSCodeDebugger on my end, so you shouldn't see those. But for your reference, here are the 6 tests, along with the package that VSCode Debugger doesn't have in its dependencies: contract.jl --> Combinatorics combiner.jl --> Combinatorics trg.jl --> QuadGK ctmrg.jl --> QuadGK readwrite.jl --> HDF5 examples.jl --> Suppressor

[mtfishman]

Out of curiosity, when you run all of the tests, are you doing julia>] test ITensors or are you doing julia> include("runtests")? If you are only doing one of them, could you try the other one and let us know if you get different results?

If the main issue is the MPS printing method test, I would vote to just remove that test anyway.

[sujay-kazi]

I was actually doing neither of the two: I was clicking on the "Run Without Debugging" option under the "Run" tab on Visual Studio Code (this is how I ran each test file individually and how I ran runtests.jl to run them all at once).

I decided to follow your instructions above in the Julia REPL, and I actually got different results, which adds to the confusion. When I used your first suggestion (julia>] test ITensors), I actually got no failed tests at all (see the first 2 attached screenshots). However, when I used your second suggestion (include("C:/Users/user/ITensors.jl/test/runtests.jl")), I got even more errors than before (see the next 10 attached screenshots for all of the errors). After combing through these errors, I am confident that none have anything to do with my modifications to the code. A number of them seem to be about a package not being in the current path, which I imagine can be fixed fairly easily, and I think some of them (in-place operations in broadcast.jl) are the same as before, but some of them are new.

[sujay-kazi]

Also, to add to this weirdness a little, I also have a separate copy of ITensors that is not a GitHub repository, i.e. I installed it using the "add" command rather than the "git clone" command. When I ran the tests on them (include("C:/Users/user/.julia/packages/ITensors/tgmxZ/test/runtests.jl")), I now got far fewer errors, even though I made changes in this folder so that all of the code matched up. In fact, the only errors that persisted were the errors about a certain package not being found in the current path. I have attached a screenshot for this one too.

[mtfishman]

Thanks for trying that out, that is helpful. I think there are a few potential issues going on.

1. I can see from the first output that the test is using NDTensors v0.1.6. The latest version of ITensors should be using a more recent version (the latest registered version, ITensors v0.1.14, relies on NDTensors v0.1.11). Are you sure you have the most up to date versions of ITensors and NDTensors? Is it possible you need to merge the master branch into your local branch? What happens when you run julia>] status ITensors and julia>] status NDTensors?

2. The tests rely on some test-only dependencies, like Combinatorics and QuadGK. These would get automatically installed when you do julia>] test ITensors, but not when you do julia> include("runtests.jl"). To do julia> include("runtests.jl"), you would have to install them manually. You can see which packages the tests rely on here: https://github.com/ITensor/ITensors.jl/blob/master/Project.toml#L33. It looks like maybe you would need to add HDF5 as well, though I'm not sure why since that should be installed when you install ITensors, that could be related to having too old of a version.

3. I haven't used Visual Studio Code, maybe when you run them in that way it is running it in a strange environment, which is leading to the MPS printing test failing (that's just a guess). I'm curious what happens when you fix issues 1. and 2. above and run it in the same way. If the only issue left is the MPS printing test, we can just remove that one.

[mtfishman]

Just saw your latest post, it sounds like the issue was that you didn't have the latest version of ITensors from your clone.

In general, I would recommend developing ITensors by first doing julia>] add ITensors and then doing julia>] dev ITensors. This creates a development version of ITensors (a git clone) in the directory ~/.julia/dev/ITensors, and you can develop in a branch there. You can switch back to the non-development version with julia>] free ITensors. You should also make sure to pull the latest versions of ITensors when you are in the directory ~/.julia/dev/ITensors to make sure everything is up to date.

I'm not sure how this workflow translates to use Visual Studio Code, however. I started a section of the docs that is meant to be a guide for developing ITensors here, I should probably add more to that. Could you make some suggestions about what would be helpful to have there, particularly in relation to developing using Visual Studio Code? I can add some pointers about the best ways to test the code and make sure it is up to date with the latest master branch.

[mtfishman]

I think this issue is more of a documentation issue, about the best practices for developing and testing ITensors. So I'll close in favor of #289.