search

SCIInstitute / ShapeWorks

ShapeWorks
http://sciinstitute.github.io/ShapeWorks/
Other
103 stars 32 forks source link

ShapeWorks 6.1 Testing #1273

Closed jadie1 closed 3 years ago

jadie1 commented 3 years ago

Please edit and add a ✅ indicating success and ❌ indicating failure or 🕒 for a test in progress with your username when you complete a task for a given platform.

When a test fails, please add a github issue, add it to the 6.1 release, and link it (*the issue when it's fixed and ready to test again). Also, go ahead and add new tasks that might not already be on here.

Please use the most recent release candidate for all testing (be careful which shapeworks is in your $PATH). The most recent is RC14: https://github.com/SCIInstitute/ShapeWorks/releases/tag/v6.1.0-rc14

Example:

Windows Mac Linux
Notebooks 🕒 (@archanasri - A)
Usecase: Ellipsoid ✅ (@akenmorris - A)
Usecase: All tiny-test ❌ (#1073) ✅ (@cchriste - A)
Systems chart Username System
(@HeavenlyBerserker - A) Ubuntu 20.04.2 LTS (Linux) [Binaries]
(@HeavenlyBerserker - B) Microsoft Windows 10 Education (Windows) [Release Branch]
(@iyerkrithika21 - A) Microsoft Windows 10 Home OS Build 19042.1052 (Windows) [Binaries]
(@archanasri - A) macOS Big Sur 11.4 (OSX) [Binaries]
(@cchriste - A) maxOS Big Sur 11.3.1 (OSX)
(@cchriste - B) openSUSE 15.1 (Linux)
(@cchriste - C) (Windows)
(@akenmorris - A) Windows 10
(@jadie1 - A) openSUSE 15.0 (Linux)
(@username - B) System (System Type)

Instructions for running and testing items

Testing modes:

  1. Binary downloads
    • Go to https://github.com/SCIInstitute/ShapeWorks, click releases, click the system you are testing, download and unzip the first asset.
    • Set your paths to the release binaries instead of your regular paths. Make sure python scripts are running them.
  2. Release/master branch
    • Fetch, checkout master, and pull; Make sure it builds. If you're on your usual repository directory, you might need to delete the dependencies directory and rerun source build_dependencies.sh.
  3. Clean installation
    • Find a different machine or create a VM and download the binaries (or clone branches and build if you're feeling industrious).
    • Run 1 or 2 items in each category.

General instructions

Notebooks

  1. Go to ShapeWorks/Examples/Python/notebooks/tutorials on terminal/CMD.
  2. Run the jupyter notebook command.
  3. Open whichever file you wish to test.
  4. Make sure it runs to completion and each input matches what the explanation says.

UseCases

  1. Go to ShapeWorks/Examples/Python/ on terminal/CMD or on your favorite IDE that runs python.
  2. Run the python RunUseCase.py --use_case <use_case> command, where is replaced by the command given. For instance python RunUseCase.py --use_case femur, or python RunUseCase.py --use_case femur --groom_images.
  3. Wait until it finishes. A Studio Analyze window should pop up. Make sure sampling, correspondence, and analyze->PCA->animate are reasonable.

Studio

  1. Go to ShapeWorks/Examples/Python/<item> on terminal/CMD, where "" is the item you're testing.
  2. Run the command ShapeWorksStudio <item>.xlsx.
  3. Click on "Run Groom" and make sure it's reasonable. Click on the "Optimize" tab, and click "Run Optimize". Go to the "Analyze" tab and make sure sampling, correspondence, and analyze->PCA->animate are reasonable.

Studio: MeshGrooming

  1. Download test_meshes.zip and unzip.
  2. Run command ShapeWorksStudio, or open ShapeWorksStudio somehow.
  3. "Start New Project" -> "+ to import shapes" -> Select all 4 downloaded shapes and open.
  4. Play with the mesh groom parameters and the views. Adjust parameters and make sure it still makes sense; e.g. check Groom->Smooth and adjust laplacian, then "Run Groom"; e.g. move around, click Center, increase viewers, run groom with other parameters, etc.

Ok, now the real thing!

Windows Mac Linux
Clean installation ❎ (@cchriste - C) both Anaconda and git-bash, default and user-selected install dirs, #1296 🕒(@HeavenlyBerserker - B dev instructions) ❎ (@cchriste - A) #1296) ✅ (@archanasri - A) (user instructions) ✅ (@jadie1 - A (user instructions - RC12))
Notebooks: getting-started-with-notebooks ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A) ✅ (@jadie1 - A (RC8))
Notebooks: getting-started-with-segmentations ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A) ❌(#1314)
Notebooks: getting-started-with-exploring-segmentations ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A) ❌(#1314)
Notebooks: getting-started-with-meshes ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A) ❌(#1314)
Notebooks: getting-started-with-data-augmentation ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) (with num_samples=5) ✅ (@cchriste - A) ✅ (@HeavenlyBerserker - A) ✅ (@jadie1 - A (RC8))
Notebooks: getting-started-with-shape-cohort-generation ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A) ✅ (@jadie1 - A (RC8))
Usecase: ellipsoid ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A) ✅ (@jadie1 - A (RC8))
Usecase: ellipsoid_cut ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A) ✅ (@jadie1 - A (RC8))
Usecase: ellipsoid_evaluate ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A) ✅ (@jadie1 - A (RC8))
Usecase: ellipsoid_pca ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A) ✅ (@jadie1 - A (RC8))
Usecase: ellipsoid_fd ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A) ✅ (@jadie1 - A (RC8))
Usecase: ellipsoid_mesh ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A) ✅ (@jadie1 - A (RC8))
Usecase: ellipsoid_multiple_domain ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A) ✅ (@jadie1 - A (RC8))
Usecase: ellipsoid_multiple_domain_mesh ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A)
Usecase: femur ✅ (@akenmorris - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A)
Usecase: femur --groom_images ✅ (@akenmorris - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A)
Usecase: femur_mesh ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A)
Usecase: femur_cut ✅ (@akenmorris - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A )
Usecase: femur_cut --groom_images ✅ (@akenmorris - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A)
Usecase: left_atrium ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A)
Usecase: left_atrium --groom_images ✅ (@cchriste - C) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A )
Usecase: lumps ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A)
Usecase: thin_cavity_bean ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A)
Usecase: supershapes_1mode_contour ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A)
Usecase: deep_ssm* ✅ (@cchriste - C) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A )
Studio: ellipsoid ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A)) ✅ (@HeavenlyBerserker - A )
Studio: FeatureMap ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A )
Studio: MeshGrooming ✅ (@iyerkrithika21 - A) ✅ (@archanasri - A) ✅ (@HeavenlyBerserker - A )
...

* Edit deep_ssm.py so that num_samples = 3 on line 76 and "epochs": 3, on line 132, otherwise it will take upwards of two days.

Please keep comments on this issue to a minimum. Let's try to keep the status in the table and not in the comments.

akenmorris commented 3 years ago

I think we can drop all the tiny-tests, no? GH Actions runs them on all platforms.

jadie1 commented 3 years ago

I think we can drop all the tiny-tests, no? GH Actions runs them on all platforms.

Agreed

akenmorris commented 3 years ago

What are the OSX and Windows issues? We lost the installers?

cchriste commented 3 years ago

What are the OSX and Windows issues? We lost the installers?

OSX doesn't have an installer (.app in the comment above), but Windows had one. You just have to click on Downloads to see it.

sheryjoe commented 3 years ago

@cchriste @akenmorris @jadie1 @iyerkrithika21 @archanasri @riddhishb @HeavenlyBerserker @jmozing2 @RichLisonbee

Where are we on the release testing?

HeavenlyBerserker commented 3 years ago

@sheryjoe I think we should assign specific testing tasks to each person. Otherwise, we'll witness diffusion of responsibility in play here.

sheryjoe commented 3 years ago

@HeavenlyBerserker As we did in the 6.0 release (#1073), the table in this issue should help each one to "proactively" start testing depending on the system(s) he/she has access to.

HeavenlyBerserker commented 3 years ago

I'll test all the use cases on Ubuntu 20.04.2 LTS today.

Diffusion of responsibility is a sociopsychological phenomenon whereby a person is less likely to take responsibility for action or inaction when other bystanders or witnesses are present. Considered a form of attribution, the individual assumes that others either are responsible for taking action or have already done so.

sheryjoe commented 3 years ago

I'll test all the use cases on Ubuntu 20.04.2 LTS today.

Can you edit the table to mark in-progress testing on Ubuntu to avoid duplicate efforts on a similar platform?

sheryjoe commented 3 years ago

@jmozing2 (the only ortho user with Mac) won't be able to test the release today. @MacUsers (@riddhishb @archanasri @cchriste @akenmorris) could you proceed with testing the release on Mac? Please update the table accordingly to avoid duplicate testing on a similar platform. Thanks.

akenmorris commented 3 years ago

I don't think we have a build yet. #1283 needs to be merged first. Otherwise none of the python stuff works at all, my understanding.

sheryjoe commented 3 years ago

@jadie1 could you test use cases and studio on OpenSUSE?

sheryjoe commented 3 years ago

@HeavenlyBerserker please test studio on Ubuntu.

sheryjoe commented 3 years ago

@iyerkrithika21 I recall that you have a window machine, could you take the lead on windows testing?

HeavenlyBerserker commented 3 years ago

I don't think we have a build yet. #1283 needs to be merged first. Otherwise none of the python stuff works at all, my understanding.

@akenmorris Should this be tackled first? When will this be ready? Should I test Studio in the meantime, or should I wait?

sheryjoe commented 3 years ago

@cchriste what is the status of #1283?

sheryjoe commented 3 years ago

I don't think we have a build yet. #1283 needs to be merged first. Otherwise none of the python stuff works at all, my understanding.

@akenmorris Should this be tackled first? When will this be ready? Should I test Studio in the meantime, or should I wait?

@HeavenlyBerserker I think you should be able to start testing Studio.

akenmorris commented 3 years ago

@sheryjoe , yes you can test Studio now. The artifacts from PR #1283 should work for Python. I'd like to get it merged and create an RC.

HeavenlyBerserker commented 3 years ago

@akenmorris Should I test master? There is no release_v6.1 branch.

akenmorris commented 3 years ago

@HeavenlyBerserker , sure.

akenmorris commented 3 years ago

1283 is merged. I am trying to get an RC1 built, but we're having a lot of problems with GitHub Actions today. Lots of jobs are failing with network problems or being "cancelled by the remote provider".

HeavenlyBerserker commented 3 years ago

How do we tell correctness?

Anything else we need to test for?

Also, we should include instructions for what to look for in tests in the future. Otherwise, I can only make sure it runs and the results are somewhat reasonable.

Edit: I added instructions to the first comment. Please let me know if we have to look for anything else.

cchriste commented 3 years ago

1280 (pr #1283) fixed and merged to master: conda_installs now installs the python module on all platforms.

Prove me wrong.

cchriste commented 3 years ago

Correctness should be typically objective and measurable, especially wrt unit testing. But we currently have many tests deemed successful if they simply finish without crashing.

Everyone: please file specific issues for things like this as you test.

On Mon, Jun 14, 2021 at 12:45 PM Hong Xu @.***> wrote:

How do we tell correctness?

  • I'm assuming that for Studio, we run grooming and optimize with default parameters and make sure the particle sampling, correspondence, and animate are reasonable?
  • For use cases, we just make sure sampling, correspondence, and animate are reasonable, right? Anything else we need to test for?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SCIInstitute/ShapeWorks/issues/1273#issuecomment-860910018, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJT3ELIRGQ4HR6ST32BXPDTSZE4FANCNFSM46KWIFZQ .

cchriste commented 3 years ago

1280 (pr #1283) fixed and merged to master: conda_installs now installs the python module on all platforms.

Prove me wrong.

Kudos to @akenmorris: #1297 I have a fix demonstrated locally, pull request coming soon. Clean installation testing for all platforms will need to be re-tested, along with at least the ability to execute RunUseCase for all platforms (the results will not be different; just want to make sure they can still import the python module).

cchriste commented 3 years ago

The new ShapeWorks installation is now ready to be tested (use rc3 tag). For the Python module It's much simpler and no longer requires duplication of libraries or clever modification of runpaths. A new parameter was added to RunUseCase.py, --shapeworks_source, that developers can use to ensure their module modifications are imported (they should also be sure to use --shapeworks_path to indicate the path to their build/bin directory).

jadie1 commented 3 years ago

Update from Tanner who is on windows and unable to edit the issue: Tested feature map and mesh grooming in studio and got the results in the attached pictures. I'm not sure if the feature map picture is just a visual bug (or if that's how its supposed to look), but in the mesh grooming test the particles didn't line up with the model. This occurred after changing some grooming parameters and rerunning it. image image

akenmorris commented 3 years ago

Thanks Tanner! I've created an issue for this: #1309

akenmorris commented 3 years ago

Updated links for RC3.

akenmorris commented 3 years ago

Update links for RC5.

jadie1 commented 3 years ago

@cchriste I followed the instructions for the Linux binary build for RC8 and it worked (use cases and notebooks ran succesfully). When I did the same thing for RC10 it installed successfully but when I ran a use case I got this error:

image

I do have the environment activated, not sure how to debug further.

akenmorris commented 3 years ago

@jadie1 , can you try RC12?

https://github.com/SCIInstitute/ShapeWorks/releases/tag/v6.1.0-rc12

I tried it and it worked for me. RC10 didn't work for me on Linux either.

jadie1 commented 3 years ago

@akenmorris RC12 worked for me as well

akenmorris commented 3 years ago

What's left?

sheryjoe commented 3 years ago

I added more docs on the getting-started/shape to cover good and bad surfaces and what is shape modeling.

sheryjoe commented 3 years ago

@akenmorris I added the tour video and the workshop videos (partitioned into three parts) to the docs. Nothing else for my side. Other than #1354, any thing else left for the release?

akenmorris commented 3 years ago

RC14 here:

https://github.com/SCIInstitute/ShapeWorks/releases/tag/v6.1.0-rc14

sheryjoe commented 3 years ago

Thanks all for your great job in getting the 6.1 release out! Happy to hit the close button for this issue.