search

DIDSR / eeDAP

Evaluation Environment for Digital and Analog Pathology
http://didsr.github.io/eeDAP/
Other
16 stars 20 forks source link

Run eeDAP as an NCIPhub tool #52

Open brandon-gallas opened 7 years ago

brandon-gallas commented 7 years ago

Hello everyone,

I think there might be a easier method.

We can edit our code to only generate xml file but not show WSI FOV. Then compile and install it on NCIPhub. Similar as our previous APP one NCIPhub, we can download and xml and output files from NCIPhub. And then we use imagescope to show the WSI with FOV annotations. In this case, we don't have to upload image to NCIPhub, and also don't need to worry about color. There might be a problem that it is a little hard to open imagescope automatically by NCIPhub.

Best wish from China, Qi

On Friday, August 25, 2017 12:01 PM, "Gallas, Brandon D." wrote:

Yes, NCIPhub is the desired place for developing and hosting this tool in the cloud. It’s FREE.

I am adding Qi to this thread. He is in China for another two-six weeks. He may have a delay to his trip because of his visa. We are delaying the trip to Memorial Sloan Kettering and the reader study until October. That said, it will still be nice if we can run this in the cloud sooner to test and maybe even collect data in digital mode.

Qi created a tool for the Matlab gui that I showed both of you. I think the NCIPhub server has a licensed version of Matlab. If they grant us space to upload big images, we are in business for digital mode with two shortcomings: • We won’t have a special feature, opening a third-party app because it only runs on Windows. • There will be some questions for us to deal with down the road regarding color.

Have a good weekend. Brandon

From: Berman, Benjamin * Sent: Friday, August 25, 2017 12:59 PM To: Gallas, Brandon D.; purva sawant Subject: RE: cloud version of eeDAP

Hi Brandon and Purva,

Briefly looking into running eeDAP from a container. As far as Docker compatibility is concerned, it seems there would be two directions:

  1. Compile eeDAP using Matlab Compiler (a Matlab product). Then install the Matlab Compiler Runtime (MCR) into the container along with eeDAP. From my understanding, this ends up being an overly large container (MCR is designed to handle all of Matlab’s capabilities).
  2. Generate eeDAP using Matlab Coder (another Matlab product) as C and C++ code. Then use a C compiler, and load the compiled eeDAP into the container. Could be doable, and much more portable, but I am sure there will be plenty of debugging to do.

Interesting to find on a few stack-overflow exchanges that Matlab developers are getting onboard this contained, portable code idea. I wouldn’t be surprised if we see yet another Matlab product within the next few years to help achieve this. Attended an NCI meeting on quantitative imaging a few months ago and everyone was talking about Docker.

Just wanted to add my 2 cents here. I haven’t looked into the NCIP hub, but I know that Brandon and Qi (when he gets back) have plenty of experience with that.

Have a nice weekend

-Ben

From: Gallas, Brandon D. Sent: Thursday, August 24, 2017 5:30 PM To: purva sawant Cc: Berman, Benjamin * Subject: RE: cloud version of eeDAP

Purva, Thanks for talking with me today.

I think we agreed that you would look into a plan to install eeDAP in a container that would serve the program from the cloud. While you are working with Amazon for Aldo’s project, I’m interested in seeing if the NCIP hub would work for us. It’s free and they are quite friendly since they are government (NCI) and they want more users on their site. They have always honored my requests for more space and often email me to urge me to create tools. Please start there.

Below are links to the eeDAP source and compiled app, documentation for producing tools at NCIPhub, and the NCIPhub main page. Notice that the documentation is at hubzero. NCIPhub is a specific server of that software hosted by the NCI. https://github.com/DIDSR/eeDAP https://hubzero.org/documentation/2.1.0/users/tools https://nciphub.org/ https://nciphub.org/resources/tools

For now, I’d like to know how long you think the development of this might take. If possible I would like to use it for a study Sept 11.

Let me know if you have any questions. Also Ben Berman is on the cc line. He has run eeDAP and can sometimes help with that.

Brandon

brandon-gallas commented 7 years ago

@qigongFDA, @purvaksawant, @bpberman

Below is Purva's 2 cents:

If we plan to go along cloud based GUI, it seems that the compiler is free of cost, but the server running the MATLAB codes will charge $0.18/hour: https://www.mathworks.com/products/parallel-computing/parallel-computing-on-the-cloud/distriben-ec2-pricing.html since we have to compute using MATLAB computing server only.

Creating a docker like container for a Matlab version is not credible, since we would not have to individually install different packages/environments for the same in a MATLAB environment.

If we want to run the code on the cloud and not be charged an extra amount, we might have want to shift to another language(python).

My github account name is : purvaksawant

-Purva

brandon-gallas commented 7 years ago

BDG adding Purva's 2 cents on the cloud based version of GUI:

If we plan to go along cloud based GUI, it seems that the compiler is free of cost, but the server running the MATLAB codes will charge $0.18/hour: https://www.mathworks.com/products/parallel-computing/parallel-computing-on-the-cloud/distriben-ec2-pricing.html since we have to compute using MATLAB computing server only.

Creating a docker like container for a Matlab version is not credible, since we would not have to individually install different packages/environments for the same in a MATLAB environment.

If we want to run the code on the cloud and not be charged an extra amount, we might have want to shift to another language(python).

My github account name is : purvaksawant

-Purva

brandon-gallas commented 7 years ago

In response to Qi,

Let's keep it simple and not worry about ImageScope (software to view and annotate digital pathology images) for now.

In response to Purva,

MATLAB is installed on the NCIPhub server.

I don't understand your comment on installing different packages/environments.

The development time to move to python would be enormous.

Brandon

purvaksawant commented 7 years ago

We would be likely to use a container when there are different languages of codes, and different packages. If the entire GUI is on MATLAB, I don;t see the need to create the application inside a container. Even the simplest container is at least 800MBs of space.

About NCIPhub: Do you mean we need to host the GUI on the NCIPhub here and have it's front end over the cloud(like a website GUI)? If so, we have a lot of firewalls which would stop the inbound traffic. Correct me if I am wrong.

Purva

brandon-gallas commented 7 years ago

We need to put eeDAP inside a container so that collaborators external to FDA can use it easily: no install, no license.

NCIPhub is the server that will host eeDAP. It is free. You saw it run. There are no firewall issues.

Brandon

purvaksawant commented 7 years ago

Will MCR not do that? The developer would need to distribute the MATLAB run time library (MCRInstaller.exe which the target computer needs to run/install), the executable file that is created by the compiler (YourApp.exe or whatever it's called), and any ancillary files that are needed such as splash images, documentation, sample data files etc. (because these are used at run-time and are not built in to the executable).

brandon-gallas commented 7 years ago

MCR will do that. However, we don't want to ask collaborators to install anything. They are clinicians, not engineers.

The NCIPhub server has Matlab, making life a little bit easier.

purvaksawant commented 7 years ago

Is the Imagescope compatible with 32 bit windows and with Windows server 12?

brandon-gallas commented 7 years ago

Imagescope is compatible with 32 bit windows. Not sure about Windows server 12.

brandon-gallas commented 6 years ago

We can run windows functions at NCIPhub. See #64

qigongFDA commented 6 years ago
  1. New branch linuxVersion for linux code
  2. Edit code for linux system and compile it. (imagescope doesn't work)
  3. Create a tool eedapc in NCIPhub.
  4. Upload source code and compiled file to NCIPhub

Waiting for NCIP hub help us to install the tool.

brandon-gallas commented 6 years ago

Good start. I have a few questions. What are the fundamental differences in eeDAP for linux? Can we isolate these differences and pull them out of the main code as separate functions? Why did you give eeDAP a different name on NCIPhub? There appears to be a "c" at the end, or it is a typo. Is there a link to the NCIP hub tool development progress?

B

qigongFDA commented 6 years ago

I have a meeting with NCIPhub supporter today. We discuss about matlab version, invoke file, and .sh file. I compile a new version .sh file. He will install the corresponding matlab MCR on the hub.

qigongFDA commented 6 years ago

Erich and I figure out the invoke file should looks like this.

!/bin/sh

/usr/bin/invoke_app "$@" -t eedapc \ -w captive \ -u matlab_mcr-R2015a \ -C "@tool/bin/run_eeDAP_cloud.sh /apps/share64/debian7/matlab_mcr/R2015a/v85"\ -c filexfer

qigongFDA commented 6 years ago

The Administration GUI can be displayed on the hub. But there are some problem.

  1. The hub doesn't like changing directory and keyboard functions.
  2. The working directory is not software file saved directory. In other word, when I click browse input file button it shows a "data/sessions" directory, which might be use for upload (download) file to (from) the hub. I can't find dapsi file.

I edit the code, compile and upload to the hub a new version. Hope they solve the first problem. I need help from Erich to understand and solve the second problem.

brandon-gallas commented 6 years ago

Interesting. You may want to discuss with Erich a strategy to allow a user (or maybe just us) to upload different study materials (images and input files). Of course, we also need to be able to download results.

qigongFDA commented 6 years ago

  1. We fixed the functions. When we run it on the hub there is no error.

  2. We should be careful about linux compiled .sh file. It uses ctrl + M (\r) to start a new line. However, the hub's system can't process it. Solution: a. Use Notepad++ open the .sh. b. Click Show All Characters button. c. In find, choose "Extended" and Replase "\r" by "".

  3. We are still working on space to save WSI and input files.

qigongFDA commented 6 years ago

eeDAP cloud is working on NCIP hub. It is waiting for publish. WSI, input, output, icc files are saved in /data/tools/eedapc. For now, only development group users have writing permission for this directory. Permitted user can use SFTP (WinSCP) to communicate with the directory. Host name: nciphub.org Port number: 22 User name: NCIPhub Username (not email) Passwords: password to login NCIPhub.

qigongFDA commented 6 years ago

Several edits for eeDAP cloud (keep updating)

  1. Disable MicroRT mode
  2. Change input file default browse directory to /data/tools/eedapc/inputFile.