Error in reset session

[Reasat]

In the start a session page, I keep facing this error when I press the reset session button

Traceback (most recent call last):
  File "run_model_server.py", line 1153, in <module>
    run()
  File "run_model_server.py", line 174, in run
    dset = dataset.Dataset(dataSetPath)
  File "/src/gitrepo/predict-rest-api/dataset.py", line 19, in __init__
    self.f = h5py.File(path, 'r')
  File "/usr/local/lib/python2.7/dist-packages/h5py/_hl/files.py", line 408, in __init__
    swmr=swmr)
  File "/usr/local/lib/python2.7/dist-packages/h5py/_hl/files.py", line 173, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 88, in h5py.h5f.open
IOError: Unable to open file (file read failed: time = Thu Oct 29 09:41:48 2020
, filename = '/datasets/myproject/', file descriptor = 4, errno = 21, error message = 'Is a directory', buf = 0x7ffea18242c0, total read size = 8, bytes this sub-read = 8, bytes actually read = 18446744073709551615, offset = 0)

After the first encounter of the error, every time I run python run_model_server.py the code keeps crashing with the same error. Which file is the code trying to read in this portion?

This is my project directory structure

[slee172]

Hi @Reasat The error message you've encountered in the start session is "Unable to open file" which means that you missed some of the files in your directory. In order to launch HistomicsML2, the files should be correctly located in the directory. See more details in here https://histomicsml2.readthedocs.io/en/latest/data-import.html After seeing your project directory structure, I found that you missed some files (a dataset file (i.e. HistomicsML_dataset.h5) and a pca model(i.e. pca_model_sample.pkl). Have you created the dataset file before? If then, you should move the file under the "myproject" directory. You can see more details here https://histomicsml2.readthedocs.io/en/latest/data-create.html

[Reasat]

Hello Sanghoon Thank you for pointing out the missing .h5 file! Could you tell me which command in https://histomicsml2.readthedocs.io/en/latest/data-create.html creates the HistomicsML_dataset.h5 file?

Here's what I understood about the function of the scripts, CreateSlideInformation.py --> slide_info.csv create_tiff.sh --> convert ‘.svs’ to ‘.tif’ SuperpixelSegmentation.py --> boundaries and centroids FeatureExtraction.py --> what is the output file and where is it located? Is it the HistomicsML_dataset.h5 file?

[Reasat]

Actually, I can't find the output files after running the FeatureExtraction.py script I think it ran without any error. But I am not sure if the extraction ended successfully. Here's the terminal output.

(base) E:\HistomicsML\myproject>docker run -it --rm --name extractfeatures -v E:\HistomicsML\myproject:/myproject cancerdatascience/hml_dataset_cpu:1.0 python scripts/FeatureExtraction.py --projectName myproject
INFO:root:Created LRU Cache for 'tilesource' with 10 maximum size
INFO:root:Notice: Could not import MapnikTileSource
INFO:matplotlib.font_manager:font search path ['/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf', '/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/afm', '/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/pdfcorefonts']
INFO:matplotlib.font_manager:generated new fontManager
/usr/local/lib/python2.7/dist-packages/nimfa/examples/all_aml.py:118: UserWarning: Matplotlib must be installed to run ALL AML example.
  warn("Matplotlib must be installed to run ALL AML example.")
/usr/local/lib/python2.7/dist-packages/nimfa/examples/medulloblastoma.py:109: UserWarning: Matplotlib must be installed to run Medulloblastoma example.
  warn("Matplotlib must be installed to run Medulloblastoma example.")
/usr/local/lib/python2.7/dist-packages/nimfa/examples/cbcl_images.py:92: UserWarning: Matplotlib must be installed to run CBCL images example.
  warn("Matplotlib must be installed to run CBCL images example.")
/usr/local/lib/python2.7/dist-packages/nimfa/examples/documents.py:111: UserWarning: Matplotlib must be installed to run Documents example.
  warn("Matplotlib must be installed to run Documents example.")
/usr/local/lib/python2.7/dist-packages/nimfa/examples/orl_images.py:104: UserWarning: Matplotlib must be installed to run ORL images example.
  warn("Matplotlib must be installed to run ORL images example.")
/usr/local/lib/python2.7/dist-packages/nimfa/examples/recommendations.py:55: UserWarning: Matplotlib must be installed to run Recommendations example.
  warn("Matplotlib must be installed to run Recommendations example.")
/usr/local/lib/python2.7/dist-packages/nimfa/examples/gene_func_prediction.py:68: UserWarning: Matplotlib must be installed to run Gene Function prediction example.
  warn("Matplotlib must be installed to run Gene Function prediction example.")
Using TensorFlow backend.
WARNING:ctk_cli.module:Required element 'step' not found within 'constraints'
WARNING:ctk_cli.module:Required element 'step' not found within 'constraints'
WARNING:ctk_cli.module:Required element 'step' not found within 'constraints'
WARNING:ctk_cli.module:Required element 'step' not found within 'constraints'
WARNING:ctk_cli.module:Required element 'name' not found within 'double'
WARNING:ctk_cli.module:Required element 'step' not found within 'constraints'
WARNING:ctk_cli.module:Required element 'name' not found within 'double'
WARNING:ctk_cli.module:Required element 'name' not found within 'double'
WARNING:ctk_cli.module:Required element 'step' not found within 'constraints'

>> CLI Parameters ...

Namespace(channelSize=3, columnSize=1, compactness=50, fcn=4096, inputPCAModel=None, max_mag=40.0, max_tile_size=16384.0, min_fgnd_frac=0.001, min_fgnd_superpixel=10, min_var_superpixel=0.0015, num_threads_per_worker=1, num_workers=-1, patchSize=128, patchSizeResized=224, pca_dim=64, pca_sample_scale=0.1, projectName='myproject', reference_mu_lab=[8.63234435, -0.11501964, 0.03868433], reference_std_lab=[0.57506023, 0.10403329, 0.01364062], rg_ratio_superpixel=1.2, sample_fraction=0.1, scheduler='', superpixelSize=64)

>> Reading VGG pre-trained model ...

Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels.h5
553467904/553467096 [==============================] - 37s 0us/step
553476096/553467096 [==============================] - 37s 0us/step
2020-10-29 20:14:53.642801: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-10-29 20:14:54.174961: W tensorflow/core/framework/allocator.cc:122] Allocation of 411041792 exceeds 10% of system memory.
2020-10-29 20:14:54.926386: W tensorflow/core/framework/allocator.cc:122] Allocation of 411041792 exceeds 10% of system memory.
2020-10-29 20:15:06.590785: W tensorflow/core/framework/allocator.cc:122] Allocation of 411041792 exceeds 10% of system memory.

[slee172]

@Reasat

1. The output of the dataset will be located in the current directory where you ran the docker.
2. The command for creating the dataset is here https://histomicsml2.readthedocs.io/en/latest/data-create.html#generate-features-and-pca-transformation
3. In order to create the dataset, you should follow the instruction https://histomicsml2.readthedocs.io/en/latest/data-create.html
   • First of all, it seems like you are using CPUs. I would recommend you to use GPUs rather than using CPUs. Creating a dataset with multiple whole slide images may require a lot of computational resources. But, you can do the work using CPUs in the case of testing with a small-size slide. We've already created two dockers supporting both CPUs and GPUs.
   • Second, in order to create the dataset, you need to follow the instruction. 1) Download dockers https://histomicsml2.readthedocs.io/en/latest/data-create.html#download-the-dataset-creation-container 2) Create directories https://histomicsml2.readthedocs.io/en/latest/data-create.html#create-directories 3) Create slide information https://histomicsml2.readthedocs.io/en/latest/data-create.html#create-slide-information-table 4) Convert svs-formatted slides to tiff-formatted https://histomicsml2.readthedocs.io/en/latest/data-create.html#convert-whole-slide-images 5) Create boundaries and centroids https://histomicsml2.readthedocs.io/en/latest/data-create.html#generate-superpixel-segmentation Note that this step will use multiprocessors on your machine. This step will create the boundary and centroid files under the boundary and centroid directories. You have to confirm that the files are under the directories after completing this step. 6) Create a dataset https://histomicsml2.readthedocs.io/en/latest/data-create.html#generate-features-and-pca-transformation
   • Third and hopefully the last, you can confirm the dataset under the "myproject" directory.

Let me close this issue since the issue is not about the error in the reset session.