Error Running example data in templates/2_Pixie_Cluster_Pixels.ipynb

[mikemcka]

Please refer to our FAQ and look at our known issues before opening a bug report.

Describe the bug A clear and concise description of what the bug is.

Hi, I have been trying to run template notebook 2_Pixie_Cluster_Pixels.ipynb using the example dataset loaded during the notebook, I get the error running Cell 3.2: Assign pixel SOM clusters:

---

ValueError Traceback (most recent call last) Cell In[15], line 2 1 # use pixel SOM weights to assign pixel clusters ----> 2 pixel_som_clustering.cluster_pixels( 3 fovs, 4 channels, 5 base_dir, 6 pixel_pysom, 7 data_dir=pixel_data_dir, 8 multiprocess=multiprocess, 9 batch_size=batch_size 10 ) 12 # generate the SOM cluster summary files 13 pixel_som_clustering.generate_som_avg_files( 14 fovs, 15 channels, (...) 19 pc_chan_avg_som_cluster_name=pc_chan_avg_som_cluster_name 20 )

File /vast/imaging/MikeyM/ark_env/lib/python3.11/site-packages/ark/phenotyping/pixel_som_clustering.py:208, in cluster_pixels(fovs, channels, base_dir, pixel_pysom, data_dir, multiprocess, batch_size, num_parallel_pixels, overwrite) 203 cols_to_drop.append(col) 205 sample_fov = sample_fov.drop( 206 columns=cols_to_drop 207 ) --> 208 misc_utils.verify_same_elements( 209 enforce_order=True, 210 norm_vals_columns=pixel_pysom.norm_data.columns.values, 211 pixel_data_columns=sample_fov.columns.values 212 ) 214 # ensure the SOM weights columns are valid indexes 215 misc_utils.verify_same_elements( 216 enforce_order=True, 217 pixel_som_weights_columns=pixel_pysom.weights.columns.values, 218 pixel_data_columns=sample_fov.columns.values 219 )

File /vast/imaging/MikeyM/ark_env/lib/python3.11/site-packages/alpineer/misc_utils.py:236, in verify_same_elements(enforce_order, warn, **kwargs) 225 warnings.warn( 226 err_str 227 % ( (...) 233 ) 234 ) 235 else: --> 236 raise ValueError( 237 err_str 238 % ( 239 list_one_name, 240 list_two_name, 241 list_one_cast[first_bad_index], 242 list_two_cast[first_bad_index], 243 first_bad_index, 244 ) 245 ) 246 return True

ValueError: Lists norm vals columns and pixel data columns ordered differently: values CD14 and CD3 do not match at index 0

I have not modified the code in any way prior to running the notebook,

I have attempted to get around the error so I can give the interactive GUI a try by downloading the example_pixel_output_dir.zip from the data repository (https://huggingface.co/datasets/angelolab/ark_example/tree/main/data/pixie) on hugging face and uploading it to my ark-analysis data file in place of the one you generate during the notebook. the following cells to 3.2 do run however the GUI is a blank object in section 4 onwards.

I also had some trouble using the install commands on github: git clone -b v0.7.0 https://github.com/angelolab/ark-analysis.git cd ark-analysis conda env create -f environment.yml

Running these commands throws an error and causes the build to fail when installing scikit image==0.19.3. I managed to install scikit-image seperately using conda and then using conda update with the .yaml file on the environment

I have attached a list of all the packages I have installed in my environment called requirements.txt.

Expected behavior A clear and concise description of what you expected to happen.

To Reproduce Please either copy/paste or screenshot all of the code you ran which produced the error. In addition, copy the full error message.

[mikemcka]

requirements.txt

[cliu72]

Hi @mikemcka! We have seen this issue before (https://github.com/angelolab/ark-analysis/issues/1120) and it should have been addressed in this pull request: https://github.com/angelolab/ark-analysis/pull/1114. In that same PR, we also updated the scikit-image version to be <0.19.3 because 0.19.3 was causing issues, which makes me think you may be using an older version of the codebase. In the requirements.txt file that you shared, it says "ark-analysis @ file:///vast/imaging/MikeyM/ark-analysis", so it's hard to confirm which version of ark you are using.

Can you try creating a completely new conda environment with python version 3.11 (conda create -n new_ark_env python=3.11). Then install ark-analysis using pip install ark-analysis (this should install the latest version, 0.7.1) and try running the notebook again. Let me know if this works!

[mikemcka]

Hi Candace,

I have followed the instructions and created the environment and reinstalled pixie following your previous reply and I got this this error:

error: Command "c++ -std=gnu11 -pthread -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -DENABLE_PYTHON_MODULE -DPYTHRAN=3 -DPYTHRAN_BLAS_NONE -I/tmp/pip-build-env-4e1jdgsu/overlay/lib/python3.11/site-packages/pythran -I/tmp/pip-build-env-4e1jdgsu/overlay/lib/python3.11/site-packages/numpy/_core/include -I/tmp/pip-build-env-4e1jdgsu/overlay/lib/python3.11/site-packages/numpy/_core/include -Ibuild/src.linux-x86_64-3.11/numpy/distutils/include -I/stornext/System/data/tools/jupyter/jupyter-latest/include -I/stornext/System/data/apps/python/python-3.11.4/include/python3.11 -c skimage/feature/brief_pythran.cpp -o build/temp.linux-x86_64-3.11/skimage/feature/brief_pythran.o -MMD -MF build/temp.linux-x86_64-3.11/skimage/feature/brief_pythran.o.d -std=c++11 -fno-math-errno -fvisibility=hidden -fno-wrapv -Wno-unused-function -Wno-int-in-bool-context -Wno-unknown-warning-option -fopenmp -msse -msse2 -msse3" failed with exit status 1 INFO: ########### EXT COMPILER OPTIMIZATION ########### INFO: Platform : Architecture: x64 Compiler : gcc

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2
    Generated   : none
  INFO: CCompilerOpt.cache_flush[864] : write cache to path ->

/tmp/pip-install-334ry6d4/scikit-image_f7ff0677a0b748a5923de9e7381889d2/build/temp.linux-x86_64-3.11/ccompiler_opt_cache_ext.py [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for scikit-image Building wheel for asciitree (pyproject.toml) ... done Created wheel for asciitree: filename=asciitree-0.3.3-py3-none-any.whl size=5033 sha256=cb02b5fa6ea574f7523883dc5966f9ebfedac4a9d5ae4a5804a1d058389f2904 Stored in directory: /stornext/Home/data/allstaff/m/mckay.m/.cache/pip/wheels/71/c1/da/23077eb3b87d24d6f3852ed1ed1a1ac2d3c885ad6ebd2b4a07 Successfully built asciitree Failed to build scikit-image ERROR: Could not build wheels for scikit-image, which is required to install pyproject.toml-based projects

Have you guys seen this before?

On Sat, Jul 6, 2024 at 8:38 AM Candace Liu @.***> wrote:

Hi @mikemcka https://github.com/mikemcka! We have seen this issue before (#1120 https://github.com/angelolab/ark-analysis/issues/1120) and it should have been addressed in this pull request: #1114 https://github.com/angelolab/ark-analysis/pull/1114. In that same PR, we also updated the scikit-image version to be <0.19.3 because 0.19.3 was causing issues, which makes me think you may be using an older version of the codebase. In the requirements.txt file that you shared, it says "ark-analysis @ file:///vast/imaging/MikeyM/ark-analysis", so it's hard to confirm which version of ark you are using.

Can you try creating a completely new conda environment with python version 3.11 (conda create -n new_ark_env python=3.11). Then install ark-analysis using pip install ark-analysis (this should install the latest version, 0.7.1) and try running the notebook again. Let me know if this works!

— Reply to this email directly, view it on GitHub https://github.com/angelolab/ark-analysis/issues/1142#issuecomment-2211476991, or unsubscribe https://github.com/notifications/unsubscribe-auth/A374V5QZ23D5Q6ZE75HIFK3ZK4N5HAVCNFSM6AAAAABJNGIQKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJRGQ3TMOJZGE . You are receiving this because you were mentioned.Message ID: @.***>

[cliu72]

Looks like there's an issue installing scikit-image (as you noted before). In that same environment, can you try pip install scikit-image==0.19.2 (the version here is important, .19.3 was causing issues), and then try pip install ark-analysis again?

[mikemcka]

Hi, I am still getting the same error with 0.19.2 cc1plus: warning: unrecognized command line option "-Wno-unknown-warning-option" [enabled by default] cc1plus: warning: unrecognized command line option "-Wno-int-in-bool-context" [enabled by default] error: Command "g++ -pthread -B /vast/imaging/MikeyM/new_ark_env/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /vast/imaging/MikeyM/new_ark_env/include -fPIC -O2 -isystem /vast/imaging/MikeyM/new_ark_env/include -fPIC -DENABLE_PYTHON_MODULE -DPYTHRAN=3 -DPYTHRAN_BLAS_NONE -I/tmp/pip-build-env-3308n73s/overlay/lib/python3.11/site-packages/pythran -I/tmp/pip-build-env-3308n73s/overlay/lib/python3.11/site-packages/numpy/_core/include -I/tmp/pip-build-env-3308n73s/overlay/lib/python3.11/site-packages/numpy/_core/include -Ibuild/src.linux-x86_64-3.11/numpy/distutils/include -I/vast/imaging/MikeyM/new_ark_env/include/python3.11 -c skimage/feature/brief_pythran.cpp -o build/temp.linux-x86_64-3.11/skimage/feature/brief_pythran.o -MMD -MF build/temp.linux-x86_64-3.11/skimage/feature/brief_pythran.o.d -std=c++11 -fno-math-errno -fvisibility=hidden -fno-wrapv -Wno-unused-function -Wno-int-in-bool-context -Wno-unknown-warning-option -fopenmp -msse -msse2 -msse3" failed with exit status 1 INFO: ########### EXT COMPILER OPTIMIZATION ########### INFO: Platform : Architecture: x64 Compiler : gcc

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2
    Generated   : none
  INFO: CCompilerOpt.cache_flush[864] : write cache to path ->

/tmp/pip-install-k2xn7zkl/scikit-image_1f08f65b7f354d7ea7ad64e4d531bc51/build/temp.linux-x86_64-3.11/ccompiler_opt_cache_ext.py [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for scikit-image Failed to build scikit-image ERROR: Could not build wheels for scikit-image, which is required to install pyproject.toml-based projects

On Tue, Jul 9, 2024 at 4:14 AM Candace Liu @.***> wrote:

Looks like there's an issue installing scikit-image (as you noted before). In that same environment, can you try pip install scikit-image==0.19.2 (the version here is important, .19.3 was causing issues), and then try pip install ark-analysis again?

— Reply to this email directly, view it on GitHub https://github.com/angelolab/ark-analysis/issues/1142#issuecomment-2214867873, or unsubscribe https://github.com/notifications/unsubscribe-auth/A374V5RNT7WSLQJQ2AUWI3DZLLJIHAVCNFSM6AAAAABJNGIQKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJUHA3DOOBXGM . You are receiving this because you were mentioned.Message ID: @.***>

[cliu72]

@mikemcka Hmm this seems to be an issue with installing scikit-image (not ark-analysis, though ark-analysis depends on scikit-image). From a quick google search (https://github.com/Sanster/IOPaint/issues/281, https://github.com/scikit-image/scikit-image/issues/6765), it could be related to the python version - however, on my Windows machine and with python version 3.11, I was able to install scikit-image with no problems. I haven't seen this exact error before. I would suggest posting in the scikit-image issues page (https://github.com/scikit-image/scikit-image/issues) - perhaps they could help you there.

[cliu72]

Oh actually, I see in your original issue that you were able to install scikit-image 0.19.3 using conda. Perhaps you could try that for scikit-image 0.19.2 as well, and then try installing ark-analysis.

[mikemcka]

Scikit-image 0.19.2 seems to be too old for the new environment however 0.19.3 installs using conda, is there a newer version of scikit that would work? Could not solve for environment specs The following packages are incompatible ├─ pin-1 is installable with the potential options │ ├─ pin-1 1 would require │ │ └─ python 3.11. , which can be installed; │ └─ pin-1 1 would require │ └─ python 3.11. , which can be installed; └─ scikit-image 0.19.2 is not installable because there are no viable options ├─ scikit-image 0.19.2 would require │ └─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported; ├─ scikit-image 0.19.2 would require │ └─ python >=3.7,<3.8.0a0 , which conflicts with any installable versions previously reported; ├─ scikit-image 0.19.2 would require │ └─ python >=3.8,<3.9.0a0 , which conflicts with any installable versions previously reported; └─ scikit-image 0.19.2 would require └─ python >=3.9,<3.10.0a0 , which conflicts with any installable versions previously reported.

On Tue, Jul 9, 2024 at 10:23 AM Candace Liu @.***> wrote:

Oh actually, I see in your original issue that you were able to install scikit-image 0.19.3 using conda. Perhaps you could try that for scikit-image 0.19.2 as well, and then try installing ark-analysis.

— Reply to this email directly, view it on GitHub https://github.com/angelolab/ark-analysis/issues/1142#issuecomment-2215717996, or unsubscribe https://github.com/notifications/unsubscribe-auth/A374V5Q7GQSZNSWXQOXGUQ3ZLMUQ7AVCNFSM6AAAAABJNGIQKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJVG4YTOOJZGY . You are receiving this because you were mentioned.Message ID: @.***>

[cliu72]

Unfortunately, we've only tested ark-analysis with scikit-image<0.19.3, so we can't guarantee that a newer version would work. However, ark-analysis should work with python 3.10, so you could try making a new environment with python 3.10 and try installing scikit-image 0.19.2 there.

[mikemcka]

Cool, I have reinstalled the environment with python=3.10 and scikit-image = 0.19.2 and I have run the code in notebook 2 successfully on the sample data up until section 4 where you visualise the clustering and running the code in the first cell generates an empty matplotlib figure. running the cell after throws an error FileNotFoundError: The file/path, pixel_meta_cluster_mapping.csv, could not be found... What I did: I used git clone -b v0.7.1 https://github.com/angelolab/ark-analysis.git I used conda create -n new_ark_env python=3.10 I activated the environment and cd to ark-analysis I then ran pip install ark-analysis I made sure to set up a kernel using ipykernel and running python -m ipykernel install --user --name=new_ark_env --display-name "Python (new_ark_env)" to select the kernel in the notebook

Thanks again for the help with this

On Wed, Jul 10, 2024 at 3:57 AM Candace Liu @.***> wrote:

Unfortunately, we've only tested ark-analysis with scikit-image<0.19.3, so we can't guarantee that a newer version would work. However, ark-analysis should work with python 3.10, so you could try making a new environment with python 3.10 and try installing scikit-image 0.19.2 there.

— Reply to this email directly, view it on GitHub https://github.com/angelolab/ark-analysis/issues/1142#issuecomment-2218326167, or unsubscribe https://github.com/notifications/unsubscribe-auth/A374V5T5DAXLBINA2HM3TCTZLQP7HAVCNFSM6AAAAABJNGIQKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJYGMZDMMJWG4 . You are receiving this because you were mentioned.Message ID: @.***>

[cliu72]

Are you using jupyter lab or jupyter notebook? We recommend using jupyter lab. You can just type jupyter lab in your console - functionally it's similar to jupyter notebook. Let me know if you can see the heatmap in jupyter lab.

[mikemcka]

Hi, I was perviously running it in jupyter notebook but I have now run the notebook in jupyter lab and the heatmap is still empty. I have been opening notebooks using both jupyter notebook and jupyter lab through a GUI which creates slurm jobs so you can run the notebooks on compute nodes. Is the version of jupyter lab important?

[cliu72]

Hm I personally haven't tested different versions of jupyter lab, so not sure if that is the issue. The project toml for ark does specify ipython>=8.4,<9 and jupyterlab>=3.4.3,<4, so I would try to make sure those versions are used. If you think the compute nodes/slurm might be the issue, maybe you can try running it locally first, just to make sure you can get it working there. Most laptops should have enough RAM to run the example dataset.

[mikemcka]

Good news, its finally working, I installed it locally and created an enviroment in anaconda navigator following the same process I mentioned above, opened it in jupyter lab and ran all the code, I noticed there was two printouts of loading widget and I intially thought it wasnt working so I tabbed out and came back about 10 minutes later and it was there, it must have taken a while to load. Thankyou so much for all the help. time to get to work!

[cliu72]

Great! Good luck!