Error when run chess_extract

yanchunzhang commented 3 years ago

Hi, I always got this error recently when run the chess_extarct command: do you have any idea about this error?

Traceback (most recent call last): File "/home/ch220811/.local/bin/chess", line 585, in Chess() File "/home/ch220811/.local/bin/chess", line 75, in init getattr(self, args.command)([sys.argv[0]] + sys.argv[option_ix:]) File "/home/ch220811/.local/bin/chess", line 567, in extract args.closing_square) File "/home/ch220811/.local/lib/python3.6/site-packages/chess/get_structures.py", line 148, in extract_structures filter1 = filters.threshold_otsu(filter_positive, nbins=size) File "/home/ch220811/.local/lib/python3.6/site-packages/skimage/filters/thresholding.py", line 285, in threshold_otsu hist, bin_centers = histogram(image.ravel(), nbins, source_range='image') File "/home/ch220811/.local/lib/python3.6/site-packages/skimage/exposure/exposure.py", line 139, in histogram hist, bin_edges = np.histogram(image, bins=nbins, range=hist_range) File "<__array_function__ internals>", line 6, in histogram File "/home/ch220811/software/lib/python3.6/site-packages/numpy/lib/histograms.py", line 856, in histogram decrement = tmp_a < bin_edges[indices] IndexError: index -9223372036854775808 is out of bounds for axis 0 with size 102

Somtimes I also got this error: /slurm/reports/var/spool/slurm/d/job466467/slurm_script: line 15: 45401 Killed /home/ch220811/.local/bin/chess extract ${TSV} ${MATRIX1} ${MATRIX2} ${OUT} Is this because my task got killed by my IT manager or the system due to memory size limit?

Thank you!

sgalan commented 3 years ago

Dear Yanchun Zhang,

It seems that this error comes from a bug from numpy. The threshold for the histogram is too high for numpy to work properly. Maybe you may consider some fixes that are suggested in here: https://github.com/numpy/numpy/issues/8627.

I will also think about alternatives,

Sorry for the inconvenience,

S

El lun, 14 dic 2020 a las 18:24, Yanchun Zhang (notifications@github.com) escribió:

Hi, I always got this error recently when run the chess_extarct command: do you have any idea about this error?

Traceback (most recent call last): File "/home/ch220811/.local/bin/chess", line 585, in Chess() File "/home/ch220811/.local/bin/chess", line 75, in init getattr(self, args.command)([sys.argv[0]] + sys.argv[option_ix:]) File "/home/ch220811/.local/bin/chess", line 567, in extract args.closing_square) File "/home/ch220811/.local/lib/python3.6/site-packages/chess/get_structures.py", line 148, in extract_structures filter1 = filters.threshold_otsu(filter_positive, nbins=size) File "/home/ch220811/.local/lib/python3.6/site-packages/skimage/filters/thresholding.py", line 285, in threshold_otsu hist, bin_centers = histogram(image.ravel(), nbins, source_range='image') File "/home/ch220811/.local/lib/python3.6/site-packages/skimage/exposure/exposure.py", line 139, in histogram hist, bin_edges = np.histogram(image, bins=nbins, range=hist_range) File "<array_function internals>", line 6, in histogram File "/home/ch220811/software/lib/python3.6/site-packages/numpy/lib/histograms.py", line 856, in histogram decrement = tmp_a < bin_edges[indices] IndexError: index -9223372036854775808 is out of bounds for axis 0 with size 102

Thank you!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vaquerizaslab/chess/issues/25, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADZPOZHIXHFFQGJB7ENEJD3SUZC6NANCNFSM4U3CGDRQ .

--

Silvia Galan Martínez - PhD student

Centre Nacional d'Anàlisi Genòmica-Centre de Regulació Genòmica (CNAG-CRG)

Structural Genomics Dpt.

Parc Científic de Barcelona – Torre I

Baldiri Reixac, 4

08028 Barcelona

Tel +34 9340 20580

Email silvia.galan@cnag.crg.eu

yanchunzhang commented 3 years ago

Thank you for your reply, @sgalan ! I see the numpy_histogram bug you mentioned but I'm still not sure how to avoid this happening. Why do you think there could be a number larger than 2**53 generated in the process of chess_extract? What the numbers used for histogam reresent? And do i need to modify the code of chess_extract or numpy_histogram by myself?

sgalan commented 3 years ago

Hi Yanchun Zhang,

Which matrices are you using as an input for this command? The first thing that you can try to do is to change the lines 148 and 154 in which the histogram is computed in order to binarize the image. It uses all the values from the image in order to compute a threshold to binarize the image. Now it's using the whole image to do so, you can try to put a smaller value, like the half of the size of your image in the parameter nbins.

Hope it helps,

S

El lun, 14 dic 2020 a las 20:59, Yanchun Zhang (notifications@github.com) escribió:

Thank you for your reply, @sgalan https://github.com/sgalan ! I see the numpy_histogram bug you mentioned but I'm still not sure how to avoid this happening. Why do you think there could be a number larger than 2**53 generated in the process of chess_extract? What the numbers used for histogam reresent? And do i need to modify the code of chess_extract or numpy_histogram by myself?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vaquerizaslab/chess/issues/25#issuecomment-744675442, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADZPOZCYTNYTRQGHNZMONVDSUZVCBANCNFSM4U3CGDRQ .

--