search

marcnol / pyHiM

Multiplexed DNA-FISH data analysis pipeline
GNU Lesser General Public License v3.0
4 stars 2 forks source link

processHiMmatrix.py error #32

Closed MarionBardou closed 2 years ago

MarionBardou commented 2 years ago

With Olivier, we have been trying to run processHiMmatrix.py, but we have the same error before the end of the task. In the file attached, you can find a copy of my terminal. processHiMmatrix_error_pyHiM.txt

To give a bit more context, it is not working with data segmented with stardist whether it is in 2D or in 3D. It seems to be related to astropy

MarionBardou commented 2 years ago

I tried to run it on Lifou with an older version of pyHiM and it does not work either. I guess it must come from the "buildsPWDmatrix_3D_order:0_ROI:2.ecsv" file.

marcnol commented 2 years ago

I do not know where the error is coming from. The buildsPWDmatrix_3D_order file does have very large fields but I am not sure this is the issue.

In fact if I go to your folder: /mnt/grey/DATA/AnalyzedData_2022/Experiment_4_Sandrine_Marion_pupa_HiRes_42nt/002_ROI_RNA/buildsPWDmatrix the error I get is of a missing buildsPWDmatrix_HiMscMatrix.npy and buildsPWDmatrix_uniqueBarcodes.ecsvfiles.

So I renamed them as 

ln -s buildsPWDmatrix_3D_mask1_HiMscMatrix.npy buildsPWDmatrix_HiMscMatrix.npy
ln -s buildsPWDmatrix_3D_mask1_uniqueBarcodes.ecsv buildsPWDmatrix_uniqueBarcodes.ecsv

then run processHiMmatrix in the /mnt/grey/DATA/AnalyzedData_2022/Experiment_4_Sandrine_Marion_pupa_HiRes_42nt/tmp folder and everything worked with

processHiMmatrix

but not with

processHiMmatrix --d3

So, I am not sure what is going on yet!

Can u check what you get when you run without the --d3 option??

marcnol commented 2 years ago

I wrote a new script trace_combinator that can combine trace tables from different ROIs, or experiments. This combined trace table can then be used as input to build_matrixwithin pyHiM to construct the single cell matrix files and ensemble maps.

See details in the 2.2 Combining traces section of Running_pyHiM.

olivier469110 commented 2 years ago

Adding the following line in processHiMmatrix.py solve the problem

csv.field_size_limit(sys.maxsize)

marcnol commented 2 years ago

should we apply this patch to the upstream 'development' branch? or this error only appears for the new 'refactoring_buildPWDmatrix' branch??

marcnol commented 2 years ago

Olivier, can you do a pull request to the dev branch with this change?

olivier469110 commented 2 years ago

It's now modified in the refactoring_PWDmatrix branch

Olivier Messina, PhD Student CNRS, INSERM, Centre de Biochimie Structuralehttps://www.nollmannlab.org/,https://www.nollmannlab.org/ Mechanisms of DNA Segregation and Remodeling https://www.nollmannlab.org/

CNRS UMR 5048 - UM - INSERM 1054 60 rue de Navacelles, 34090 Montpellier (France) e-mail : @. / @. Twitter : https://twitter.com/MessinaOlivier1

De : NollmannLab @.> Envoyé : lundi 28 février 2022 12:28 À : marcnol/pyHiM @.> Cc : Olivier @.>; Comment @.> Objet : Re: [marcnol/pyHiM] processHiMmatrix.py error (Issue #32)

Olivier, can you do a pull request to the dev branch with this change?

— Reply to this email directly, view it on GitHubhttps://github.com/marcnol/pyHiM/issues/32#issuecomment-1054160920, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANOUGJGNCBFEYHGQ44HAJHDU5NL5FANCNFSM5OVKH4ZA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.Message ID: @.***>

marcnol commented 2 years ago

great. Should we close for now and see what/if other errors occur?

olivier469110 commented 2 years ago

Yes I'm doing some tests with low numbers of images in a folder to see if I can run the pipeline from begining to the end. If it's working I will launch it in a folder containing a full experiment to check if I can reconstruct the matrix.

Olivier Messina, PhD Student CNRS, INSERM, Centre de Biochimie Structuralehttps://www.nollmannlab.org/,https://www.nollmannlab.org/ Mechanisms of DNA Segregation and Remodeling https://www.nollmannlab.org/

CNRS UMR 5048 - UM - INSERM 1054 60 rue de Navacelles, 34090 Montpellier (France) e-mail : @. / @. Twitter : https://twitter.com/MessinaOlivier1

De : NollmannLab @.> Envoyé : lundi 28 février 2022 14:55 À : marcnol/pyHiM @.> Cc : Olivier @.>; Comment @.> Objet : Re: [marcnol/pyHiM] processHiMmatrix.py error (Issue #32)

great. Should we close for now and see what/if other errors occur?

— Reply to this email directly, view it on GitHubhttps://github.com/marcnol/pyHiM/issues/32#issuecomment-1054280832, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANOUGJGIEV4DLHTKHXRWDA3U5N5GDANCNFSM5OVKH4ZA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.Message ID: @.***>