search

Deep-MI / fsqc

A set of quality control scripts for FreeSurfer- and FastSurfer-processed structural MRI data
https://deep-mi.org
MIT License
56 stars 14 forks source link

unbuntu terminal frozen while encounter issue with surface overlaid #8

Closed XinQi7788 closed 2 years ago

XinQi7788 commented 3 years ago

Hi Kersten:

I am using the latest updated qatools-python, but encountered the same “a problem occurred with the surface overlays” issue while running some ANDI T1w images on a ubuntu machine.

When I noticed the “A problem occurred with the surface overlays” message, I clicked Control + C, then clicked Enter, the program kept running the rest of list of images.

Here are examples of image lists make the Linux terminal froze while running qatools-python. ADNI_011_S_0022_I32389 ADNI_033_S_0739_I119135 ADNI_035_S_4082_I296308

Can you please share a link that I can upload FreeSurfer results of those images?

Looking forward to your reply

Xin

kdiers commented 3 years ago

Hi, I think that the previous upload link should still work; let me know if I should send it again. Thanks, Kersten

XinQi7788 commented 3 years ago

The zipped files are uploaded to the previous link already. There is a file from those 3 scans encountered an issue while copying to local pc, thus the zipped files do not include the file as below.

Thank you very much and looking forward to your reply.

Best Regards Xin

XinQi7788 commented 3 years ago

Hi Kersten:

Does qatools-python have a docker installation document?

Now for each scan image (subject), it takes about 10 min to finish the qatools-python processing on its FreeSurfer results assessment. If it has a docker installation document, that would be easy for its parallel computing on many images simultaneously.

Thank you very much

Best Regards Xin

kdiers commented 3 years ago

Hi Xin,

thanks for the upload! I ran these files, and have to admit that I could not see any problem with them. So the issue remains a mystery to some extent.

But maybe your Docker suggestion could also help with that: it's not only nice for parallelization, but also provides a more controlled environment with regard to other libraries etc., which may be a source of this error.

So it should be worth giving it a try. I still need to create a DockerFile / image; this could be done until the end of the week, I suppose.

Best, Kersten

XinQi7788 commented 3 years ago

Thanks a lot for helping on those issues.

Looking forward to the Docker File.

Best Xin

Xin Qi, Ph. D. Director, Data Science Neurology, Biostatistics Eisai Inc. Phone: 201-509-7046

From: Kersten Diers @.> Sent: Monday, October 18, 2021 11:32:01 AM To: Deep-MI/qatools-python @.> Cc: Xin Qi @.>; Author @.> Subject: [EXTERNAL] Re: [Deep-MI/qatools-python] unbuntu terminal frozen while encounter issue with surface overlaid (#8)

Hi Xin,

thanks for the upload! I ran these files, and have to admit that I could not see any problem with them. So the issue remains a mystery to some extent.

But maybe your Docker suggestion could also help with that: it's not only nice for parallelization, but also provides a more controlled environment with regard to other libraries etc., which may be a source of this error.

So it should be worth giving it a try. I still need to create a DockerFile / image; this could be done until the end of the week, I suppose.

Best, Kersten

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDeep-MI%2Fqatools-python%2Fissues%2F8%23issuecomment-945897707&data=04%7C01%7Cxin_qi%40eisai.com%7C7ecd74ae9ca34740528508d9924c706b%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637701679267989446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2F%2F0NebhKoAbIXUXmgtG84vlPTl8F4VIDpqAG2U8HJKQ%3D&reserved=0, or unsubscribehttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FASMEXYLRCWB5L6OHFF5ABSLUHQ4XDANCNFSM5F6OXLPQ&data=04%7C01%7Cxin_qi%40eisai.com%7C7ecd74ae9ca34740528508d9924c706b%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637701679267999410%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=c7jHZsYUTUGea3fRK80A60%2F%2FROiWfPWU5ovGOsEj9bY%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cxin_qi%40eisai.com%7C7ecd74ae9ca34740528508d9924c706b%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637701679267999410%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nhZl1PoR3tXbe6LrnpW8JBhVMfPE71xNJOzlYuPoU9o%3D&reserved=0 or Androidhttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cxin_qi%40eisai.com%7C7ecd74ae9ca34740528508d9924c706b%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637701679268009363%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=EiD0jsXucyvu%2FjwO3Nk9LKdJLktsJybURZ9LW%2FG1txg%3D&reserved=0.

[This e-mail message may contain privileged, confidential and/or proprietary information of Eisai. If you believe that it has been sent to you in error, please contact the sender immediately and delete the message including any attachments, without copying, using, or distributing any of the information contained therein. This e-mail message should not be interpreted to include a digital or electronic signature that can be used to authenticate an agreement, contract or other legal document, nor to reflect an intention to be bound to any legally-binding agreement or contract.]

XinQi7788 commented 3 years ago

Hi Kersten:

I will re-test those 3 files in this week individually and let you know how it goes.

Best Regards Xin

XinQi7788 commented 3 years ago

Hi Kersten:

I am testing those 3 subjects having issue to freeze the terminal showing " A problem occurred with the surface overlays".

The 3 subjects are ADNI_011_S_0022_I32389 ADNI_033_S_0739_I119135 ADNI_035_S_4082_I296308

I also upload the qatools-python results of those 3 subjects to the previous shared link.

Maybe you can try to solve the mystery from the information shared.

Once it freezes the terminal, then I press the Ctrl + C, then it shows "^CERROR: fornix module failed for subject ADNI_011_S_0022_I32389 Finished subject ADNI_011_S_0022_I32389 at 2021-10-21 00:26 UTC "

################################################################################################

For subject ADNI_011_S_0022_I32389 "A problem occurred with the surface overlays A problem occurred with the surface overlays A problem occurred with the surface overlays ...... A problem occurred with the surface overlays

^CERROR: fornix module failed for subject ADNI_011_S_0022_I32389 Finished subject ADNI_011_S_0022_I32389 at 2021-10-21 00:26 UTC

Running outlier detection module ...

Done"

################################################################################################ For subject ADNI_033_S_0739_I119135 " A problem occurred with the surface overlays A problem occurred with the surface overlays A problem occurred with the surface overlays ...... A problem occurred with the surface overlays ^CERROR: fornix module failed for subject ADNI_033_S_0739_I119135 Finished subject ADNI_033_S_0739_I119135 at 2021-10-21 00:41 UTC

Running outlier detection module ...

Done"

################################################################################################# For subject ADNI_035_S_4082_I296308 A problem occurred with the surface overlays A problem occurred with the surface overlays A problem occurred with the surface overlays ...... A problem occurred with the surface overlays ^CERROR: fornix module failed for subject ADNI_035_S_4082_I296308 Finished subject ADNI_035_S_4082_I296308 at 2021-10-21 00:54 UTC

Running outlier detection module ...

Done"

Best Regards Xin

From: Xin Qi Sent: Monday, October 18, 2021 8:08 PM To: Deep-MI/qatools-python @.***> Subject: RE: [EXTERNAL] Re: [Deep-MI/qatools-python] unbuntu terminal frozen while encounter issue with surface overlaid (#8)

Hi Kersten:

I will re-test those 3 files in this week individually and let you know how it goes.

Best Regards Xin

From: Kersten Diers @.**@.>> Sent: Monday, October 18, 2021 11:32 AM To: Deep-MI/qatools-python @.**@.>> Cc: Xin Qi @.**@.>>; Author @.**@.>> Subject: [EXTERNAL] Re: [Deep-MI/qatools-python] unbuntu terminal frozen while encounter issue with surface overlaid (#8)

Hi Xin,

thanks for the upload! I ran these files, and have to admit that I could not see any problem with them. So the issue remains a mystery to some extent.

But maybe your Docker suggestion could also help with that: it's not only nice for parallelization, but also provides a more controlled environment with regard to other libraries etc., which may be a source of this error.

So it should be worth giving it a try. I still need to create a DockerFile / image; this could be done until the end of the week, I suppose.

Best, Kersten

- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDeep-MI%2Fqatools-python%2Fissues%2F8%23issuecomment-945897707&data=04%7C01%7Cxin_qi%40eisai.com%7C7ecd74ae9ca34740528508d9924c706b%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637701679267989446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2F%2F0NebhKoAbIXUXmgtG84vlPTl8F4VIDpqAG2U8HJKQ%3D&reserved=0, or unsubscribehttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FASMEXYLRCWB5L6OHFF5ABSLUHQ4XDANCNFSM5F6OXLPQ&data=04%7C01%7Cxin_qi%40eisai.com%7C7ecd74ae9ca34740528508d9924c706b%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637701679267999410%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=c7jHZsYUTUGea3fRK80A60%2F%2FROiWfPWU5ovGOsEj9bY%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cxin_qi%40eisai.com%7C7ecd74ae9ca34740528508d9924c706b%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637701679267999410%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nhZl1PoR3tXbe6LrnpW8JBhVMfPE71xNJOzlYuPoU9o%3D&reserved=0 or Androidhttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cxin_qi%40eisai.com%7C7ecd74ae9ca34740528508d9924c706b%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637701679268009363%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=EiD0jsXucyvu%2FjwO3Nk9LKdJLktsJybURZ9LW%2FG1txg%3D&reserved=0.

[This e-mail message may contain privileged, confidential and/or proprietary information of Eisai. If you believe that it has been sent to you in error, please contact the sender immediately and delete the message including any attachments, without copying, using, or distributing any of the information contained therein. This e-mail message should not be interpreted to include a digital or electronic signature that can be used to authenticate an agreement, contract or other legal document, nor to reflect an intention to be bound to any legally-binding agreement or contract.]

kdiers commented 3 years ago

Hi Xin,

thanks for the info and for uploading the output - that was indeed helpful!

I have to admit that I was looking in the wrong place at the first, i.e. the screenshots and surface overlays within the 'screenshots' module. However, this module is also called from the 'fornix' module for visualization, and this is where the error occurs. I.e. I now could replicate the problem (we're entering an infinite loop); fixing it, as far as I can tell at the moment, will be straightforward.

Will upload a fix until Monday.

Best,

Kersten

XinQi7788 commented 3 years ago

That’s great and mystery is resolved 😊. Thanks, Kersten!

Good weekend and I will pull the new fixed version in next week.

Best Xin

kdiers commented 2 years ago

Hi, the infinite-loop issue is hopefully fixed now; I am leaving the issue open, though, for the docker file, which is in the works. Will update soon.

Best, Kersten

XinQi7788 commented 2 years ago

Thanks for the update!

Looking forward to the Docker file.

Best Regards Xin

Xin Qi, Ph. D. Director, Data Science Neurology, Biostatistics Eisai Inc. Phone: 201-509-7046

From: Kersten Diers @.> Sent: Monday, October 25, 2021 10:12:30 AM To: Deep-MI/qatools-python @.> Cc: Xin Qi @.>; Author @.> Subject: [EXTERNAL] Re: [Deep-MI/qatools-python] unbuntu terminal frozen while encounter issue with surface overlaid (#8)

Hi, the infinite-loop issue is hopefully fixed now; I am leaving the issue open, though, for the docker file, which is in the works. Will update soon.

Best, Kersten

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDeep-MI%2Fqatools-python%2Fissues%2F8%23issuecomment-950971300&data=04%7C01%7Cxin_qi%40eisai.com%7Cc6b062b3cb884d2ab67a08d997c17d60%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637707679553001942%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=y%2FFY0%2B4wcAPgYjRi2X7L71yopjXOfns%2F2kPo8s3V4QI%3D&reserved=0, or unsubscribehttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FASMEXYPUQCAQYUMNAQQL4MLUIVQU5ANCNFSM5F6OXLPQ&data=04%7C01%7Cxin_qi%40eisai.com%7Cc6b062b3cb884d2ab67a08d997c17d60%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637707679553011899%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3NZZmgF1qwpmGzk9qKb81l1979g1WUC3I9Xhn7hraMk%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cxin_qi%40eisai.com%7Cc6b062b3cb884d2ab67a08d997c17d60%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637707679553021857%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9DBFJD%2BAAhq%2B6%2B7%2BnWC%2BhOYaW%2Fcrnvo5xnYrGEB9kF8%3D&reserved=0 or Androidhttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cxin_qi%40eisai.com%7Cc6b062b3cb884d2ab67a08d997c17d60%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637707679553021857%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=20qAG0FM%2FYmqBPdkmAhUPFqtXwm5ID9QCT%2FTNUXXA4g%3D&reserved=0.

[This e-mail message may contain privileged, confidential and/or proprietary information of Eisai. If you believe that it has been sent to you in error, please contact the sender immediately and delete the message including any attachments, without copying, using, or distributing any of the information contained therein. This e-mail message should not be interpreted to include a digital or electronic signature that can be used to authenticate an agreement, contract or other legal document, nor to reflect an intention to be bound to any legally-binding agreement or contract.]

kdiers commented 2 years ago

Hi, I've added a Dockerfile including a new page for documenation.

Best, Kersten

XinQi7788 commented 2 years ago

Fantastic! Will test it later! Thanks Kersten!

Best Xin

Xin Qi, Ph. D. Director, Data Science Neurology, Biostatistics Eisai Inc. Phone: 201-509-7046

From: Kersten Diers @.> Sent: Thursday, October 28, 2021 8:19:19 AM To: Deep-MI/qatools-python @.> Cc: Xin Qi @.>; Author @.> Subject: [EXTERNAL] Re: [Deep-MI/qatools-python] unbuntu terminal frozen while encounter issue with surface overlaid (#8)

Hi, I've added a Dockerfile including a new page https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDeep-MI%2Fqatools-python%2Fblob%2Facb4f80444a25d33aaca8f2b8edbc886955a3cb6%2Fdocker%2FDocker.md&data=04%7C01%7Cxin_qi%40eisai.com%7Cb813419144a942a0e38b08d99a0d2cde%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637710203662129118%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BdCLRbiDKXh94Ha2mny4PxhhlqmLyIKiAc8ZMgfVAIQ%3D&reserved=0 for documenation.

Best, Kersten

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDeep-MI%2Fqatools-python%2Fissues%2F8%23issuecomment-953790642&data=04%7C01%7Cxin_qi%40eisai.com%7Cb813419144a942a0e38b08d99a0d2cde%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637710203662129118%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=gnu7zmFoFrcfyfJP2yHqHUF2MIlIDHaXhoNQ1LDHfb0%3D&reserved=0, or unsubscribehttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FASMEXYMYR3E63VFISO4F3J3UJFEVPANCNFSM5F6OXLPQ&data=04%7C01%7Cxin_qi%40eisai.com%7Cb813419144a942a0e38b08d99a0d2cde%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637710203662139074%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=p04Obv6PINGLfm%2F8tjokqumiaDC1fDOXu56d0bYR3g0%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cxin_qi%40eisai.com%7Cb813419144a942a0e38b08d99a0d2cde%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637710203662139074%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=RFXe9EweCA9W25YPDgfQIZv7EkCh%2F9q9oXmQtlcUMIY%3D&reserved=0 or Androidhttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cxin_qi%40eisai.com%7Cb813419144a942a0e38b08d99a0d2cde%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637710203662149031%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ylbYH7hh40ktW2M3xAz%2B4SXxLPyWb5HDi%2FvvuaEr75A%3D&reserved=0.

[This e-mail message may contain privileged, confidential and/or proprietary information of Eisai. If you believe that it has been sent to you in error, please contact the sender immediately and delete the message including any attachments, without copying, using, or distributing any of the information contained therein. This e-mail message should not be interpreted to include a digital or electronic signature that can be used to authenticate an agreement, contract or other legal document, nor to reflect an intention to be bound to any legally-binding agreement or contract.]

XinQi7788 commented 2 years ago

Hi Kersten:

Currently the qatools-python offers the screenshots and screenshots-html modules.

Besides current screenshots, if I would like to have extra screenshots at different locations with different views, such like below scenarios.

1). if I would like to have more coronal sections, like a total of 3 coronal to properly see the lateral ventricles and hippocampus from head to tail; 2). If I would like to have 2 sagittal more lateral like +30mm and -30mm from what you have currently (currently look very close to the midline) - would be nice to cut through the hippocampus; 3). If 1 more axial like +30mm along with current ones.

I think we should revise the createScreenshots.py in the qatools-python. Can you please tell me how to change the code to achieve creating screenshots such as multiple views at different location cuts?

Thank you very much.

Best Regards Xin

kdiers commented 2 years ago

Hi Xin,

there are some "expert options" within the script, which are a little bit hidden at the moment, and the "--screenshots_views" option may provide what you are looking for. Coordinates should be in FreeSurfer's RAS system.

Therefore, before making changes to the code, my suggestion is to give it a try and see if and how it works. If this is not what you are looking for, let me know.

Take a look at the help section in the qatoolspython/qatoolspython.py script, around lines 163 ff. currently, for some details.

Best regards,

Kersten

XinQi7788 commented 2 years ago

Hi Kersten:

Thank you so much for the quick response and information. We will look into it with your suggestions.

Best Xin

From: Kersten Diers @.***> Sent: Wednesday, November 10, 2021 12:12 PM To: Deep-MI/qatools-python Cc: Xin Qi; State change Subject: [EXTERNAL] Re: [Deep-MI/qatools-python] unbuntu terminal frozen while encounter issue with surface overlaid (#8)

Hi Xin,

there are some "expert options" within the script, which are a little bit hidden at the moment, and the "--screenshots_views" option may provide what you are looking for. Coordinates should be in FreeSurfer's RAS system.

Therefore, before making changes to the code, my suggestion is to give it a try and see if and how it works. If this is not what you are looking for, let me know.

Take a look at the help section in the qatoolspython/qatoolspython.py script, around lines 163 ff. currently, for some details.

Best regards,

Kersten

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHubhttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDeep-MI%2Fqatools-python%2Fissues%2F8%23issuecomment-965557676&data=04%7C01%7Cxin_qi%40eisai.com%7C09ea752dfcfd451a04d108d9a46d45f7%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637721611511628219%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=LiAeYvZzJwm12i%2BlP2O9ZmrsTW0WTy3D166GPd94PYI%3D&reserved=0, or unsubscribehttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FASMEXYLSH5M464E2ZRFI4ITULKRXRANCNFSM5F6OXLPQ&data=04%7C01%7Cxin_qi%40eisai.com%7C09ea752dfcfd451a04d108d9a46d45f7%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637721611511638176%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=q7lKTH1BHs8IEWZlJFJ5ASSbL7hT1F7gLpdsaAbeWBE%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cxin_qi%40eisai.com%7C09ea752dfcfd451a04d108d9a46d45f7%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637721611511638176%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yOMvwSDMq9VpcV2YxVKsmqQheM6jwKqB5zJLZK9lbRg%3D&reserved=0 or Androidhttps://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cxin_qi%40eisai.com%7C09ea752dfcfd451a04d108d9a46d45f7%7C013327ea41e944308dfacbb90ba7b1fa%7C0%7C0%7C637721611511648133%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xR4nmRkyhZXxbvYIwA85%2F6tjGxd9IsEwXgY3J96wIJs%3D&reserved=0.

[This e-mail message may contain privileged, confidential and/or proprietary information of Eisai. If you believe that it has been sent to you in error, please contact the sender immediately and delete the message including any attachments, without copying, using, or distributing any of the information contained therein. This e-mail message should not be interpreted to include a digital or electronic signature that can be used to authenticate an agreement, contract or other legal document, nor to reflect an intention to be bound to any legally-binding agreement or contract.]

CindyQi7788 commented 2 years ago

Hi Kersten:

Happy New Year of 2022!

I tried the --screenshots_views with different x, y, z locations that works well. Thank you so much!

However, if I would like to change the --screenshots_base to some image, such as hypothalamic_subunits_seg.v1.mgz, how to make it for a list of subjects, not just a single one. Here is the ERROR if using –subjects-file flag along with –screenshot_base.

ERROR: can only use the --screenshots_base argument for a single subject, which must be specified using –subject

Looking forward to your reply.

Best Regards Xin

CindyQi7788 commented 2 years ago

Can we create screenshots with a html file of hypothalamic subunits for a list of subjects? Just similar like for fornix by adding flag as --fornix --fornix-html?

Thank you very much. Xin

kdiers commented 2 years ago

Happy New Year to you, too!

1.

I have to admit that the current behavior of --screenshots_base when processing multiple subjects is to some extent the expected behavior; this argument was intended to allow for greater customization of the screenshots module when used as a stand-alone tool, and initially I did not want to make the assumption of identical filenames and identical (relative) path names of custom base images for multiple subjects. Without identical file and path names, specification of these images on the command line would require a long list of inputs and this would be rather cumbersome. The same logic applies to the --screenshots_surf and --screenshots_overlay arguments.

The workaround at the moment is of course to use the tool in single-subject mode, but I see that this may not be ideal either. And I see that in some scenarios the assumption of identical file / path names across subjects may be justified, and in the end I can still check for presence of the given file.

2.

If I understand correctly, your idea is to create semi-transparent, colored masks of the hypothalamic subunits, possibly with a contour, and overlay them onto (a part of) the anatomical image, is it? I guess it would also make sense for the subfields of hippocampus and amygdala, for which dedicated toolboxes also exist. In principle this can be done, but as far as I can see at the moment, it is slightly more complicated than the CC / fornix module, because we need a heuristic to find good coordinates where to do the screenshots, for example.

Let me give it a try for both 1. and 2., I'll see if it's feasible and get back on the issue. WIll take ~ 1-2 weeks, I guess.

Best regards, Kersten

CindyQi7788 commented 2 years ago

Thanks, Kersten for such a quick reply!

Yes, all your understandings are correct!

For no. 2 request, to find a good coordinates, just like the current version, to give a --screenshots_views with x, y, z coordinates that could be customized by user along with corresponding --screenshots_layout, that would be useful.

It would be great that such screenshots with a html file functionalities for the subfields of hippocampus, amygdala, thalamus and hypothalamus exist in the toolbox.

Best Regards Xin

kdiers commented 2 years ago

Hi Xin,

I've uploaded a new version with some modifications for requests 1. and 2.

The --screenshots_base argument offers more flexibility now, i.e. can use any volume file within the //mri directory; must have the same name across subjects. If files have different names across subjects or are not stored within each subject's mri subdirectory, then the program must be used in single-subject mode. I've written an explanation in the qatoolspython.py help section, but would like to keep them as 'expert' options for a while before fully documenting them (e.g. in the README.md).

Also, I've added --hypothalamus / --hypothalamus-html options. They work just like the fornix module. Coordinates for display are not required, but chosen based on the centroids of the subregions. Similiar to the above, I would like to keep them as 'expert' options for a while before fully documenting them (e.g. in the README.md); they should not interfere with other processing. Maybe try it out, and if you have any suggestions, let me know. For example, maybe croppping of the images might make sense.

Similar modules for the hippocampus and thalamus will likely be added within approx. 2 weeks.

Best, Kersten

CindyQi7788 commented 2 years ago

Hi Kersten:

Thank you so much for improving the tools with our requests. Very appreciate! I will test the new functions soon and let you know.

Best, Xin

CindyQi7788 commented 2 years ago

Hi Kersten:

The outlier module is to detect extreme values among the subcortical ('aseg') segmentations. Is it possible to include another outlier modules to detect extreme values of hypothalmus, fornix, hippocampus and thalamus segmentations?

Thank you very much.

Best, Xin

CindyQi7788 commented 2 years ago
  1. The screenshots on hypothalamus.png are too small to visualize. I agree that maybe cropping of the images might make sense. The size of screenshot on fornix looks good.
  2. Only the first screenshot has the hypothalamus overlay. Can we have different cuts/orientation views only focused on hypothalamus region?
  3. New modules of large ROIs with different cuts / views on hypothalamus, thalamus, hippocampus, amygdala?

Thank you very much.

Best, Xin

kdiers commented 2 years ago

Hi Xin, thanks for your helpful feedback. I'll push some updates during the next days:

1) I agree that using a cut-out is a better idea than using the full image. Will implement this. 2) Just to confirm, because I didn't see this during testing: only the first image shows the hypothalamus labels, the others just show the background image, even if enlarged? Will also check this. 3) Yes, new modules for hippocampus/amygdala and thalamus are also planned. Will take around 1 week per module. 4) Including the ROI segmentation stats in the outlier module is also a good idea. I'll put that on the list.

Best, Kersten

CindyQi7788 commented 2 years ago

Dear Kersten:

Thank you so much for your feedback and looking forward to your updates.

Best Xin

On Thu, Feb 3, 2022 at 3:19 PM Kersten Diers @.***> wrote:

Hi Xin, thanks for your helpful feedback. I'll push some updates during the next days:

  1. I agree that using a cut-out is a better idea than using the full image. Will implement this.
  2. Just to confirm, because I didn't see this during testing: only the first image shows the hypothalamus labels, the others just show the background image, even if enlarged? Will also check this.
  3. Yes, new modules for hippocampus/amygdala and thalamus are also planned. Will take around 1 week per module.
  4. Including the ROI segmentation stats in the outlier module is also a good idea. I'll put that on the list.

Best, Kersten

— Reply to this email directly, view it on GitHub https://github.com/Deep-MI/qatools-python/issues/8#issuecomment-1029365249, or unsubscribe https://github.com/notifications/unsubscribe-auth/AENN7AXKC3QUR7XGMSGGHP3UZLPMHANCNFSM5F6OXLPQ . You are receiving this because you commented.Message ID: @.***>

kdiers commented 2 years ago

Hi Xin,

I've made some changes and additions to the toolbox as discussed before. Specifically, I've addressed issues 1, 2, and partially 3 (only the hippocampus module) from the above list. Looking forward to your feedback (for example, do the views / sections of the hypothalamus and hippocampus/amygdala make sense, or should we add additional ones, or change the cut-out, etc.).

Please note that I've created and made the additions to a new freesurfer-module-dev branch (as opposed to the default freesurfer-module-releases branch), since there may be more additions and revisions in the future.

Eventually I will merge new content from freesurfer-module-dev branch to freesurfer-module-releases, but until then, please take care to download / checkout the freesurfer-module-dev branch for testing purposes.

Best,

Kersten

CindyQi7788 commented 2 years ago

Hi Kersten:

Thank you for updates! I cloned the freesurfer-module-dev branch for testing now and will let you know how the screenshots looks like later.

After adding flags of --hippocampus --hippocampus-html --hippocampus-label, the error shows as below. qatools.py: error: unrecognized arguments: --hippocampus --hippocampus-html --hippocampus-label

Best Xin

kdiers commented 2 years ago

Hi,

just to make sure: does a git status command show that you are on the dev branch? If not, please use git checkout freesurfer-module-dev to switch to that branch.

Thanks, Kersten

CindyQi7788 commented 2 years ago

Sorry, my mistake! I am testing using the freesurfer-module-dev now... so far looks fine. I will share more feedback later.

Thank you very much. Xin

CindyQi7788 commented 2 years ago

Hi Kersten:

The screenshots of subfields of hypothalamus, hippocampus/amygdala look very nice. After my consulting with my colleague, if any additional cut-outs are suggested, I will let you know. Thank you so much for implementing it.

Best Xin

CindyQi7788 commented 2 years ago

Hi Kersten:

Below are some questions and Error message encountered while testing on the hypothalamus screenshot module.

  1. For the screenshots of hippocampus/amygdala, hypothalamus, et al, are those cuts at the same location? (Is this standard MNI space or subject space?).

  2. When only use --hypothalamus --hypothalamus-html, such as for ~1700 subjects's IDs are listed inside the subjects-file, but only 8 subjects' screenshots are created inside the html file although the hypothalamus.png of each subject is created inside the hypothalamus folder.

  3. After the last subject's processing is done, got TypeError is shown as below. I guess it is maybe because there are some empty space at the bottom of the list of subjects-file, such as subjectID.txt. Traceback (most recent call last): File "qatools.py", line 7, in _qatools() File "/home/Xin/qatools-python-freesurfer-module-dev/qatoolspython/qatools_cmd.py", line 17, in _qatools qatoolspython.run_qatools(subjects_dir, output_dir, subjects, subjects_file, shape, screenshots, screenshots_html, screenshots_base, screenshots_overlay, screenshots_surf, screenshots_views, screenshots_layout, fornix, fornix_html, hypothalamus, hypothalamus_html, hippocampus, hippocampus_html, hippocampus_label, outlier, outlier_table, fastsurfer) File "/home/Xin/qatools-python-freesurfer-module-dev/qatoolspython/qatoolspython.py", line 1457, in run_qatools _do_qatools(subjects_dir, output_dir, subjects, shape, screenshots, screenshots_html, screenshots_base, screenshots_overlay, screenshots_surf, screenshots_views, screenshots_layout, fornix, fornix_html, hypothalamus, hypothalamus_html, hippocampus, hippocampus_html, hippocampus_label, outlier, outlier_table, fastsurfer) File "/home/Xin/qatools-python-freesurfer-module-dev/qatoolspython/qatoolspython.py", line 1418, in _do_qatools print("

    <a href=\""+os.path.join('hypothalamus', subject, os.path.basename(imagesHypothalamusDict[subject]))+"\">" File "/home/Xin/miniconda3/lib/python3.8/posixpath.py", line 142, in basename p = os.fspath(p) TypeError: expected str, bytes or os.PathLike object, not list

Best Regards. Xin

CindyQi7788 commented 2 years ago

Hi Kersten:

To following the above item 2 and 3.

For 2, I tested again, got the same result, only 8 subjects screenshots are shown inside the hypothalamus html file For 3, the empty space at the bottom of the subject-file are moved, so the last line of the list is a subject ID, but still received the same error as above.

Looking forward to your reply.

Best Xin

kdiers commented 2 years ago

Hi Xin,

thanks for your message. Some external, large-scale testing is really helpful, because I do not necessarily see all issues with my tests.

Re 1: The screenshots for Amygdala/Hippocampus and Hypothalamus are roughly the same (=equivalent) location. I am using the centroids of (a selection of) subfields / subregions to define the views. In that sense, I believe they are roughly comparable across cases, eventhough everything still remains in native space.

Re 2&3: I believe those issues are related. I speculate that the hypothalamic module for your subject no. 9 failed for some unknown reason (why would be interesting to know), and I did not catch this properly for the creation of the html overview. I have uploaded a fix to the dev branch, and hope it's working now.

Thanks, Kersten

XinQi7788 commented 2 years ago

Hi Kersten:

Thank you so much for a quick reply and update!

I started to test the new uploaded dev branch and will let you know how it goes sometime later tomorrow.

BTW, for the subject no. 9, how did you know it is no. 9 and what is its ID if you can tell? The no. 9 subject on the list, its hypothalamus screenshot looks fine to me.

Best, Xin

kdiers commented 2 years ago

Great, thanks.

I am speculating it's subject no. 9 because apparently the preceding 8 cases went through properly. And I could reproduce the reported error by making one of my test cases failing the hypothalamus module, which later caused the html generation to terminate while processing this particular case. There might still be another reason, but at least that's my current hypothesis. Note that in the html output the subjects are sorted alphanumerically, so if your list isn't sorted or sorted differently, this might explain why your no. 9 (from the list) appears to be OK. The ID I can't tell, sorry.

Best, Kersten

XinQi7788 commented 2 years ago

Hi Kersten: The failed hypothalamus screenshot case is identified. Here its result is attached. Maybe you can figure out why :).

The new version works well. On the html file, the failed case only shows the subject ID without breaking the processing keeping going. Thank you very much!.

Best, Xin Hypothalamas_screenshots_Failed.zip

XinQi7788 commented 2 years ago

Hi Kersten:

So far, the qatools-python works well in processing the subjects sequentially. Each subject takes about 7-14 mins according to different computer configs ( I am using 4-core CPU now), but to process thousands of subjects, it takes many days or weeks to get it done. Especially the outliers, html files to make the final processing above the all subjects' processing are done. If we would like to to conduct parallel computing on each subject, then integrate all the subjects results to get outliers, html files et al, do you have suggestion on how to use the toolkit?

Looking forward to your reply and thank you very much!

Best, Xin

kdiers commented 2 years ago

Hi Xin,

thanks, and good point; I think there are two potential solutions: one would be to selectively switch off (parts of) components that are not strictly necessary, but take a lot of time. Generally, screenshots do take some time, and also the surface / contour overlays need some time to be computed. I would need to take a closer look in the next days how much time actually, and if there is potential for saving time without sacrificing too much information. The other potential solution of course is parallelization; here, the easy-to-implement solution would be to start multiple instances of the program at the same time, and in the end have another single run to integrate all existing results. In contrast, full parallelization within a single run of the program might take some more implementation effort. I will think about during the next days, and get back to you.

Best, Kersten

XinQi7788 commented 2 years ago

Hi Kersten:

Thank you so much for the update. The other potential solution sounds good to me.

Also, Can qatools-python provide cortical parcellation screening shots and numerical outliers of the cortical parcellations as well? This is more important than parallelization requests currently :).

Best Regards Xin

kdiers commented 2 years ago

Hi Xin,

I've adapted the outlier module to include cortical, hippocampal/amygdalar, and hypothalamic values, and have uploaded a new version to the dev branch for evaluation purposes. The parallelization enhancement remains on the TODO list, and so do the brainstem and thalamus extensions. Cortical screenshots will require some time, because there is little existing to code to build upon at the moment.

Best, Kersten

XinQi7788 commented 2 years ago

Hi Kersten:

Thank you for your so quick feedback. I have started to test the updated outlier module and will let you know how it goes.

Best, Xin

XinQi7788 commented 2 years ago

Kersten:

For the normative values taken from the literature (see References) when calculating normative values of outliers, can you please tell which references from the citations are? Thanks a lot.

kdiers commented 2 years ago

The normative values are derived from:

Potvin O, Mouiha A, Dieumegarde L, Duchesne S, & Alzheimer's Disease Neuroimaging Initiative; 2016; Normative data for subcortical regional volumes over the lifetime of the adult human brain; Neuroimage: 137, 9-20; doi.org/10.1016/j.neuroimage.2016.05.016

XinQi7788 commented 2 years ago

The normative values are derived from:

Potvin O, Mouiha A, Dieumegarde L, Duchesne S, & Alzheimer's Disease Neuroimaging Initiative; 2016; Normative data for subcortical regional volumes over the lifetime of the adult human brain; Neuroimage: 137, 9-20; doi.org/10.1016/j.neuroimage.2016.05.016

Thanks, Kersten! From the outlier results, it is easy to get more false-positive outliers compared to a visual read from our experience so far.

Best, Xin

XinQi7788 commented 2 years ago

Hi Kersten:

For different surfaces views, there are no --surfaces_views and --surfaces_layout flags.

There are only 4 views of surfaces in the html file. Can we have different views of surfaces like the screenshots_view as below ? Thank you very much.

This is only for different screenshots views as below. --screenshots_views x=-30 x=-20 x=20 x=30 y=-30 y=-20 y=-10 y=0 z=10 z=30 --screenshots_layout 2 5

Best regards Xin

kdiers commented 2 years ago

Hi Xin,

thanks for your message. I am not sure that a --surface_views argument is really meaningful, since we are looking onto a 2D surface, as opposed to slices through a volume. In particular, slicing through the surface might just give its boundary contours. So I am a bit hesitating to implement a solution with arbitrary coordinates. But I am happy to discuss.

One potential improvement, however, could be (optional) views from top and bottom, and front and back - i.e. to add more fixed viewing positions.

Best regards, Kersten

XinQi7788 commented 2 years ago

Hi Kersten:

For the surface views, can we have 6 views of left_lateral, left_medial, left_transverse, right_lateral, right_medial, and right_transverse? And I am not sure inflated view are useful for us.

Here is an example created by using another tool for your reference, but we would like to use qatools-python alone to get surface and volume screenshots with a html file.

Looking forward to your reply and thank you very much for your always help! surface_views_example.zip

Best Regards, Xin

XinQi7788 commented 2 years ago

Hi Kersten:

No need to spend time revising the surfaces screenshots. The current version is fine. Thank you so much!

Best, Xin