[Error]ParticipantAFQ

[sleeeeeeepy]

Hi. I am currently working with dwi data that has been preprocessed in MRtrix3.0 and would like to use the pyafq without BIDS required in order to obtain all tract information. In the future, I would like to the extract the white matter fibers associated with pain and eventually quantitatively analyze which fibers in the groups with and without pain show significant differences in DTI-FA values. So, Firstly, I tried ParticipantAFQ with jupyter notebook. However, Some errors existed and I could not obtain the result. Is there a solution to this problem? My code and the error I receive are below. Best regards,

In[1]: import os os.getcwd() Out[1]: '/media/sf_share/pyAFQ_test/my_dataset/sub-0001/dwi' In [2] : ls Out [2]: sub-0001_dwi.bvec tract/ sub-0001_dwi.bval sub-0001_dwi.nii.gz* In [3]: from AFQ.api.participant import ParticipantAFQ output_directory = "./tract" myafq = ParticipantAFQ( "sub-0001_dwi.nii.gz", "sub-0001_dwi.bval", "sub-0001_dwi.bvec", output_dir = output_directory) In[4]: myafq.export_all()

AttributeError Traceback (most recent call last) /tmp/ipykernel_6180/3296916702.py in ----> 1 myafq.export_all()

~/.local/lib/python3.8/site-packages/AFQ/api/participant.py in getattribute(self, attr) 140 141 # attr not found, allow typical AttributeError --> 142 return object.getattribute(self, attr)

AttributeError: 'ParticipantAFQ' object has no attribute 'export_all'

[arokem]

Hello! Sounds like an interesting project.

That looks like a bug, but I believe that it's already fixed on master. How did you install pyAFQ? Are you able to upgrade the software by running the following: pip install --upgrade git+https://github.com/yeatmanlab/pyAFQ.git ?

[sleeeeeeepy]

Thank you very much for kind response. After I upgrade the software, pyAFQ starts runnning. However, next error has come; "SSLCertVerificationError. I was able to get the "MNI" but not the others due to this error. Is there a solution to this problem? My code and the error I receive are below. Best regards,

In [3]: from AFQ.api.participant import ParticipantAFQ output_directory = "./tract" myafq = ParticipantAFQ( "sub-0001_dwi.nii.gz", "sub-0001_dwi.bval", "sub-0001_dwi.bvec", output_dir = output_directory) In[4]: myafq.export_all()

INFO:AFQ:Saving ./tract/sub-0001_dwi_desc-b0_dwi.nii.gz INFO:AFQ:Saving ./tract/sub-0001_dwi_desc-brain_mask.nii.gz 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73647/73647.0 [00:28<00:00, 2617.15it/s] INFO:AFQ:Saving ./tract/sub-0001_dwi_model-CSD_desc-diffmodel_dwi.nii.gz INFO:AFQ:Saving ./tract/sub-0001_dwi_model-CSD_desc-APM_dwi.nii.gz

Optimizing level 2 [max iter: 10000] Optimizing level 1 [max iter: 1000] Optimizing level 0 [max iter: 100] Optimizing level 2 [max iter: 10000] Optimizing level 1 [max iter: 1000] Optimizing level 0 [max iter: 100] Optimizing level 2 [max iter: 10000] Optimizing level 1 [max iter: 1000] Optimizing level 0 [max iter: 100]

INFO:AFQ:Saving ./tract/sub-0001_dwi_desc-prealign_from-DWI_to-MNI_xform.npy INFO:AFQ:Saving ./tract/sub-0001_dwi_desc-mapping_from-DWI_to-MNI_xform.nii.gz INFO:AFQ:Saving ./tract/sub-0001_dwi_space-template_desc-b0_dwi.nii.gz INFO:AFQ:Saving ./tract/sub-0001_dwi_space-subject_desc-template_dwi.nii.gz INFO:AFQ:Saving ./tract/sub-0001_dwi_model-DTI_desc-diffmodel_dwi.nii.gz INFO:AFQ:Saving ./tract/sub-0001_dwi_model-DTI_desc-FA_dwi.nii.gz INFO:AFQ:Saving ./tract/sub-0001_dwi_desc-seed_mask.nii.gz INFO:AFQ:Saving ./tract/sub-0001_dwi_desc-stop_mask.nii.gz INFO:AFQ:Loading Image... INFO:AFQ:Generating Seeds... INFO:AFQ:Getting Directions... INFO:AFQ:Tracking... 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 34566/34566 [06:45<00:00, 85.34it/s] INFO:AFQ:Saving ./tract/sub-0001_dwi_space-RASMM_model-probCSD_tractography.trk INFO:AFQ:Preparing Segmentation Parameters INFO:AFQ:Preprocessing Streamlines INFO:AFQ:Assigning Streamlines to Bundles

---

SSLCertVerificationError Traceback (most recent call last) /usr/lib/python3.8/urllib/request.py in do_open(self, http_class, req, **http_conn_args) 1353 try: -> 1354 h.request(req.get_method(), req.selector, req.data, headers, 1355 encode_chunked=req.has_header('Transfer-encoding'))

/usr/lib/python3.8/http/client.py in request(self, method, url, body, headers, encode_chunked) 1255 """Send a complete request to the server.""" -> 1256 self._send_request(method, url, body, headers, encode_chunked) 1257

/usr/lib/python3.8/http/client.py in _send_request(self, method, url, body, headers, encode_chunked) 1301 body = _encode(body, 'body') -> 1302 self.endheaders(body, encode_chunked=encode_chunked) 1303

/usr/lib/python3.8/http/client.py in endheaders(self, message_body, encode_chunked) 1250 raise CannotSendHeader() -> 1251 self._send_output(message_body, encode_chunked=encode_chunked) 1252

/usr/lib/python3.8/http/client.py in _send_output(self, message_body, encode_chunked) 1010 del self._buffer[:] -> 1011 self.send(msg) 1012

/usr/lib/python3.8/http/client.py in send(self, data) 950 if self.auto_open: --> 951 self.connect() 952 else:

/usr/lib/python3.8/http/client.py in connect(self) 1424 -> 1425 self.sock = self._context.wrap_socket(self.sock, 1426 server_hostname=server_hostname)

/usr/lib/python3.8/ssl.py in wrap_socket(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, session) 499 # ctx._wrap_socket() --> 500 return self.sslsocket_class._create( 501 sock=sock,

/usr/lib/python3.8/ssl.py in _create(cls, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, context, session) 1039 raise ValueError("do_handshake_on_connect should not be specified for non-blocking sockets") -> 1040 self.do_handshake() 1041 except (OSError, ValueError):

/usr/lib/python3.8/ssl.py in do_handshake(self, block) 1308 self.settimeout(None) -> 1309 self._sslobj.do_handshake() 1310 finally:

SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)

During handling of the above exception, another exception occurred:

URLError Traceback (most recent call last) /tmp/ipykernel_2702/3296916702.py in ----> 1 myafq.export_all()

~/.local/lib/python3.8/site-packages/AFQ/api/participant.py in export_all(self, viz, xforms, indiv) 180 start_time = time() 181 seg_algo = self.export("segmentation_params").get("seg_algo", "AFQ") --> 182 export_all_helper(self, seg_algo, xforms, indiv, viz) 183 self.logger.info( 184 f"Time taken for export all: {time() - start_time}")

~/.local/lib/python3.8/site-packages/AFQ/api/utils.py in export_all_helper(api_afq_object, seg_algo, xforms, indiv, viz) 122 123 if indiv: --> 124 api_afq_object.export("indiv_bundles") 125 if seg_algo == "AFQ": 126 api_afq_object.export("rois")

~/.local/lib/python3.8/site-packages/AFQ/api/participant.py in export(self, attr_name) 152 if section is None: 153 return self.wf_dict[attr_name] --> 154 return self.wf_dict[section][attr_name] 155 156 def export_all(self, viz=True, xforms=True,

~/.local/lib/python3.8/site-packages/pimms/calculation.py in getitem(self, k) 464 effval = self.efferents.get(k, self) 465 if effval is self: --> 466 self._run_node(self.plan.efferents[k]) 467 return self.efferents[k] 468 else:

~/.local/lib/python3.8/site-packages/pimms/calculation.py in _run_node(self, node) 528 #except Exception: #(h, found) = (None, False) 529 # ensure we have a result --> 530 if not found: res = node(self) 531 # process the result: 532 effs = reduce(lambda m,v: m.set(v[0],v[1]), six.iteritems(res), self.efferents)

~/.local/lib/python3.8/site-packages/pimms/calculation.py in call(self, *args, *kwargs) 84 if name not in opts: 85 raise ValueError('required parameter %s not given to %s' % (name, self.name)) ---> 86 args.append(opts[name]) 87 result = self.function(args) 88 if is_map(result):

~/.local/lib/python3.8/site-packages/pimms/util.py in getitem(self, k) 787 return val 788 def getitem(self, k): --> 789 return self._examineval(k, ps.PMap.getitem(self, k)) 790 def iterkeys(self): 791 for (k,) in ps.PMap.iteritems(self):

~/.local/lib/python3.8/site-packages/pimms/util.py in _examine_val(self, k, val) 783 return val 784 else: --> 785 val = val() 786 object.setattr(self, '_memoized', self._memoized.set(vid, val)) 787 return val

~/.local/lib/python3.8/site-packages/pimms/util.py in () 1050 coll = collect(maps) 1051 if choose_fn is None: choose_fn = _choose_last -> 1052 def curry_choice(k, args): return lambda:choose_fn(k, args) 1053 resmap = lazy_map({k:curry_choice(k, v) for (k,v) in six.iteritems(coll)}) 1054 # if they're all itables of the same size, return an itable

~/.local/lib/python3.8/site-packages/pimms/util.py in _choose_last(k, vs) 1019 def _choose_last(k, vs): 1020 '_choose_last(k, vs) yields vs[-1][k].' -> 1021 return vs[-1][k] 1022 def merge(*args, **kwargs): 1023 '''

~/.local/lib/python3.8/site-packages/pimms/calculation.py in getitem(self, k) 464 effval = self.efferents.get(k, self) 465 if effval is self: --> 466 self._run_node(self.plan.efferents[k]) 467 return self.efferents[k] 468 else:

~/.local/lib/python3.8/site-packages/pimms/calculation.py in _run_node(self, node) 528 #except Exception: #(h, found) = (None, False) 529 # ensure we have a result --> 530 if not found: res = node(self) 531 # process the result: 532 effs = reduce(lambda m,v: m.set(v[0],v[1]), six.iteritems(res), self.efferents)

~/.local/lib/python3.8/site-packages/pimms/calculation.py in call(self, *args, *kwargs) 84 if name not in opts: 85 raise ValueError('required parameter %s not given to %s' % (name, self.name)) ---> 86 args.append(opts[name]) 87 result = self.function(args) 88 if is_map(result):

~/.local/lib/python3.8/site-packages/pimms/util.py in getitem(self, k) 787 return val 788 def getitem(self, k): --> 789 return self._examineval(k, ps.PMap.getitem(self, k)) 790 def iterkeys(self): 791 for (k,) in ps.PMap.iteritems(self):

~/.local/lib/python3.8/site-packages/pimms/util.py in _examine_val(self, k, val) 783 return val 784 else: --> 785 val = val() 786 object.setattr(self, '_memoized', self._memoized.set(vid, val)) 787 return val

~/.local/lib/python3.8/site-packages/pimms/util.py in () 1050 coll = collect(maps) 1051 if choose_fn is None: choose_fn = _choose_last -> 1052 def curry_choice(k, args): return lambda:choose_fn(k, args) 1053 resmap = lazy_map({k:curry_choice(k, v) for (k,v) in six.iteritems(coll)}) 1054 # if they're all itables of the same size, return an itable

~/.local/lib/python3.8/site-packages/pimms/util.py in _choose_last(k, vs) 1019 def _choose_last(k, vs): 1020 '_choose_last(k, vs) yields vs[-1][k].' -> 1021 return vs[-1][k] 1022 def merge(*args, **kwargs): 1023 '''

~/.local/lib/python3.8/site-packages/pimms/calculation.py in getitem(self, k) 464 effval = self.efferents.get(k, self) 465 if effval is self: --> 466 self._run_node(self.plan.efferents[k]) 467 return self.efferents[k] 468 else:

~/.local/lib/python3.8/site-packages/pimms/calculation.py in _run_node(self, node) 528 #except Exception: #(h, found) = (None, False) 529 # ensure we have a result --> 530 if not found: res = node(self) 531 # process the result: 532 effs = reduce(lambda m,v: m.set(v[0],v[1]), six.iteritems(res), self.efferents)

~/.local/lib/python3.8/site-packages/pimms/calculation.py in call(self, *args, *kwargs) 85 raise ValueError('required parameter %s not given to %s' % (name, self.name)) 86 args.append(opts[name]) ---> 87 result = self.function(args) 88 if is_map(result): 89 if len(result) != len(self.efferents) or not all(e in result for e in self.efferents):

in wrapper_has_args_func(dwi, data_imap, mapping_imap, tractography_imap, segmentation_params, base_fname, tracking_params) ~/.local/lib/python3.8/site-packages/AFQ/tasks/decorators.py in wrapper_as_file(*args, **kwargs) 131 segmentation_params=segmentation_params) 132 if not op.exists(this_file): --> 133 img_trk_np_or_csv, meta = func(*args[:og_arg_count], **kwargs) 134 135 logger.info(f"Saving {this_file}") ~/.local/lib/python3.8/site-packages/AFQ/tasks/segmentation.py in segment(dwi, data_imap, mapping_imap, tractography_imap, segmentation_params) 58 start_time = time() 59 segmentation = seg.Segmentation(**segmentation_params) ---> 60 bundles = segmentation.segment( 61 bundle_dict, 62 tg, ~/.local/lib/python3.8/site-packages/AFQ/segmentation.py in segment(self, bundle_dict, tg, fdata, fbval, fbvec, mapping, reg_prealign, reg_template, img_affine, reset_tg_space) 343 # We only care about midline crossing if we use AFQ: 344 self.cross_streamlines() --> 345 fiber_groups = self.segment_afq() 346 elif self.seg_algo.startswith("reco"): 347 fiber_groups = self.segment_reco() ~/.local/lib/python3.8/site-packages/AFQ/segmentation.py in segment_afq(self, tg) 605 606 self.logger.info("Assigning Streamlines to Bundles") --> 607 for bundle_idx, bundle in enumerate(self.bundle_dict): 608 self.logger.info(f"Finding Streamlines for {bundle}") 609 warped_prob_map, include_roi, exclude_roi,\ ~/.local/lib/python3.8/site-packages/AFQ/api/bundle_dict.py in __iter__(self) 370 371 def __iter__(self): --> 372 self.gen_all() 373 return iter(self._dict) 374 ~/.local/lib/python3.8/site-packages/AFQ/api/bundle_dict.py in gen_all(self) 305 if bundle_name not in self._dict: 306 if not self.templates_loaded: --> 307 self.load_templates() 308 self._gen(bundle_name) 309 if self.templates_loaded: ~/.local/lib/python3.8/site-packages/AFQ/api/bundle_dict.py in load_templates(self) 199 if self.seg_algo == "afq": 200 self.templates =\ --> 201 afd.read_templates(as_img=False) 202 # For the arcuate, we need to rename a few of these 203 # and duplicate the SLF ROI: ~/.local/lib/python3.8/site-packages/AFQ/data/fetch.py in read_templates(as_img, resample_to) 565 tic = time.perf_counter() 566 --> 567 template_dict = _fetcher_to_template( 568 fetch_templates, 569 as_img=as_img, ~/.local/lib/python3.8/site-packages/AFQ/data/fetch.py in _fetcher_to_template(fetcher, as_img, resample_to) 83 if isinstance(resample_to, str): 84 resample_to = nib.load(resample_to) ---> 85 files, folder = fetcher() 86 template_dict = {} 87 for f in files: ~/.local/lib/python3.8/site-packages/AFQ/data/fetch.py in fetcher() 72 return files, folder 73 else: ---> 74 return _make_fetcher( 75 name, folder, baseurl, remote_fnames, local_fnames, 76 doc=doc, **make_fetcher_kwargs)() ~/.local/lib/python3.8/site-packages/dipy/data/fetcher.py in fetcher() 195 files[n] = (baseurl + f, md5_list[i] if 196 md5_list is not None else None) --> 197 fetch_data(files, folder, data_size) 198 199 if msg is not None: ~/.local/lib/python3.8/site-packages/dipy/data/fetcher.py in fetch_data(files, folder, data_size) 144 all_skip = False 145 _log('Downloading "%s" to %s' % (f, folder)) --> 146 _get_file_data(fullpath, url) 147 check_md5(fullpath, md5) 148 if all_skip: ~/.local/lib/python3.8/site-packages/dipy/data/fetcher.py in _get_file_data(fname, url) 94 95 def _get_file_data(fname, url): ---> 96 with contextlib.closing(urlopen(url)) as opener: 97 try: 98 response_size = opener.headers['content-length'] /usr/lib/python3.8/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context) 220 else: 221 opener = _opener --> 222 return opener.open(url, data, timeout) 223 224 def install_opener(opener): /usr/lib/python3.8/urllib/request.py in open(self, fullurl, data, timeout) 523 524 sys.audit('urllib.Request', req.full_url, req.data, req.headers, req.get_method()) --> 525 response = self._open(req, data) 526 527 # post-process response /usr/lib/python3.8/urllib/request.py in _open(self, req, data) 540 541 protocol = req.type --> 542 result = self._call_chain(self.handle_open, protocol, protocol + 543 '_open', req) 544 if result: /usr/lib/python3.8/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args) 500 for handler in handlers: 501 func = getattr(handler, meth_name) --> 502 result = func(*args) 503 if result is not None: 504 return result /usr/lib/python3.8/urllib/request.py in https_open(self, req) 1395 1396 def https_open(self, req): -> 1397 return self.do_open(http.client.HTTPSConnection, req, 1398 context=self._context, check_hostname=self._check_hostname) 1399 /usr/lib/python3.8/urllib/request.py in do_open(self, http_class, req, **http_conn_args) 1355 encode_chunked=req.has_header('Transfer-encoding')) 1356 except OSError as err: # timeout error -> 1357 raise URLError(err) 1358 r = h.getresponse() 1359 except: URLError:

[arokem]

Not 100% sure, but you can try (based on this page) to upgrade the certifi library:

pip install --upgrade certifi

Does that resolve your issue?

[sleeeeeeepy]

Thank you for the quick response. I tried upgrading certifi, but I still get the authentication error. Therefore, I was able to avoid the authentication error by typing the following command;

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

I then wanted to visualize all the bundle information, but I got an error here as well. How can I overcome this error? Best regards,

In [4]: from AFQ.api.participant import ParticipantAFQ output_directory = "./tract" myafq = ParticipantAFQ( "sub-0001_dwi.nii.gz", "sub-0001_dwi.bval", "sub-0001_dwi.bvec", output_dir = output_directory) In[5]: myafq.export_all() INFO:AFQ:Saving ./tract/clean_bundles/sub-0001_dwi_space-RASMM_model-probCSD_algo-AFQ_desc-ATRL_tractography.trk INFO:AFQ:Saving ./tract/clean_bundles/sub-0001_dwi_space-RASMM_model-probCSD_algo-AFQ_desc-ATRR_tractography.trk INFO:AFQ:Saving ./tract/clean_bundles/sub-0001_dwi_space-RASMM_model-probCSD_algo-AFQ_desc-CGCL_tractography.trk ・・・・・・ INFO:AFQ:Generating colorful lines from tractography... INFO:AFQ:Loading Stateful Tractogram... INFO:AFQ:Generating colorful lines from tractography... INFO:AFQ:Time taken for export all: 2086.1295886039734 In [6]: import matplotlib.pyplot as plt import nibabel as nib import plotly In[7]: bundle_html = myafq.export("all_bundles_figure") In[8]: plotly.io.show(bundle_html["01"][0])

TypeError Traceback (most recent call last) /tmp/ipykernel_3265/3135208643.py in ----> 1 plotly.io.show(bundle_html["01"][0]) TypeError: list indices must be integers or slices, not str

[arokem]

Looks like everything worked just fine. The error you see is a consequence of using a key that refers to a subject not in your dataset. But it looks like all the files were generated as expected. If you want to further discuss how to view the results, pop over to our discussions board and post there: https://github.com/yeatmanlab/pyAFQ/discussions, but I think that this issue can be closed.