search

PennLINC / qsirecon

Reconstruction of preprocessed q-space images (dMRI)
https://qsirecon.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
3 stars 1 forks source link

network connection required for mrtrix_multishell_msmt_pyafq_tractometry? #15

Open DVSneuro opened 2 months ago

DVSneuro commented 2 months ago

Summary

We were trying to use the mrtrix_multishell_msmt_pyafq_tractometry recon specification on our local computing cluster and we are getting network connection errors (see below). The computing nodes don't connect out to the internet, so I'm wondering if there are files we can download to prevent QSIprep from trying to retrieve files from the internet?

Additional details

What were you trying to do?

We were trying to run mrtrix_multishell_msmt_pyafq_tractometry.

What did you expect to happen?

We expected the program to run, but it seems to require a network connection to retrieve other files.

What actually happened?

Network errors: OSError: [Errno 101] Network is unreachable The full contents of crash file is pasted below:

Node: qsirecon_wf.sub-10918_mrtrix_multishell_msmt_pyafq_tractometry.sub_10918_space_T1w_desc_preproc_recon_wf.pyafq_tractometry.run_afq
Working directory: /scratch/qsirecon_wf/sub-10918_mrtrix_multishell_msmt_pyafq_tractometry/sub_10918_space_T1w_desc_preproc_recon_wf/pyafq_tractometry/run_afq

Node inputs:

bval_file = <undefined>
bvec_file = <undefined>
dwi_file = <undefined>
itk_file = <undefined>
kwargs = {'directions': 'prob', 'max_angle': 30.0, 'sphere': None, 'seed_mask': None, 'seed_threshold': 0, 'n_seeds': 1, 'random_seeds': False, 'rng_seed': None, 'stop_mask': None, 'stop_threshold': 0, 'step_size': 0.5, 'min_length': 50, 'max_length': 250, 'odf_model': 'CSD', 'tracker': 'local', 'nb_points': False, 'nb_streamlines': False, 'seg_algo': 'AFQ', 'reg_algo': None, 'clip_edges': False, 'parallel_segmentation': {'n_jobs': -1, 'engine': 'joblib', 'backend': 'loky'}, 'progressive': True, 'greater_than': 50, 'rm_small_clusters': 50, 'model_clust_thr': 1.25, 'reduction_thr': 25, 'refine': False, 'pruning_thr': 12, 'b0_threshold': 50, 'prob_threshold': 0, 'roi_dist_tie_break': False, 'dist_to_waypoint': None, 'rng': None, 'return_idx': False, 'presegment_bundle_dict': None, 'presegment_kwargs': {}, 'filter_by_endpoints': True, 'dist_to_atlas': 4, 'save_intermediates': None, 'n_points': 100, 'clean_rounds': 5, 'distance_threshold': 3, 'length_threshold': 4, 'min_sl': 20, 'stat': 'mean', 'min_bval': None, 'max_bval': None, 'filter_b': True, 'robust_tensor_fitting': False, 'csd_response': None, 'csd_sh_order': None, 'csd_lambda_': 1, 'csd_tau': 0.1, 'gtol': 0.01, 'brain_mask_definition': None, 'bundle_info': None, 'reg_template_spec': 'mni_T1', 'mapping_definition': None, 'reg_subject_spec': 'power_map', 'profile_weights': 'gauss', 'scalars': ['dti_fa', 'dti_md'], 'import_tract': None, 'sbv_lims_bundles': [None, None], 'volume_opacity_bundles': 0.3, 'n_points_bundles': 40, 'sbv_lims_indiv': [None, None], 'volume_opacity_indiv': 0.3, 'n_points_indiv': 40, 'viz_backend_spec': 'plotly_no_gif', 'virtual_frame_buffer': False, 'omp_nthreads': 8}
mask_file = <undefined>
tck_file = <undefined>

Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node run_afq.

Traceback:
    Traceback (most recent call last):
      File "/opt/conda/envs/qsiprep/lib/python3.10/urllib/request.py", line 1348, in do_open
        h.request(req.get_method(), req.selector, req.data, headers,
      File "/opt/conda/envs/qsiprep/lib/python3.10/http/client.py", line 1283, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/opt/conda/envs/qsiprep/lib/python3.10/http/client.py", line 1329, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/opt/conda/envs/qsiprep/lib/python3.10/http/client.py", line 1278, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/opt/conda/envs/qsiprep/lib/python3.10/http/client.py", line 1038, in _send_output
        self.send(msg)
      File "/opt/conda/envs/qsiprep/lib/python3.10/http/client.py", line 976, in send
        self.connect()
      File "/opt/conda/envs/qsiprep/lib/python3.10/http/client.py", line 1448, in connect
        super().connect()
      File "/opt/conda/envs/qsiprep/lib/python3.10/http/client.py", line 942, in connect
        self.sock = self._create_connection(
      File "/opt/conda/envs/qsiprep/lib/python3.10/socket.py", line 845, in create_connection
        raise err
      File "/opt/conda/envs/qsiprep/lib/python3.10/socket.py", line 833, in create_connection
        sock.connect(sa)
    OSError: [Errno 101] Network is unreachable

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
        runtime = self._run_interface(runtime)
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsiprep/interfaces/pyafq.py", line 110, in _run_interface
        myafq.export_all()
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/AFQ/api/participant.py", line 182, in export_all
        export_all_helper(self, seg_algo, xforms, indiv, viz)
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/AFQ/api/utils.py", line 124, in export_all_helper
        api_afq_object.export("indiv_bundles")
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/AFQ/api/participant.py", line 154, in export
        return self.wf_dict[section][attr_name]
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/calculation.py", line 470, in __getitem__
        self._run_node(self.plan.efferents[k])
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/calculation.py", line 534, in _run_node
        if not found: res = node(self)
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/calculation.py", line 90, in __call__
        args.append(opts[name])
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/util.py", line 889, in __getitem__
        return self._examine_val(k, ps.PMap.__getitem__(self, k))
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/util.py", line 885, in _examine_val
        val = val()
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/util.py", line 1164, in <lambda>
        def curry_choice(k, args): return lambda:choose_fn(k, args)
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/util.py", line 1133, in _choose_last
        return vs[-1][k]
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/calculation.py", line 470, in __getitem__
        self._run_node(self.plan.efferents[k])
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/calculation.py", line 534, in _run_node
        if not found: res = node(self)
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/calculation.py", line 90, in __call__
        args.append(opts[name])
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/util.py", line 889, in __getitem__
        return self._examine_val(k, ps.PMap.__getitem__(self, k))
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/util.py", line 885, in _examine_val
        val = val()
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/util.py", line 1164, in <lambda>
        def curry_choice(k, args): return lambda:choose_fn(k, args)
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/util.py", line 1133, in _choose_last
        return vs[-1][k]
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/calculation.py", line 470, in __getitem__
        self._run_node(self.plan.efferents[k])
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/calculation.py", line 534, in _run_node
        if not found: res = node(self)
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/pimms/calculation.py", line 91, in __call__
        result = self.function(*args)
      File "<string>", line 2, in wrapper_has_args_func
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/AFQ/tasks/decorators.py", line 133, in wrapper_as_file
        img_trk_np_or_csv, meta = func(*args[:og_arg_count], **kwargs)
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/AFQ/tasks/segmentation.py", line 60, in segment
        bundles = segmentation.segment(
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/AFQ/segmentation.py", line 345, in segment
        fiber_groups = self.segment_afq()
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/AFQ/segmentation.py", line 607, in segment_afq
        for bundle_idx, bundle in enumerate(self.bundle_dict):
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/AFQ/api/bundle_dict.py", line 373, in __iter__
        self.gen_all()
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/AFQ/api/bundle_dict.py", line 308, in gen_all
        self.load_templates()
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/AFQ/api/bundle_dict.py", line 202, in load_templates
        afd.read_templates(as_img=False)
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/AFQ/data/fetch.py", line 571, in read_templates
        template_dict = _fetcher_to_template(
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/AFQ/data/fetch.py", line 89, in _fetcher_to_template
        files, folder = fetcher()
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/AFQ/data/fetch.py", line 78, in fetcher
        return _make_fetcher(
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/dipy/data/fetcher.py", line 197, in fetcher
        fetch_data(files, folder, data_size)
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/dipy/data/fetcher.py", line 146, in fetch_data
        _get_file_data(fullpath, url)
      File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/dipy/data/fetcher.py", line 96, in _get_file_data
        with contextlib.closing(urlopen(url)) as opener:
      File "/opt/conda/envs/qsiprep/lib/python3.10/urllib/request.py", line 216, in urlopen
        return opener.open(url, data, timeout)
      File "/opt/conda/envs/qsiprep/lib/python3.10/urllib/request.py", line 519, in open
        response = self._open(req, data)
      File "/opt/conda/envs/qsiprep/lib/python3.10/urllib/request.py", line 536, in _open
        result = self._call_chain(self.handle_open, protocol, protocol +
      File "/opt/conda/envs/qsiprep/lib/python3.10/urllib/request.py", line 496, in _call_chain
        result = func(*args)
      File "/opt/conda/envs/qsiprep/lib/python3.10/urllib/request.py", line 1391, in https_open
        return self.do_open(http.client.HTTPSConnection, req,
      File "/opt/conda/envs/qsiprep/lib/python3.10/urllib/request.py", line 1351, in do_open
        raise URLError(err)
    urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>

Reproducing the bug

Here's the code to reproduce the bug:

singularity run --cleanenv \
    -B ${TEMPLATEFLOW_DIR}:/opt/templateflow \
    -B ${MPLCONFIGDIR_DIR}:/opt/mplconfigdir \
    -B $maindir:/base \
    -B ~/work/tools/licenses:/opts \
    -B $scratchdir:/scratch \
    ~/work/tools/qsiprep-0.21.4.sif \
    /base/bids /base/derivatives/qsirecon-tractometry \
    participant --participant_label $sub \
    --output-resolution 2 \
    --recon_input /base/derivatives/qsiprep \
    --recon_spec mrtrix_multishell_msmt_pyafq_tractometry \
    --fs-license-file /opts/fs_license.txt \
    --skip_bids_validation \
    -w /scratch

Note that the dataset is BIDS valid. We had to use --skip_bids_validation because we were getting issues similar to this Neurostars post with a large dataset (https://neurostars.org/t/clarifying-file-reading-in-bids-validator-vs-fmriprep/5195/8).

In case it's helpful, these data are on OpenNeuro: https://openneuro.org/datasets/ds005123/versions/1.0.1

Thanks for any help!

Best wishes, David

arokem commented 2 months ago

This happens because pyAFQ downloads its templates/waypoints/etc. on runtime, but may be addressed by the integration of pyAFQ download into the image in PennLINC/qsirecon#17.

@36000: does pyAFQ download also run the templateflow code to download the MNI template? That might still be an issue in this scenario.