Official detailed User Guide

[dPys]

• PyNets version:
• Python version:
• Operating System:

Description

Describe what you were trying to get done. Tell us what happened, what went wrong, and what you expected to happen.

What I Did

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

[Oblivion83]

Hi Derek,

Just started using Pynets a few days ago on Ubuntu 18.04, Python 3.6.5 (PyCharm 18.1).

I actually have a number of issues with Pynets, and I'm hoping you can help. I'll start with the problem closest to completion.

I have processed an .nii file as per Situation B with Powers atlas. However, when I try to visualize the results, I get an error message (Errno21):

---

import pandas as pd pickle_path = '/home/oblivion/coords_power_2011' df = pd.read_pickle(pickle_path) Traceback (most recent call last): File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 166, in try_read return read_wrapper(lambda f: pkl.load(f)) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 147, in read_wrapper is_text=False) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/common.py", line 406, in _get_handle f = open(path_or_buf, mode) IsADirectoryError: [Errno 21] Is a directory: '/home/oblivion/coords_power_2011' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 171, in try_read lambda f: pc.load(f, encoding=encoding, compat=False)) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 147, in read_wrapper is_text=False) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/common.py", line 406, in _get_handle f = open(path_or_buf, mode) IsADirectoryError: [Errno 21] Is a directory: '/home/oblivion/coords_power_2011' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 177, in read_pickle return try_read(path) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 175, in try_read lambda f: pc.load(f, encoding=encoding, compat=True)) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 147, in read_wrapper is_text=False) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/common.py", line 406, in _get_handle f = open(path_or_buf, mode) IsADirectoryError: [Errno 21] Is a directory: '/home/oblivion/coords_power_2011' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 166, in try_read return read_wrapper(lambda f: pkl.load(f)) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 147, in read_wrapper is_text=False) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/common.py", line 406, in _get_handle f = open(path_or_buf, mode) IsADirectoryError: [Errno 21] Is a directory: '/home/oblivion/coords_power_2011' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 171, in try_read lambda f: pc.load(f, encoding=encoding, compat=False)) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 147, in read_wrapper is_text=False) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/common.py", line 406, in _get_handle f = open(path_or_buf, mode) IsADirectoryError: [Errno 21] Is a directory: '/home/oblivion/coords_power_2011' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/code.py", line 91, in runcode exec(code, self.locals) File "", line 1, in File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 180, in read_pickle return try_read(path, encoding='latin1') File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 175, in try_read lambda f: pc.load(f, encoding=encoding, compat=True)) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/pickle.py", line 147, in read_wrapper is_text=False) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/io/common.py", line 406, in _get_handle f = open(path_or_buf, mode) IsADirectoryError: [Errno 21] Is a directory: '/home/oblivion/coords_power_2011'

---

Am I directing python to the correct directory? There are a few pkl files in coords_power_2011.

Hope you can give some suggestions.

Thanks :)

[dPys]

Hi @Oblivion83 ,

So it appears that you are listing your pickle path as a directory (i.e. and not file) path.

In other words, the path should actually be set as: pickle_path = '/home/oblivion/coords_power_2011/{pickle_file}.pkl'

Can you ls the contents of /home/oblivion/coords_power_2011 and please send the result? Thanks!

[Oblivion83]

Here's the ls of /home/oblivion/coords_power_2011:

"10004_b'Power 2011 atlas'_Default_func_adj_mat_sps_0.3_4_network.png" 10004_Default_est_sps_unthresholded_mat.npy 10004_Default_net_metrics_sps_0.3_4_neat.csv index.html "10004_b'Power 2011 atlas'_sps_Default_0.3_4_func_glass_viz.png" 10004_Default_fdg_sps_network.js 10004_Default_wb_net_ts.npy labelnames_plotting.pkl 10004_Default_connectogram_sps_network.js 10004_Default_fdg_sps_network.json coords_plotting.pkl net_metric_list_Default 10004_Default_connectogram_sps_network.json 10004_Default_net_metrics_sps_0.3_4 Default_func_coords_rsn.pkl 10004_Default_est_sps_0.3_4.npy 10004_Default_net_metrics_sps_0.3_4.csv Default_func_labelnames_rsn.pkl

---

Ok so I set the pickle_path to '/home/oblivion/coords_power_2011/coords_plotting.pkl' (as a test) and python didn't object. :)

Actually in the above folder, I noticed some CSV files like "10004_Default_net_metrics_sps_0.3_4.csv ", I'm guessing those are the outputs I will want play with? If so, is this bit:

import glob import os import pandas as pd

working_path = r'/work/04171/dpisner/data/ABM/network_analysis/' # use your path name_of_network_pickle = 'net_metrics_sps_Default_mean'

allFiles = [] for ID in os.listdir(working_path): path_name = workingpath + ID + '/' + ID + '' + name_of_network_pickle if os.path.isfile(path_name): print(path_name) allFiles.append(path_name)

frame = pd.DataFrame() list_ = []

for file_ in allFiles: df = pd.readpickle(file) nodecols = [s for s in list(df.columns) if isinstance(s, int) or any(c.isdigit() for c in s)] list.append(df)

frame = pd.concat(list_)

out_path = working_path + '/' + name_of_network_pickle + '_output.csv' frame.to_csv(out_path)

..............................no longer necessary?

Thanks so much for responding!

[dPys]

You're welcome! That bit is for concatenating all of the graph measures generated for the individual subjects into a group database :) So you will want to set the name_of_network_pickle = to 10004_Default_net_metrics_sps_0.3_4_neat.csv and your working_path to /home/oblivion/coords_power_2011 to run it. :)

[Oblivion83]

Tried: import pandas as pd Backend TkAgg is interactive backend. Turning interactive mode on. pickle_path = '/home/oblivion/coords_power_2011/coords_plotting.pkl' df = pd.read_pickle(pickle_path) df [(-16, 28, 52), (48, 35, -12), (-2, 41, 16), (11, -53, 16), (-2, 25, 43), (2, -23, 30), (-2, -35, 30), (-34, 55, 3), (-2, -37, 43), (-25, -39, -7), (-45, -60, 21), (-20, 44, 39), (27, 16, -16), (-6, 50, -1), (-43, -64, 34), (50, -28, -3), (-34, 19, 50), (52, -1, -16), (52, 7, -30), (-2, 44, -8), (-68, -23, -16), (-6, -55, 26), (-6, 48, 23), (22, 32, 48), (55, -43, 21), (34, 37, -12), (-2, 37, 35), (-48, 25, -1), (-20, -21, -19), (6, 41, -5), (6, -48, 30), (-45, 30, -12), (0, 30, 26), (9, 53, 3), (52, 32, 1), (-55, -12, -10), (-57, -30, -3), (-2, -48, 12), (6, 48, -14), (-27, 52, 21), (6, -59, 34), (11, 35, 19), (-52, 3, -26), (-11, -55, 16), (55, -44, 37), (22, 39, 39), (6, 53, 16), (52, -59, 35), (64, -12, -19), (-11, 44, 7), (34, 16, -7)] import glob import os import pandas as pd name_of_network_pickle = '10004_Default_net_metrics_sps_0.3_4_neat.csv' working_path = '/home/oblivion/coords_power_2011' allFiles = [] for ID in os.listdir(working_path): path_name = workingpath + ID + '/' + ID + '' + name_of_network_pickle if os.path.isfile(path_name): print(path_name) allFiles.append(pathname) frame = pd.DataFrame() list = [] for file_ in allFiles: df = pd.readpickle(file) nodecols = [s for s in list(df.columns) if isinstance(s, int) or any(c.isdigit() for c in s)] list.append(df) frame = pd.concat(list_) out_path = working_path + '/' + name_of_network_pickle + '_output.csv' frame.to_csv(out_path)

.....and got:

File "/usr/lib/python3.6/code.py", line 91, in runcode exec(code, self.locals) File "", line 16, in File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 225, in concat copy=copy, sort=sort) File "/home/oblivion/.local/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 259, in init raise ValueError('No objects to concatenate') ValueError: No objects to concatenate

---

Did I choose the right pickle_path?

[dPys]

So, you are currently trying to average with the coordinates of the graph...

You will want to set name_of_network_pickle = 'Default_net_metrics_sps'

[Oblivion83]

Sorry Derek still getting the same error.

pickle_path = '/home/oblivion/coords_power_2011/coords_plotting.pkl' working_path = r'/home/oblivion/coords_power_2011' name_of_network_pickle = '10004_Default_net_metrics_sps_0.3_4_neat.csv'

:(

[dPys]

Try running this:

import glob
import os
import pandas as pd

working_path =  r'/home/oblivion/coords_power_2011'
name_of_network_pickle = 'Default_net_metrics_sps'

allFiles = []
for ID in os.listdir(working_path):
path_name = working_path + ID + '/' + ID + '_' + name_of_network_pickle
if os.path.isfile(path_name):
print(path_name)
allFiles.append(path_name)

frame = pd.DataFrame()
list_ = []

for file_ in allFiles:
df = pd.read_pickle(file_)
node_cols = [s for s in list(df.columns) if isinstance(s, int) or any(c.isdigit() for c in s)]
list_.append(df)

frame = pd.concat(list_)

out_path = working_path + '/' + name_of_network_pickle + '_output.csv'
frame.to_csv(out_path)

[Oblivion83]

Nope, still didn't work, I even tried changing the pickle path to use other .pkl files, and still getting the same error :(

Really sorry about taking up so much of your time.

[dPys]

Hi @Oblivion83 Please send your revised code.

-Derek

[Oblivion83]

Hi Derek, thanks for checking back. I did a fresh install of Ubuntu 18.04 and re-loaded everything. ATM things are working fine for the most part (re situation A and B), although situation C took a really long time. Is that normal? I killed the process as I thought it had hung. I also got Errno12 message and my PC began to lag badly.

As for situation A and B, I have managed to run a number of scans. Here is an ls of /home/kavinash/coords_dosenbach_2010/:

10001_02_est_partcorr_0.2_4.npy 10001_est_partcorr_unthresh_mat.npy 10001_PA_02_net_metrics_partcorr_0.2_4.csv 10001_PA_net_metrics_partcorr_0.2_4_neat.csv 10002_net_metrics_partcorr_0.2_4_neat.csv 10004_wb_net_ts.npy 10001_02_est_partcorr_unthresh_mat.npy 10001_net_metrics_partcorr_0.2_4 10001_PA_02_net_metrics_partcorr_0.2_4_neat.csv 10001_PA_wb_net_ts.npy 10002_wb_net_ts.npy net_metric_list 10001_02_net_metrics_partcorr_0.2_4 10001_net_metrics_partcorr_0.2_4.csv 10001_PA_02_wb_net_ts.npy 10001_wb_net_ts.npy 10004_est_partcorr_0.2_4.npy 10001_02_net_metrics_partcorr_0.2_4.csv 10001_net_metrics_partcorr_0.2_4_neat.csv 10001_PA_est_partcorr_0.2_4.npy 10002_est_partcorr_0.2_4.npy 10004_est_partcorr_unthresh_mat.npy 10001_02_net_metrics_partcorr_0.2_4_neat.csv 10001_PA_02_est_partcorr_0.2_4.npy 10001_PA_est_partcorr_unthresh_mat.npy 10002_est_partcorr_unthresh_mat.npy 10004_net_metrics_partcorr_0.2_4 10001_02_wb_net_ts.npy 10001_PA_02_est_partcorr_unthresh_mat.npy 10001_PA_net_metrics_partcorr_0.2_4 10002_net_metrics_partcorr_0.2_4 10004_net_metrics_partcorr_0.2_4.csv 10001_est_partcorr_0.2_4.npy 10001_PA_02_net_metrics_partcorr_0.2_4 10001_PA_net_metrics_partcorr_0.2_4.csv 10002_net_metrics_partcorr_0.2_4.csv 10004_net_metrics_partcorr_0.2_4_neat.csv

Here is an ls for /home/kavinash/coords_power_2011/:

10001_coords_power_2011_Default_func_adj_mat_sps_0.3_4_network.png 10001_Default_wb_net_ts.npy 10004_Default_net_metrics_sps_0.3_4_neat.csv 10001_coords_power_2011_sps_Default_0.3_4_func_glass_viz.png 10004_coords_power_2011_Default_func_adj_mat_sps_0.3_4_network.png 10004_Default_wb_net_ts.npy 10001_Default_connectogram_sps_network.js 10004_coords_power_2011_sps_Default_0.3_4_func_glass_viz.png coords_plotting.pkl 10001_Default_connectogram_sps_network.json 10004_Default_connectogram_sps_network.js crash-20180626-132257-kavinash-extract_ts_rsn_coords_node-c971fad1-95c0-41c7-87fa-93b42d33c765.pklz 10001_Default_est_sps_0.3_4.npy 10004_Default_connectogram_sps_network.json crash-20180626-132300-kavinash-imp_est-061273c7-5611-4517-b7e1-0c64fed8c54b.pklz 10001_Default_est_sps_unthresholded_mat.npy 10004_Default_est_sps_0.3_4.npy Default_func_coords_rsn.pkl 10001_Default_fdg_sps_network.js 10004_Default_est_sps_unthresholded_mat.npy Default_func_labelnames_rsn.pkl 10001_Default_fdg_sps_network.json 10004_Default_fdg_sps_network.js index.html 10001_Default_net_metrics_sps_0.3_4 10004_Default_fdg_sps_network.json labelnames_plotting.pkl 10001_Default_net_metrics_sps_0.3_4.csv 10004_Default_net_metrics_sps_0.3_4 net_metric_list_Default 10001_Default_net_metrics_sps_0.3_4_neat.csv 10004_Default_net_metrics_sps_0.3_4.csv

Which files should I use for output visualization?

[dPys]

Yes, it is normal for example C is to take a long time, particularly if you are not running on HPC, because you are essentially iterating the pipeline across two different RSNs (Default & Cont) at 5 different thresholds, and therefore generating 10 different models. If you wish to restrict CPU/memory resources so that your PC doesn't lag, you can try adding the -pm 1,2 flag which will restrict the workflow to using only 1 core and 2 gb of RAM. Note that if you do this, the workflow will take longer to run. Also note that to perform any clustering with the workflow (see example D), it will minimally require 2 cores and 4 gb of RAM.

Finally, if you wish to plot/visualize the results, you just need to add the -plt flag to the command-line call.

derek;

[Oblivion83]

Hi Derek could I have your email address?

I'm actually new to all aspects of programming, and my new supervisor is making me learn by using PyNets. I'd like to continue our correspondence via e-mail, if its ok with you.

-Kavi

[dPys]

dpisner at utexas dot edu