Disappearing atlases

[jaredpz]

I'm noticing on the github that the only atlases under the atlas folder are miccai, power264, segmentation3, and segmentation6, however on at least one installation of XCP that I use I have many more atlases, which includes the following list:

aal glasser gordon333 harvardOxford jlf miccai power264 powerVoxel schaefer100 schaefer200 schaefer400 segmentation3 segmentation6 sharma talairach yeo

Why are these atlases not on github, are there some issues with using them? I've successfully run the fcon and roiquant modules using at least some of these atlases before, so if there was an issue I wasn't aware.

Would it be possible to get these atlases, or at least the gordon and schaefer atlases added to github?

Alternatively, I can just copy those atlas folders from my old install to the new one, but I'm unclear on whether or not there are any json files that need to be edited in order to make those atlases work.

Please advise.

[sattertt]

Seems important, Jared. @rastko or @cieslak any ideas?

Sent from my iPhone

On Jul 12, 2018, at 2:16 PM, jaredpz notifications@github.com<mailto:notifications@github.com> wrote:

I'm noticing on the github that the only atlases under the atlas folder are miccai, power264, segmentation3, and segmentation6, however on at least one installation of XCP that I use I have many more atlases, which includes the following list:

aal glasser gordon333 harvardOxford jlf miccai power264 powerVoxel schaefer100 schaefer200 schaefer400 segmentation3 segmentation6 sharma talairach yeo

Why are these atlases not on github, are there some issues with using them? I've successfully run the fcon and roiquant modules using at least some of these atlases before, so if there was an issue I wasn't aware.

Would it be possible to get these atlases, or at least the gordon and schaefer atlases added to github?

Alternatively, I can just copy those atlas folders from my old install to the new one, but I'm unclear on whether or not there are any json files that need to be edited in order to make those atlases work.

Please advise.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/PennBBL/xcpEngine/issues/120, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AG8Ix0kse7oqGdexDdEt21EWMzIBgRSvks5uF5JxgaJpZM4VNfDn.

[rciric]

Hello -- apologies about the incomplete documentation and the delay here. Many of the atlases aren't bundled with the pipeline system in order to limit the size of the repository. You can either copy the atlases from your old install into your new atlas directory or (in most cases -- not all are up yet) download them from here. (Gordon and most Schaefer atlases should be uploaded there in MNI space.)

Alternatively, you can just define the environmental variables BRAINATLAS and BRAINSPACE in your bash profile, and point them to the top-level directories that contain all atlas and spatial data and metadata (make sure to export those variables). Typically, at runtime, the pipeline system checks whether BRAINATLAS and BRAINSPACE point to valid directories; if they are undefined or invalid, it automatically initialises them as ${XCPEDIR}/atlas and ${XCPEDIR}/space, respectively.

[jaredpz]

Hey Rastko et al.

I was able to download and add the atlases, but I'm still not able to get them to be used for the modules: fcon, net, roiquant, and qcfc. Is there some metadata that needs to be edited somewhere to make the atlases available to XCP? I've tried running with the atlas parameters in each of those modules == all, and also specifying the specific atlases, but neither works. I only get the Power264 atlas, which comes installed with XCP. The ${subID}_atlas directory for each subject only has power264, plus segmentation and global in it, and the log files don't show any evidence that XCP is even looking for the atlases, even though they're explicitly listed in the design file.

I ran xcpReset after adding the new atlases to the ${XCPEDIR}/atlas but that also didn't help.

fc_akelkar_201807311347_TNI.SC.1.002_T001LOG.txt

An example subject log with verbosity = 2 is attached. Let me know if there's any additional info needed to help troubleshoot

[rciric]

Hmm -- let's try (1) looking at ${XCPEDIR}/space/MNI/MNI_atlas.json. Check whether all of the atlases are there. If they're not there, you should be able to add each by running ${XCPEDIR}/repairMetadata. (2) ensuring that any *_atlas variables in your design file are set to all or include the names that match all of the atlases that you want to run.

[jaredpz]

Looks like this is the culprit.

I don't have ${XCPEDIR}/repairMetadata , but I do see ${XCPEDIR}/utils/atlasMetadata

I'm still on v0.6, not yet on 0.6.1 if thats a new feature. Will the atlasMetadata function do what I need, or should I update before trying to fix the metadata?

[jaredpz]

It looks like repairMetadata is basically just a wrapper around atlasMetadata, so I tried to run atlasMetadata to add one atlas and got the errors listed below. Seems like my version of xcp has some bugs in the core json editing files.

As a result of this, the ${XCPEDIR}/space/MNI/MNI_atlas.json file is now empty, when before it had the metadata for power264. I can pull the old metadata file off github again, but seems like I won't be able to update the metadata for the new atlases until I'm able to pull v0.6.1 off github with these new features. Errors below

· gordon333 is already installed: /data/jag/jmedaglia/software/xcp/xcpEngine/atlas/gordon333
· Updating gordon333 metadata instead
/data/jag/jmedaglia/software/xcp/xcpEngine/core/functions/json_merge: line 2: -s: command not found
/data/jag/jmedaglia/software/xcp/xcpEngine/core/functions/json_keys: line 2: --raw-output: command not found
/data/jag/jmedaglia/software/xcp/xcpEngine/core/functions/json_object: line 2: . + {"gordon333" : {} }: command not found
/data/jag/jmedaglia/software/xcp/xcpEngine/core/functions/json_set: line 2: --raw-output: command not found
/data/jag/jmedaglia/software/xcp/xcpEngine/core/functions/json_set: line 2: --raw-output: command not found
/data/jag/jmedaglia/software/xcp/xcpEngine/core/functions/json_set: line 2: --raw-output: command not found
/data/jag/jmedaglia/software/xcp/xcpEngine/core/functions/json_set: line 2: --raw-output: command not found
/data/jag/jmedaglia/software/xcp/xcpEngine/core/functions/json_set: line 2: --raw-output: command not found
/data/jag/jmedaglia/software/xcp/xcpEngine/core/functions/json_set: line 2: --raw-output: command not found
/data/jag/jmedaglia/software/xcp/xcpEngine/core/functions/json_set: line 2: --raw-output: command not found
/data/jag/jmedaglia/software/xcp/xcpEngine/core/functions/json_print: line 2: --sort-keys: command not found```

[rciric]

Yep, atlasMetadata should work. It looks like your JQ_PATH isn't being read by atlasMetadata. Can you try running the following?

cat ${XCPEDIR}/core/global|grep JQ_PATH

If the JQ_PATH definition is missing, you should be able to restore it by either:

• running xcpReset
• manually editing the globals file to include export JQ_PATH=${XCPEDIR}/thirdparty/jq/jq-linux64.

I suspect, however, that it's not missing and you've found a bug. If this is the case, please let us know so that we can update atlasMetadata (and other metadata editor utilities) to read it at runtime. For now, you should be able to work around the bug by defining JQ_PATH in your bash profile as above.

[jaredpz]

Yep, JQ_PATH seems to be set fine in global

cat ${XCPEDIR}/core/global | grep JQ_PATH

# JQ_PATH stores a path to the local installation of JQ.
export JQ_PATH=/data/jag/jmedaglia/software/xcp/xcpEngine/thirdparty/jq/jq-linux64

[rciric]

With the latest commits on the v0.7.0 branch, metadata utilities (repairMetadata, atlasMetadata, standardSpace, and spaceMetadata) should automatically infer the JQ_PATH if none is otherwise provided. I also caught another bug affecting the atlas metadata build process (in the case that only the -d argument is provided) that should now be resolved.

[jaredpz]

Reopening this issue because I'm still having problems.

I've updated to v0.7.0 and run repairMetadata, which seems to have worked fine. I get a bunch of chmod permission change errors when running it, mostly in the .git directory, but I don't think thats an issue because the metadata files look good after.

Despite that, I am still not able to get the newly added atlases to work for fcon, net, roiquant, or qcfc.

I deleted all previous output and ran from scratch with v0.7.0, and based on the logs with -t 2 it seems like it still can't find the atlases. The group level atlas metadata file under ${xcpOutDir}/group/atlas.json has all of the new atlases in it, so at some level the pipeline is finding them, but the subject level atlas metadata file under ${xcpOutDir}/${id0}/${id1}/${id0}_${id1}_atlas/${id0}_${id1}_atlas.json only has the power, segmentation and global entries.

See attached subject level design file, log, and related metadata files. Let me know if there's any additional information that would be helpful.

Thanks

fc_jzimmerman_201808161709_TNI.SC.1.001_T001LOG.txt group_atlas.txt MNI_atlas.txt TNI.SC.1.001_T001_atlas.txt TNI.SC.1.001_T001.txt

[mattcieslak]

Here's one error:

>> ${XCPEDIR}/utils/nodeCoverage.R -i /data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.001/T001/prestats/TNI.SC.1.001_T001_mask.nii.gz -r /data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.001/T001/TNI.SC.1.001_T001_atlas/TNI.SC.1.001_T001_global.nii.gz -x /data/jag/jmedaglia/software/xcp070/xcpEngine/atlas/global/globalNodeIndex.1D -n /data/jag/jmedaglia/software/xcp070/xcpEngine/atlas/global/globalNodeNames.txt
Error in file(file, "rt") : cannot open the connection
Calls: as.vector -> unlist -> read.table -> file
In addition: Warning message:
In file(file, "rt") :
  cannot open file '/data/jag/jmedaglia/software/xcp070/xcpEngine/atlas/global/globalNodeIndex.1D': No such file or directory
Execution halted
· @1.3.4 Computing atlas statistics

@rciric what happened to atlas/global?

[mattcieslak]

also, @jaredpz could you attach your design file? Many of these errors might stop if you set roiquant_globals[cxt]=0. This may be related to #122.

[jaredpz]

The file TNI.SC.1.001_T001.txt is a subject level design file.

I saw that error with the global atlas, which was also there previously, I believe this is a bug. The pipeline level atlas metadata file MNI_atlas.json doesn't have an entry for global, and the directory /data/jag/jmedaglia/software/xcp070/xcpEngine/atlas/global doesn't exist.

At the subject level, the metadata file TNI.SC.1.001_T001_atlas.txt has an entry for global which looks as follows

"global": {
    "Map": "/data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.001/T001/TNI.SC.1.001_T001_atlas/TNI.SC.1.001_T001_global.nii.gz",
    "NodeIndex": "/data/jag/jmedaglia/software/xcp070/xcpEngine/atlas/global/globalNodeIndex.1D",
    "NodeNames": "/data/jag/jmedaglia/software/xcp070/xcpEngine/atlas/global/globalNodeNames.txt",
    "RegionalMeanAlff": "/data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.001/T001/roiquant/global/TNI.SC.1.001_T001_global_mean_alff.csv",
    "RegionalMeanAlffZ": "/data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.001/T001/roiquant/global/TNI.SC.1.001_T001_global_mean_alffZ.csv",
    "RegionalMeanReho": "/data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.001/T001/roiquant/global/TNI.SC.1.001_T001_global_mean_reho.csv",
    "RegionalMeanRehoZ": "/data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.001/T001/roiquant/global/TNI.SC.1.001_T001_global_mean_rehoZ.csv",
    "Space": "TNI.SC.1.001_T001_fc",
    "Type": "Map"
  },

Everything in this except the NodeIndex and NodeNames fields are referencing files made at the subject level. The map is pointing to the whole-brain mask made during the prestats module, so the global atlas is really just a whole-brain mask, in which case NodeNames and NodeIndex aren't really meaningful fields because there is only a single node.

My guess is that when roiquant_globals[cxt] design variable is set to 1, the pipeline adds this global entry to the subject level atlas metadata file and then tries to compute a global mean value for each map put into roiquant. In that process it calls ${XCPEDIR}/utils/nodeCoverage.R which requires NodeIndex and NodeNames inputs, but these are not needed for global. I would suspect the nodeCoverage.R step is probably not needed at all for doing global mean roiquant, because the global map is derived from the subject EPI instead of from an atlas that's registered to the subject EPI. By definition we will get complete node coverage for this 'atlas.'

I also believe that this error is not related to the main problem I'm having which is that the pipeline seems to be unable to find the atlases that I have installed, namely the gordon and schaefer atlases. For example, the log file looks like the fcon module basically doesn't do anything besides linking the reference volume, and it produces no errors and proceeds to the next step:

###################################################################
#  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  #
#                                                                 #
#  ☭               FUNCTIONAL CONNECTOME MODULE                ☭  #
#                                                                 #
#  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  ☭  #
###################################################################

[I][/data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.001/T001/TNI.SC.1.001_T001.nii.gz]
[O][/data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.001/T001/fcon]
· @0.1 
                             -   -   -
>> ln -sf /data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.001/T001/prestats/TNI.SC.1.001_T001_referenceVolume.nii.gz /data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.001/T001/fcon/TNI.SC.1.001_T001_referenceVolume.nii.gz

Module Workflow Map
····································································
· START
· @0.1
· FINISH
····································································

Module complete

nets module looks the same.

I can run the pipeline again, bumping up the verbosity if you think it will be helpful for troubleshooting, however IIRC when I did that previously on v0.6.1 it was not particularly helpful.

[mattcieslak]

We merged 0.6.1 and 0.7 are working to get all the bugs out before the paper is published. If you send me a copy of your design and cohort file I will run it on my local version to see if I can reproduce the errors.

[jaredpz]

I've copied the design, cohort file, and input images for one example subject to the following directory:

/data/joy/BBL-extend/share/from_jaredz

Thanks for your help Matt.

[rciric]

I found part of the problem. Specifically, in both the space- and group-level metadata, the full paths were missing:

  "power264": {
    "Citation": "Reference.bib",
    "CommunityNamesAPriori": "/data/jag/jmedaglia/software/xcp070/xcpEngine/atlas/power264/power264CommunityNames.txt",
    "CommunityPartitionAPriori": "CommunityAffiliation.1D",
    "Map": "MNI.nii.gz",
    "NodeIndex": "NodeIndex.1D",
    "NodeNames": "NodeNames.txt",
    "Space": "MNI",
    "SpaceNative": "MNI",
    "Type": "Map"
  }

Note, for instance, that Map points to simply MNI.nii.gz instead of ${BRAINATLAS}/power264/power264MNI.nii.gz. This was indeed a bug -- thanks for your help in pointing this out. The latest commit of atlasMetadata should resolve this aspect of the problem.

[jaredpz]

Thanks Rastko, I'll update and try running again

[jaredpz]

Okay, so the latest commit seems to have fixed most of the issues. fcon, nets, and qcfc now run to completion and I'm seeing most of the files I expect. roiquant however is not producing any output for the new atlases (except folders for each). In the log file I see that every time ${XCPEDIR}/utils/quantifyAtlas is called it prints the function usage, so I'm assuming something is not working here. Probably related to the issue below.

The other obvious remaining issue (which I assume is related) is that the link in the fcon module that is supposed to point to the atlas in subject EPI space is broken. It points to the subject level atlas folder, and in that folder, the file that is supposed to be the atlas is just a broken recursive link to itself. See example below.

ls -lth /data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.002/T001/TNI.SC.1.002_T001_atlas

total 76K
-rw-rw----+ 1 jzimmerman medaglia_group  16K Aug 24 18:00 TNI.SC.1.002_T001_atlas.json
-rw-rw----+ 1 jzimmerman medaglia_group  22K Aug 24 18:00 TNI.SC.1.002_T001_segmentation.nii.gz
lrwxrwxrwx  1 jzimmerman medaglia_group  107 Aug 24 18:00 TNI.SC.1.002_T001_global.nii.gz -> /data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.002/T001/prestats/TNI.SC.1.002_T001_mask.nii.gz
lrwxrwxrwx  1 jzimmerman medaglia_group  129 Aug 24 18:00 TNI.SC.1.002_T001_schaefer400.nii.gz -> /data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.002/T001/TNI.SC.1.002_T001_atlas/TNI.SC.1.002_T001_schaefer400.nii.gz
lrwxrwxrwx  1 jzimmerman medaglia_group  129 Aug 24 17:59 TNI.SC.1.002_T001_schaefer200.nii.gz -> /data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.002/T001/TNI.SC.1.002_T001_atlas/TNI.SC.1.002_T001_schaefer200.nii.gz
lrwxrwxrwx  1 jzimmerman medaglia_group  129 Aug 24 17:59 TNI.SC.1.002_T001_schaefer100.nii.gz -> /data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.002/T001/TNI.SC.1.002_T001_atlas/TNI.SC.1.002_T001_schaefer100.nii.gz
lrwxrwxrwx  1 jzimmerman medaglia_group  127 Aug 24 17:59 TNI.SC.1.002_T001_gordon333.nii.gz -> /data/jag/jmedaglia/TNI_study/rsfmriPreproc/output/TNI.SC.1.002/T001/TNI.SC.1.002_T001_atlas/TNI.SC.1.002_T001_gordon333.nii.gz
-rw-rw----+ 1 jzimmerman medaglia_group 7.2K Aug 24 17:00 TNI.SC.1.002_T001_power264.nii.gz

The final thing I'm noticing absent is the missing nodes files for each atlas. The subject level atlas metadata file has an entry for missing nodes files for each atlas, but the file doesn't exit. Will this exist only if there are nodes missing, or is it only made in the case that some nodes don't get registered to the subject EPI? Even for the power264 atlas, which has an atlas map in the subject level atlas folder I don't have a missing node file.

Attached are an example subject level atlas metadata file and the log file for this most recent run.

fc_jzimmerman_201808241641_TNI.SC.1.002_T001LOG.txt TNI.SC.1.002_T001_atlas.txt

[mattcieslak]

I think I got it working. Could you check /data/joy/BBL-extend/share/jaredz_test/cfn_output/ to see if it has everything you want?

[jaredpz]

Yeah, looks like it's all set. Do I just need to update and rerun to fix?

[mattcieslak]

If you're running from a version cloned from git you should do (from the xcpEngine directory)

git pull origin master
export XCPEDIR=`pwd`
bash xcpReset
wget https://www.dropbox.com/s/92i491mrtslb56i/space.tar.xz
tar xvfJm space.tar.xz
bash utils/repairMetadata

Make sure if you set XCPEDIR to something in a startup script that it is correct there too. All the atlases you want to use are included in xcp now so you shouldn't have to mess with BRAINATLAS anymore. In fact, you should probably unset BRAINATLAS before running the above commands.

The size of your fmri scan was causing RNifti to crash. You may need to update RNifti too. Let me know if it works!

[mattcieslak]

Also make sure you request lots of memory. I needed 18G to process your dataset