bids-standard / bids-specification

Brain Imaging Data Structure (BIDS) Specification
https://bids-specification.readthedocs.io/
Creative Commons Attribution 4.0 International
279 stars 163 forks source link

Add support for HiP-CT #1646

Closed balbasty closed 5 months ago

balbasty commented 1 year ago

Dear BIDS community

Hierarchical Phase-Contrast Tomography (HiP-CT) is an ex vivo X-ray technique that leverages high-energy X-ray sources -- specifically, the ESRF's Extremely Brilliant Source (EBS). HiP-CT is non-destructive, reaches resolutions of 1 to 10 um, and allows specific ROIs to be imaged within the sample without physical sectioning. Although the modality is obviously not widely available, the UCL group has a relatively streamlined acquisition process and we will be collecting data on multiple brains in the coming years.

We would like to add support for HiP-CT in BIDS, and we think it most naturally fits as a new modality suffix within the microscopy extension. This makes sense because micro-CT is already supported as a microscopy suffix. Also, the microscopy extension already includes tags that are required by HiP-CT data (for example, the existence of multiple images of the same samples, potentially with different resolutions and different field-of-views, labeled as chunk-). We are therefore proposing to add the hipCT suffix to the microscopy extension.

We have uploaded an exampled dataset that follows the proposed convention to DANDI: https://dandiarchive.org/dandiset/000026/draft/files?location=sub-I58%2Fses-Hip-CT%2Fmicr&page=1 We also provide three neuroglancer links that allow each volume to be visualized online, directly streaming the dandi data:

Let me also include a typical metadata file provided by the UCL group:

############# DOI ###################

Title       = "U.N."
Abstract    = "U.N."
Users       = "U.N."

############ proposal ###############

proposalTitle   ="U.N."
proposalNumber  ="md190"
proposers       ="U.N."
experimentType  ="U.N."
technique       ="U.N."
beamlineName    ="BM18"
instrumentName  ="U.N."

############ patient ################

PatientInstitue ="U.N."
PatientNumber   ="I58-LH"
PatientAge      ="U.N. years"
PatientSex      ="U.N."
PatientWeight   ="U.N. kg"
PatientSize     ="U.N. cm"
PatientMedInfo  ="U.N."

########### sample ################

SampleName  ="I58-LH_brain"
SampleInfo  ="complete heart from the body donor program of the Laboratoire d'Anatomie des Alpes Francaise (LADAF)"
SamplePrep  ="formalin fixed, progressive transfer to ethanol 70%. Vacuum degassing at each transfer to new ethanol bath, mounted with agar crushed gel at 70% ethanol"

########### scan parameters ###########

StartTime       ="20/01/2023"
SRCurrent       ="200mA mA"
IDName          ="BM18"
ScanRadix       ="HA-2300_15.13um_I58-LH_brain_C"
XStep           ="U.N."
XStages         ="U.N."
YStep           ="U.N."
YStages         ="U.N."
ZStep           ="5"
ZStages         ="32"
Projections     ="15000"
RefN            ="U.N."
DarkN           ="U.N."
RefOn           ="U.N."
ScanningMode    ="U.N."
ExposureTime    ="0.014 s"
AccExposureTime ="0.07 s"
AccFramesCount  ="5"
PixelSize       ="15.13 um"
PropDistance    ="15000 mm"
Filters         ="Sapphire 10mm SiO2 15mm"
Det_Avg_Energy  ="90 keV"
ScanGeometry    ="half-acquisition"
ScanRange       ="360 deg"
SensorName      ="IRIS15"
SensorMode      ="U.N."
SensorPixelSize ="4.25 um"
Magnification   ="0.3"
XPixelNum       ="5046"
YPixelNum       ="400"
OpticsType      ="Dzoom"
Scintillator    ="LuAG 2000 reflective"
SurfDoseRate    ="26.9 Gy/s"
ScanTime        ="20"
SeriesTime      ="10.7"

######### processing ############

RefApproach     ="U.N."
VolumeX         ="9413"
VolumeY         ="9413"
VolumeZ         ="10656"
32to16BitsMin   ="-0.17"
32to16BitsMax   ="0.31"
Jp2ComprRatio   ="10"

And the corresponding BIDS json file

{
    "IntendedFor":"sub-I58_sample-01_chunk-01_hipCT.ome.json",
    "Manufacturer":"ESRF",
    "StationName": "BM18",
    "InstitutionName":"European Synchrotron Radiation Facility",
    "InstitutionAddress":"71 Av. des Martyrs, 38000 Grenoble, France",
    "PixelSize":"[15.13, 15.13, 15.13]",
    "PixelSizeUnits":"um",
    "OtherAcquisitionParameters": "{\"StartTime\": \"20/01/2023\", \"SRCurrent\": \"200mA mA\", \"IDName\": \"BM18\", \"ScanRadix\": \"HA-2300_15.13um_I58-LH_brain_C\", \"ZStep\": \"5\", \"ZStages\": \"32\", \"Projections\": \"15000\", \"ExposureTime\": \"0.014 s\", \"AccExposureTime\": \"0.07 s\", \"AccFramesCount\": \"5\", \"PixelSize\": \"15.13 um\", \"PropDistance\": \"15000 mm\", \"Filters\": \"Sapphire 10mm SiO2 15mm\", \"Det_Avg_Energy\": \"90 keV\", \"ScanGeometry\": \"half-acquisition\", \"ScanRange\": \"360 deg\", \"SensorName\": \"IRIS15\", \"SensorPixelSize\": \"4.25 um\", \"Magnification\": \"0.3\", \"XPixelNum\": \"5046\", \"YPixelNum\": \"400\", \"OpticsType\": \"Dzoom\", \"Scintillator\": \"LuAG 2000 reflective\", \"SurfDoseRate\": \"26.9 Gy/s\", \"ScanTime\": \"20\", \"SeriesTime\": \"10.7\"}",
    "BodyPart":"Brain",
    "BodyPartDetails":"Left Hemisphere",
    "SampleEnviornment":"ex vivo",
    "SampleEmbedding":"agar crushed gel at 70% ethanol",
    "SampleFixation":"formalin",
    "SampleExtractionInstitution":"Massachusetts General Hospital"
}

Currently, the HiP-CT acquisition parameters are stored in the "OtherAcquisitionParameters" field, as a serialized JSON dictionary. But it may be better to move some of these HiP-CT-specific tags into the BIDS spec, and we would welcome any feedback from users of the other microscopy modalities.

@satra @Kevancic @dstansby

Thanks Yael

Remi-Gau commented 1 year ago

out of curiosity would it not be an issue if we one day wanted to integrate CT in BIDS: see https://bids.neuroimaging.io/bep024 ?

Remi-Gau commented 1 year ago

an issue in terms of compatibility or ease of use (where should my CT data go?)

balbasty commented 1 year ago

This is a good question.

I am a bit wary of having HiP-CT as a special case of CT for several reasons:

That said, I am neither a CT nor a HiP-CT expert, let me try to get someone who knows what they are talking about to chime in.

ucbpclw commented 1 year ago

Hi all, thought I'd jump in here to add some extra info regarding HiP-CT. @balbasty is correct for all the above. I think to summarise and add some of my own reasons to separate HiP-CT from CT: 1) HiP-CT is ex vivo only 2) It replies of propagation based phase-contrast 3) It has a hierarchical resolution format, i.e. the whole organ is imaged at an overview resolution and then specific areas can be imaged at higher resolution creating multi-scale datasets of single organ 4) As mentioed datasets are large, some of our largest aquired brain data sets are now at 14TB for one sample. 5) HiP-CT is a synchrotron technique, meaning that the meta-data fields for both acquisition and also reconstruction will be quite different and varied compared with a clinical or even a lab based CT system.

Also there is more detail on the technique in this paper , and you can see some video that give a clearer idea of the technique here.

Remi-Gau commented 1 year ago

OK it feels pretty clear from what you are saying that this is quite different from clinical CT.

From what I understand it looks like adding a microscopy suffix is needed to start supporting this.

Currently, the HiP-CT acquisition parameters are stored in the "OtherAcquisitionParameters" field, as a serialized JSON dictionary.

In many cases BIDS lets you add extra metadata, so no need to to have it serialized however better in this case to start listing the metadata that will be REQUIRED for downstream analysis and add those to the bids spec

satra commented 1 year ago

@balbasty - do you want to start a PR to the spec for this?

Remi-Gau commented 1 year ago

I would not mind also seeing an example added to the bids example repo: https://github.com/bids-standard/bids-examples#contributing

satra commented 1 year ago

looks like adding a microscopy suffix

you mean under the micr prefix. the suffix is just hipCT as proposed right?

seeing an example added to the bids example

perhaps a downscaled version for the moment could work. may be just at the coarsest levels. one of the the images that @balbasty linked to has 1 million objects :)

balbasty commented 1 year ago

Sure I can draft a PR.

The one thing I am not sure yet is which metadata should be added (if any). @ucbpclw do you think any of the metadata should be REQUIRED or RECOMMENDED to store in the sidecar json file?

And don't worry I won't push terabytes of data to the example repo! Something like the 1mm resolution level would make sense I'd say.

effigies commented 1 year ago

I don't think we inspect the contents of zarr directories. You could drop everything but any metadata files inside that may need to be inspected to ensure consistency between OME and BIDS.

ucbpclw commented 1 year ago

Do you mean which of metadata fields we sent in our example file are essential?

C


From: Yaël Balbastre @.> Sent: 08 November 2023 16:57 To: bids-standard/bids-specification @.> Cc: Walsh, Claire @.>; Mention @.> Subject: Re: [bids-standard/bids-specification] Add support for HiP-CT (Issue #1646)

⚠ Caution: External sender

Sure I can draft a PR.

The one thing I am not sure yet is which metadata should be added (if any). @ucbpclwhttps://github.com/ucbpclw do you think any of the metadata should be REQUIRED or RECOMMENDED to store in the sidecar json file?

And don't worry I won't push terabytes of data to the example repo! Something like the 1mm resolution level would make sense I'd say.

— Reply to this email directly, view it on GitHubhttps://github.com/bids-standard/bids-specification/issues/1646#issuecomment-1802291066, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADEWAE6FHY2WTDKDIKYSERLYDO2V3AVCNFSM6AAAAAA6WEWN66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBSGI4TCMBWGY. You are receiving this because you were mentioned.Message ID: @.***>

Remi-Gau commented 1 year ago

to illustrate what @effigies said, here are the content of the microscopy examples:

as you can see we tend to truncate large files to keep the repo light: https://github.com/bids-standard/bids-examples/blob/master/CONTRIBUTING.md#why-do-we-only-host-truncated-data-with-0kb-size

$ tree --si micr_S* 

[4.1k]  micr_SEM
├── [ 157]  dataset_description.json
├── [ 651]  participants.json
├── [  98]  participants.tsv
├── [ 168]  README
├── [ 330]  samples.json
├── [  60]  samples.tsv
└── [4.1k]  sub-01
    ├── [4.1k]  ses-01
    │   └── [4.1k]  micr
    │       ├── [   1]  sub-01_ses-01_sample-A_photo.jpg
    │       ├── [ 137]  sub-01_ses-01_sample-A_photo.json
    │       ├── [ 373]  sub-01_ses-01_sample-A_SEM.json
    │       └── [   1]  sub-01_ses-01_sample-A_SEM.png
    ├── [4.1k]  ses-02
    │   └── [4.1k]  micr
    │       ├── [ 137]  sub-01_ses-02_sample-A_photo.json
    │       ├── [   1]  sub-01_ses-02_sample-A_photo.tif
    │       ├── [ 373]  sub-01_ses-02_sample-A_SEM.json
    │       └── [   1]  sub-01_ses-02_sample-A_SEM.png
    ├── [ 156]  sub-01_sessions.json
    └── [  74]  sub-01_sessions.tsv
[4.1k]  micr_SEMzarr
├── [ 157]  dataset_description.json
├── [ 651]  participants.json
├── [  98]  participants.tsv
├── [ 210]  README
├── [ 330]  samples.json
├── [  60]  samples.tsv
└── [4.1k]  sub-01
    ├── [4.1k]  ses-01
    │   └── [4.1k]  micr
    │       ├── [ 373]  sub-01_ses-01_sample-A_SEM.json
    │       ├── [   1]  sub-01_ses-01_sample-A_SEM.png
    │       ├── [  70]  sub-01_ses-01_sample-A_SPIM.json
    │       └── [4.1k]  sub-01_ses-01_sample-A_SPIM.ome.zarr
    │           └── [4.1k]  0
    │               └── [4.1k]  0
    │                   └── [ 128]  0
    ├── [4.1k]  ses-02
    │   └── [4.1k]  micr
    │       ├── [ 373]  sub-01_ses-02_sample-A_SEM.json
    │       └── [   1]  sub-01_ses-02_sample-A_SEM.png
    ├── [ 156]  sub-01_sessions.json
    └── [  74]  sub-01_sessions.tsv
[4.1k]  micr_SPIM
├── [ 158]  dataset_description.json
├── [ 388]  participants.json
├── [  54]  participants.tsv
├── [ 616]  README
├── [ 330]  samples.json
├── [  83]  samples.tsv
└── [4.1k]  sub-01
    └── [4.1k]  micr
        ├── [ 387]  sub-01_sample-A_photo.json
        ├── [   1]  sub-01_sample-A_photo.png
        ├── [ 635]  sub-01_sample-A_stain-LFB_chunk-01_SPIM.json
        ├── [2.7k]  sub-01_sample-A_stain-LFB_chunk-01_SPIM.ome.tif
        ├── [ 637]  sub-01_sample-A_stain-LFB_chunk-02_SPIM.json
        ├── [2.7k]  sub-01_sample-A_stain-LFB_chunk-02_SPIM.ome.tif
        ├── [ 636]  sub-01_sample-A_stain-LFB_chunk-03_SPIM.json
        ├── [2.7k]  sub-01_sample-A_stain-LFB_chunk-03_SPIM.ome.tif
        ├── [ 635]  sub-01_sample-A_stain-LFB_chunk-04_SPIM.json
        ├── [2.7k]  sub-01_sample-A_stain-LFB_chunk-04_SPIM.ome.tif
        ├── [ 387]  sub-01_sample-B_photo.json
        ├── [   1]  sub-01_sample-B_photo.png
        ├── [ 635]  sub-01_sample-B_stain-LFB_chunk-01_SPIM.json
        ├── [2.7k]  sub-01_sample-B_stain-LFB_chunk-01_SPIM.ome.tif
        ├── [ 637]  sub-01_sample-B_stain-LFB_chunk-02_SPIM.json
        ├── [2.7k]  sub-01_sample-B_stain-LFB_chunk-02_SPIM.ome.tif
        ├── [ 635]  sub-01_sample-B_stain-LFB_chunk-03_SPIM.json
        ├── [2.7k]  sub-01_sample-B_stain-LFB_chunk-03_SPIM.ome.tif
        ├── [ 635]  sub-01_sample-B_stain-LFB_chunk-04_SPIM.json
        └── [2.7k]  sub-01_sample-B_stain-LFB_chunk-04_SPIM.ome.tif
balbasty commented 1 year ago

@ucbpclw

Do you mean which of metadata fields we sent in our example file are essential?

Yes, exactly

balbasty commented 6 months ago

Hi

I am reopening this thread. After further discussions with "synchrotron-CT" experts (@chourroutm, @dstansby, @ucbpclw), it appeared that hipCT might not be the most adequate suffix. The "H" means hierarchical and really relates to the acqusition of multiple images in the same sample. Each individual image is a (very fancy) phase-contrast CT.

They have suggested "PC" for phase-contrast, although in the current spec, "PC" relates to optical phase-contrast, not xray phase-contrast.

It might be good to fix this suffix before the next releast of the spec.

@satra @ayendiki @effigies @yarikoptic

effigies commented 6 months ago

XPC - X-ray Phase Contrast?

ayendiki commented 6 months ago

Adding the X does make it sound cooler.


From: Chris Markiewicz @.> Sent: Friday, April 26, 2024 7:53 AM To: bids-standard/bids-specification @.> Cc: ayendiki @.>; Mention @.> Subject: Re: [bids-standard/bids-specification] Add support for HiP-CT (Issue #1646)

    External Email - Use Caution

XPC - X-ray Phase Contrast?

— Reply to this email directly, view it on GitHubhttps://secure-web.cisco.com/1RJ4VJxNzwdOx__6K5a5aSKoHEBWWcoRNu31bi88izNWrHHlE3pEACHrvA_Imz4Pkx1iLs-17wyg7mQvMFnJWhnLtvMqlO41J4qyb4pkww2b_4FTUR0H9rG14DE4L4JpbPXj26a7I6zAohXT7LwH_OB3GjIc9VCcTsX2c-s3GXOW2nIADNpeBGa3dEWYGrzkut9SxcD3TCXPTPzTZSqkQbRx723V9_nLpGsqNIU6WMmz7tFfqyVv_J4DH-QqDwwCBj3xTXzmNHbTRy4pg2BSCkL-Xiu7MJy5-RTOiUFYVZ8vbrBnyqH8EBAwOuGkC4R1sfupd-z0-utxIOivmv7eMpQ/https%3A%2F%2Fgithub.com%2Fbids-standard%2Fbids-specification%2Fissues%2F1646%23issuecomment-2079237706, or unsubscribehttps://secure-web.cisco.com/1oGFgqJFwUAtlKAwMqRdsiqqCI3yVq4UA5x61nHovlUGc0-Kg19k0ttBr4m13I2Ox1ck8HvWzRv-vopF23Ij0-Ne9WFr9hFYbDXzSPsyL_wExaY0_l9pyZubBJFltOqdTJp1V-hOn4wsL8xq_rWdaTMqoJr3Dh0hprk7CxcZ1CZNd9juHILZV_xFy8mDlxU_mU_PJDnwBSVOaKSlQOY947ZXM-860g0tO_4aw-m6-l6lKKiTan9p-3fhJ__llZGvpxIvNkNECNhDEeY_p0hlHAZ9sEJU2FSZaqWQt620w07GIbfP7T6wIOxM3Z7RVW9AWJpBy0_ymvfOdoixmwiOAzg/https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADU34VYLIELDK3ILVD5V3ETY7I52XAVCNFSM6AAAAAA6WEWN66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZZGIZTONZQGY. You are receiving this because you were mentioned.Message ID: @.***>

[ { @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/bids-standard/bids-specification/issues/1646#issuecomment-2079237706", "url": "https://github.com/bids-standard/bids-specification/issues/1646#issuecomment-2079237706", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.": "Organization", "name": "GitHub", "url": "https://github.com" } } ] The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline https://www.massgeneralbrigham.org/complianceline . Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.

dstansby commented 6 months ago

Agree that it's worth dropping the "Hi" of "HiP-CT" - as mentioned above stands for Hierarchical which is about how mutiple datasets/scans relate to each other (basically some are higher resolution VOIs of others).

XPC sounds good. Does the suffix need to indicate it's CT data though? I note there's already a microCT suffix in the spec. So how about something like xPCCT? Or just PCCT for phase contrast computed tomography?

effigies commented 6 months ago

XPC sounds good. Does the suffix need to indicate it's CT data though?

The only hard and fast rule is that it needs to be alphanumeric. Otherwise, if the affected community agrees that it's precise enough, and people outside the community don't see it conflicting with something familiar to them, then it's all good.

I note there's already a microCT suffix in the spec. So how about something like xPCCT? Or just PCCT for phase contrast computed tomography?

I don't love xPCCT, but XPC and PCCT seem equally fine to me. But I don't work with these data, so I'm happy to leave it to those who do.

dstansby commented 6 months ago

If there's no particular standard/rules for what has to be in the suffix, apart from understandable by the community, then maybe it's worth keeping this as hipCT?

chourroutm commented 6 months ago

@dstansby I'd rather go for a broader name, as we're the only group/consortium doing HiP-CT (at the moment) but the community using the same underlying technique --- with or without the hierarchical component, which indeed rather corresponds to a dataset rather than a sole volume --- is already quite developed .

The different names I am aware of are:

As there is no real consensus of the naming within the community, we will not please everyone, of course. It seems to me that XPC might be sufficient as a suffix to the microscopy keyword. Otherwise, to convey the 3D, maybe XPCT is the one to go.

chourroutm commented 6 months ago

What's the opinion of the others? @ucbpclw @balbasty @satra My former collaborators were also interested in such an addition, I can include them in the decision-making.

Then, what is the course of events? Should I draft a PR with one of the two suffices?

I suppose we can prepare in the following months (once the validator recognizes the suffix) a truncated version of the open brain dataset from the Human Organ Atlas: https://human-organ-atlas.esrf.eu/datasets/572252538

satra commented 6 months ago

@chourroutm - i would leave this to you and the other experts to decide. as long as it doesn't conflict, we should be ok with any choice of a new extension.

balbasty commented 6 months ago

@chourroutm I vote XPCT, but yes it'd be great to have a validation from other groups, so that it lives up to the test of time :)

Once it's ok'd, you (or I) should revert the changes from PR #1686 and replace all occurences of hipCT by XPCT.

effigies commented 6 months ago

It would be good to get an example in https://github.com/bids-standard/bids-examples as well, so that we can update and test the validator. See the contributing guide.

ucbpclw commented 6 months ago

I would vote XPCT as well, it is the wording we are generally using when writing grants as it captures all the essential components of the technique and is specific enough to differentiate it from other X-ray or tomography examples.

chourroutm commented 6 months ago

@gobej and @DoctorEmmetBrown do you want to weigh in?

chourroutm commented 6 months ago

@effigies I've tailored an example dataset in chourroutm/bids-xpct-example/tree/master/micr_XPCTzarr that we'll amend before I draft a PR to bids-standard/bids-examples.

@ucbpclw @dstansby and eventually @balbasty can I have your inputs on the dataset? This is the file structure of the example dataset:

micr_XPCTzarr
|   dataset_description.json
|   participants.json
|   participants.tsv
|   README.md
|   samples.json
|   samples.tsv
|   
+---code
|       data_retrieval.py
|       data_retrieval_reqs.txt
|       metadata_formatter.py
|       metadata_formatter_reqs.txt
|       truncate_data.sh
|       
\---sub-LADAF-2020-31
    |   sub-LADAF-2020-31_sessions.json
    |   sub-LADAF-2020-31_sessions.tsv
    |   
    \---ses-01
        \---micr
            |   sub-LADAF-2020-31_ses-01_sample-brain_photo.jpg
            |   sub-LADAF-2020-31_ses-01_sample-brain_photo.json
            |   sub-LADAF-2020-31_ses-01_sample-brain_XPCT.json
            |   
            \---sub-LADAF-2020-31_ses-01_sample-brain_XPCT.ome.zarr
                |   .zattrs
                |   .zgroup
                |   
                +---0
                |   |   .zarray
                |   [...]
                [...]
                |
                +---4
                    |        
                    [...]
                    |           
                    \---9
                        +---0
                        |       0
                        |       1
                        |       2
                        |       
                        +---1
                        |       0
                        |       1
                        |       2
                        |       
                        \---2
                                0
                                1
                                2
dstansby commented 6 months ago

Layout looks good to me (FWIW I sucessfully adapted a HiP-CT dataset to BIDS to upload to LINC), but I think that's beyond the scope of this issue (which is just about the prefix)?

effigies commented 5 months ago

1822 changing hipCT to XPCT has been open a week with no objections. I would ask someone else to leave an approving review and encourage anybody who would object to do so as soon as possible.

fabienchauveau commented 5 months ago

I am also in favour of "XPCT".

effigies commented 5 months ago

XPCT adopted in #1822.