search

ME-ICA / tedana

TE-dependent analysis of multi-echo fMRI
https://tedana.readthedocs.io
GNU Lesser General Public License v2.1
161 stars 95 forks source link

OHBM 2021 Preparations #722

Closed handwerkerd closed 3 years ago

handwerkerd commented 3 years ago

Summary

This is a place to keep track of general OHBM + tedana/multi-echo related preparations

Additional Detail

OHBM posters and videos need to be uploaded by May 25th. That means we should have a good sense of what will be included by mid May. A poster draft that allows time for feedback should be done at least a few days before our May 21 developers' call. The poster can include a 2-3 minute video. If we are ambitious, perhaps we can include multiple speakers on the video. @handwerkerd has already volunteered to take the lead on preparing the poster, but welcomes help.

Open Science Room Education Sessions or Panels need to be submitted by May 15th ( https://ohbm.github.io/osr2021/submit/ ). That means this would need to be completed before our next dev call. Anyone want to take the lead on this. Perhaps @smoia wants to lead this effort again? Otherwise, I suspect he'd be willing to advise someone else who takes the lead.

For the past several years, we haven't had anything tedana specific at the OHBM hackathon. That may again be the case this year, but I wanted to make space for that discussion.

Next Steps

handwerkerd commented 3 years ago

I'll add, partially in response to @smoia's comment on gitter: I view the goal of the OSSIG presentation as a way to get a bunch of people using or interested in ME in one place both to let others know they're not alone, potentially facilitate a few collaborations, and to let the broader community realize that there is general interest in the topic.

That said, every year we've had a different focus for the meeting and I think that variation is important.

One brainstorming thought for this year would be to focus on multi-echo data collection & analysis in practice. It seems the user based has crossed enough of a threshold to make this an interesting focus. We could spend have the meeting having a few pre-selected people give mini talks on their experiences and then have space for more open discussion & questions during the second half. That focus would benefit from having a non-core tedana developer as an organizer and/or speaker. I’d encourage @62442katieb (based on: https://doi.org/10.1101/2021.04.01.437293 ) to be a speaker & possibly a co-organizer, but I’m not sure how many other things she’s already volunteered for with the OSR.

smoia commented 3 years ago

HI @/all! Apologies for the late reply. I wouldn't have any problem in taking the (co-)lead again on a ME panel. It's fun, it's a nice hour, there's nice people, and maybe a good convo pops up.

This said, honestly I'm not sure it makes much sense that: a. we make a panel similar to previous years and b. I (co-)lead it.

Let me explain my point starting from (b): I'm not an active developer nor following development that much any more (in fact, if I'm doing something is to slow you down with the get_coeff development). So if one of the active developers wants to do it, I'd say it would be more appropriate! It's a fun experience, and if you want some suggestions I can help with that. But, if nobody wants to put their face, I have absolutely no problem in doing it again.

However. Last year having a panel made sense, because we also had the Symposium and it was a nice combo. I'm not sure what the plan would be this year to justify the panel. Sure, it's a good moment to see the faces and names of people working with ME or around it, but unless there is an active goal we want to reach, it might feel a bit like a lost opportunity. Unless the goal is "updates on what's happening in the ME BOLD world, which can be fun and a good platform for researchers to share their latest works.

This said, if there is willingness to do something I can help as a moderator. So let me ask you: is there a specific goal you want to achieve with this OSSIG meeting? Perhaps increasing the user base? Or recruiting new developers? Or starting an action? Or none of the above?

handwerkerd commented 3 years ago

@smoia and I decided to talk about the OSR presentation this Monday, 5/10, at 12:30EST, 6:30CEST. If others, want to join, let me know & I'll send you the meeting link.

handwerkerd commented 3 years ago

@smoia and I spoke on Monday to discuss an OSR event. We decided that we should propose an emergent session rather than an educational talk or panel. That is similar to what we've done in past years and probably still best fits our goals. That means there isn't a Friday submission deadline, but sooner is better. The submission format is fairly simple and we don't actually need a list of presenters yet so there's a lot of flexibility. Our thought for this year is to focus on multi-echo in practice

The guide: https://ohbm.github.io/osr2021/submit/ Information we need: The name of one submitter + pronouns, affiliation, & email address

Preferred time zone (opinions on ETD vs UTC?)

Abstract (there’s one line, but it seems like they’re just asking for a few sentences. Note that I’m consciously not mentioning tedana in the abstract, but the related URL likely will be tedana-related): “Multi-echo fMRI in practice”: Multi-echo fMRI is an acquisition method where data is acquired at multiple echo times after each excitation pulse. This information can be used to reduce signal dropout and quantitatively identify and remove signal that is unlikely to be neural in origin. Multi-echo fMRI acquisitions and processing methods have sufficiently matured to the point where the growing developer and user community would benefit from a discussion of what it’s like to use multi-echo fMRI in practice. In this emergent session, presenters will briefly answer three questions: (1) Why are you using multi-echo fMRI? (2) Are there any challenges of multi-echo fMRI that you’ve addressed and want to highlight? (3) What issue(s) with multi-echo fMRI would you like to see addressed? These presentations will be followed with time for discussion between presenters and attendees.

Related URLS: Either tedana.readthedocs.io/ or open an issue that’s specific for this emergent session. I’m leaning towards opening an issue so that we have a place where we can list the presenters and maintain some discussion.

Please comment or give a 👍, if you're fine with this. @emdupre @dowdlelt @tsalo @jbteves @eurunuela @javiergcas @notZaki

jbteves commented 3 years ago

@handwerkerd I would prefer we make a new issue specifically and link it, which is something we've typically done for the OHBM posters anyway, and this feels similar. Preference is ETD but I'm biased 🙂 PS for future reference you can tag at ME-ICA/tedana-devs to get all contributors with commit access.

CesarCaballeroGaudes commented 3 years ago

I think the format you are proposing as an OSR event is perfect, and happy to collaborate in being a presenter or attendee. I could also provide some points regarding questions (1), (2) and (3) from my perspective. Regarding the preferred time zone, I actually prefer whatever time is later than 9 pm or 10 pm in Spain (3pm EST).

handwerkerd commented 3 years ago

Random thought. We have some OHBM-specific repos: https://github.com/ME-ICA/tedana-ohbm-2020 & https://github.com/ME-ICA/tedana-ohbm-2019 Particularly for 2020, that repo ended up being a central place to list all multi-echo fMRI content at OHBM. What do people think about leaning into that concept this year & use the name https://github.com/ME-ICA/ohbm-2021-multiecho That could be the link for the emergent session and we'd have some flexibility to adjust format as OHBM approaches rather than pointing to an issue where the format is a bit more fixed.

handwerkerd commented 3 years ago

Also, it's clear they want just one name on the application. It seems more like a point-of-contact rather than a sole presenter. Given what I'm doing now, I'm assuming I'll use my name unless someone else volunteers. We'll be able to list anyone involved with setting this up on the page for the emergent session. OK?

handwerkerd commented 3 years ago

I just tried to create https://github.com/ME-ICA/ohbm-2021-multiecho but I don't have the authority. @emdupre and Prantik are the current owners. I think non-owners can be allowed to create new repos. Elizabeth, Can you give me that privilege or just create a repo with a name like that? If Elizabeth is busy, anyone else have the authority to create a new repo?

eurunuela commented 3 years ago

I just tried to create https://github.com/ME-ICA/ohbm-2021-multiecho but I don't have the authority. @emdupre and Prantik are the current owners. I think non-owners can be allowed to create new repos. Elizabeth, Can you give me that privilege or just create a repo with a name like that? If Elizabeth is busy, anyone else have the authority to create a new repo?

Just created https://github.com/ME-ICA/ohbm-2021-multiecho 😉

handwerkerd commented 3 years ago

@eurunuela, The way it's currently set up, I can't directly push edits and (possibly because there's nothing there) I can't fork or create a pull request. At least for now, can you give me authority to push into that repo without reviews? Maybe make me an owner for that repo? Thank you for setting this up!

eurunuela commented 3 years ago

@eurunuela, The way it's currently set up, I can't directly push edits and (possibly because there's nothing there) I can't fork or create a pull request. At least for now, can you give me authority to push into that repo without reviews? Maybe make me an owner for that repo? Thank you for setting this up!

Sorry about that. I thought it would let you. Try now.

handwerkerd commented 3 years ago

@eurunuela Thank you! It works now and https://github.com/ME-ICA/ohbm-2021-multiecho is live with content. Feedback on the text is welcome. I'll plan to submit the proposal later today and we'll be able to add to that site as appropriate.

handwerkerd commented 3 years ago

I just submitted the proposal for an emergent session using the text above (with some slight grammar corrections)

62442katieb commented 3 years ago

Hi @handwerkerd and @smoia ! I'm a bit late to the party (sorry to have missed the discussion). I'm happy to contribute to an emergent session in the OSR, but I'm a tad over-extended during OHBM and my defense is June 24 (🥳 / 😅) so my contribution might be limited.

handwerkerd commented 3 years ago

@62442katieb The way we proposed the emergent session will allow for some flexibility depending on availabilty. I'll check in as it gets closer. While the run-up to a defense is unlikely to be fun, I hope you enjoy the actual defense. Congratulations!

handwerkerd commented 3 years ago

I have a very rough draft of the poster in this shared folder The poster is titled "tedana_poster_OHBM2021.pdf" and I'll keep updating that file as I improve it. The hard submission deadline is Monday, 6/14, so it's past time for me to get some feedback. It's not in a state where I want to email it to all the authors, but I'd appreciate if a subset of @emdupre (featured in the governance section) @tsalo @jbteves @eurunuela @notZaki @smoia & @dowdlelt glanced at it soonish.

Things I know still need to be done or places where feedback would be helpful:

eurunuela commented 3 years ago

Quick comment from my phone (I still haven't checked the poster) Re dynamic reports:

I'd either use the same page we created last year, or now that @notZaki has run tedana on the 88 Cambridge datasets, it would be really cool to share the reports and results of all those subjects (or at least some of them). I think it is a very good "real-life" scenario to showcase tedana's capabilities. We could even think of automating this process, along with the process of generating figures for the posters for the future too.

I'll get back to this issue later today and give feedback on the rest.

eurunuela commented 3 years ago

As I said before, it would be great to share the results on the 88 Cambridge datasets.

  • Add more pictures??? The poster has a bit too much text for my preference, but I can't think of any obvious additional figures.

I agree but I think that making the flowchart a bit more like a figure would change my opinion.

  • Are people ok with my use of Monty Python Holy Grail images on the poster (I had an idea, but I'm not sure I executed it well)

I find it a bit weird haha but I'd say it's @emdupre who should say whether she's comfortable with it or not.

  • The governance changes are big part of the poster this year. Did I summarize well?

I think it's well summarized.

  • I haven't given much thought to the video recording yet. I'd like to do closed captioning well. Last year I just published the talk and separate transcript. In the past year, has anyone gained experience on how to caption correctly & has the time on Monday to help with this?

AFAIK PowerPoint already captions as you speak and lets you record too.

handwerkerd commented 3 years ago

@notZaki & @eurunuela I know about https://osf.io/9wcb8/ but are the tedana reports for this preprocessing somewhere online? Sorry if I got out-of-the-loop on this dataset.

jbteves commented 3 years ago

@handwerkerd thanks for putting this together! A few things:

  1. I'm not sure that we have improved AFNI integration. Has anyone tried it with a completely up to date version? Particularly since we've changed the dependencies and output names?
  2. I agree, there's too much text, but I'm not sure what we could include figures-wise that we don't already have.
  3. The metric modularization explanation is too verbose in my opinion. I would modify it to something like the following:

    Calculation of metrics used for the component selection process fully refactored and modularized. Calculations are much easier to understand and it is now significantly easier to add or modify metrics. The process of modularization led to the identification and correction of several bugs.

  4. I think you should color and bold the links near the bottom, as right now they're not too distinct from the nearby text. I think this is particularly troublesome since I expect many people will be viewing this on small laptop screens.
  5. I think in "Governance," we should make a few more modifications. For example,

    In reality, most decisions were made by concensus, but we didn’t have a consistent way to fascilitate concensus. This caused unnecessary time stresses on our BDFL and made growing the project more difficult

could be

Most decisions were made by consensus, but the BDFL does not offer a consistent model for facilitating consensus. Therefore,

and then change

tedana got too big for an informal governance model, but is still and will probably remain to small for some of the highly structured governance models.

tedana has transitioned to a committee-based structure to facilitate consensus building and improve both software, which you can read about here

and then link here.

  1. I'm not sure that we should discuss the governance changes there, as the details can be easily linked in a way that some of those code change summaries can't be. I'm also not sure that anyone will be particularly interested who won't just want the full description in our documentation. I could be wrong, though. If we cut it, we might have more room to make the report bigger, which I think would be good.
notZaki commented 3 years ago

I didn't upload the tedana output of the cambridge datasets, but that could be done. Not sure if osf will allow the reports to be viewable in the browser though, so might have to consider a different hosting option.

eurunuela commented 3 years ago

I didn't upload the tedana output of the cambridge datasets, but that could be done. Not sure if osf will allow the reports to be viewable in the browser though, so might have to consider a different hosting option.

You're right. I believe osf will just show the html code instead of rendering it.

Maybe the reports themselves could go into the OHBM 2021 repo. We could have a folder per dataset with the report and a README with the link to OSF? Only if it can be automatized, of course.

handwerkerd commented 3 years ago

If OSF can host the HTML files, I think that's the most logical location. Anyone who wants to interact with them can download the relevant files. I can then copy one to the OHBM 2021 repo so that people can interact without downloading. @notZaki, Any chance you can get some or all up today or tomorrow?

I'm now working on integrating many of the other comments.

handwerkerd commented 3 years ago

@jbteves I'm working on cleaning up the wording and formatting, but, as long as we have the space, I think there's a benefit to keeping a decent amount of our governance goals and structures on the poster. We looked at other governance models, realized there wasn't a cut-and-paste model we could use for a project of our size, and adapted other models for our project. We now have something that's slightly unique that might be relevant to other projects, and might attract broader interest and discussion than people solely curious about multi-echo. I want enough info on the poster to potentially prompt discussions.

handwerkerd commented 3 years ago

New poster version uploaded. Still work to do, particularly cleaning up formatting more and improving the interactive visualization example (I may change to a Cambridge 88 dataset to highlight the fact that those data are now easily available). I think I've address most of the previous text-focused critiques, though this is still a text-heavy poster.[

With help from @jbteves I also added the following factoid: "In the past year we have changed 116 files with 5958 inserted and 3260 deleted lines of code out of ~ 15000 total lines of code!" (this includes metric modularization)

The poster is still here.

notZaki commented 3 years ago

@handwerkerd I uploaded the reports to a this github repo, and the reports can be accessed from this page.

I haven't gone through all the reports. It's possible that some subjects have nicer results than others.

emdupre commented 3 years ago

Thanks for sharing the poster, @handwerkerd , and for all of your work on it !! A few thoughts :

Related to the last point, I don't know if the author list is finalized, but I'd add a few more folks like @jsheunis if we can !

handwerkerd commented 3 years ago

@emdupre You should still be able to right click on the poster to see version 2 with the Monty Python pictures. I was still thinking of adding a crown onto your picture. 😄

I won't be able to add much more during the next 24h, but keep giving feedback and I'll try to integrate everything and clean things up over the weekend.

handwerkerd commented 3 years ago

@emdupre I vaguely remember talking to @jsheunis at submission time, but I can't find any correspondence, so perhaps this was an embarrassing oversight. If he doesn't respond here, I'll include him in the email to all co-authors and ask if he wants to be added. Anyone else who we should consider adding?

handwerkerd commented 3 years ago

@notZaki Some of those reports are concerning. 😦 It's a great sign that the reports can let us quickly see what's happening, but this also points to more work to be done on the component selection process. For example sub-04710 and sub-11310 have very high variance linear drift components being accepted. sub-04800 has almost all signal variance rejected.

On one hand, I think it’s good to show how these reports can highlight problems worth further examination. I could present one good result and one confusing result. On the other hand, I haven’t really dug into this dataset or data underlying these reports. If I’m the main presenter, I’m uncomfortable answering questions about data/results that I don’t yet understand.

At this point, I’m leaning against using a report from a dataset I personally don’t have a good handle on yet. I can more prominently highlight that these data & reports exist, but not use them in a figure.

Thoughts?

Also, there were a few bug fixes in April/May and a couple more tweaks planning post modularization. Since the Cambridge 88 data are 4 echoes and one of the fixes affected >3 echo data sets, it might be worth seeing if the newer code cleaned up the results. When did you run tedana on these data?

notZaki commented 3 years ago

@handwerkerd I would go with whatever you feel comfortable presenting.

The reports were generated using the the modularization branch (PR 591).

As you said, the reports indicate that more work needs to be done. If these reports were to be mentioned, then I suppose one way to frame these results is to say that we know that sometimes the classification is not perfect, and the two major ways this is being addressed is by: (1) on the user-end: improvements to the interactive reports to allow user to examine the results and determine if anything went wrong, and (2) on the developer-end: refactor the metric calculation & classification code so that it is easier to add/modify/debug/refine the classification.

jsheunis commented 3 years ago

@handwerkerd @emdupre

If you feel my contribution is enough to be included, I'd happily join in. I can read through the poster + abstract in the next few days, unless you're already done with edits?

handwerkerd commented 3 years ago

@jsheunis I'm happy to add you. Could you point me to a logo for your affiliation that I can add to the poster? I'm hoping to get an almost final version of the poster out in the next few hours. Feedback is appreciated. The submission deadline is around 30 hours from when I write this comment.

jsheunis commented 3 years ago

My current main affiliation is: Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Center Jülich, Germany.

Here's a link to an SVG of the logo: https://raw.githubusercontent.com/datalad-handbook/course/master/pics/fzj_logo.svg

I'm not at my laptop now unfortunately, so won't be able to give effective feedback. But i can do so around 9hours from now. Please let me know if that will still be in your acceptible time frame. If not, I think trusting all existing input is something that I'm totally okay with.

On Sun, 13 Jun 2021, 22:58 Dan Handwerker, @.***> wrote:

@jsheunis https://github.com/jsheunis I'm happy to add you. Could you point me to a logo for your affiliation that I can add to the poster? I'm hoping to get an almost final version of the poster out in the next few hours. Feedback is appreciated. The submission deadline is around 30 hours from when I write this comment.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ME-ICA/tedana/issues/722#issuecomment-860269213, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNL4NPJKS3WDDRQEMNUYO3TSULWVANCNFSM43JG2ZLA .

emdupre commented 3 years ago

Anyone else who we should consider adding?

I know we had conversations at the time of abstract submission and again at the JOSS paper drafting, so I'm a bit fuzzy on what was discussed when ! Sorry about that.

I know it's late to get substantive feedback at this point, but I'd include the JOSS co-authors on the email ! Since a lot of the content will link out to interactive content, we can keep incorporating feedback there, too.

Thanks for the confirmation, @jsheunis ! Will be great to have you onboard, and any comments you can add are always appreciated :)

handwerkerd commented 3 years ago

Thank you @emdupre ! I knew I was forgetting to put something on the poster: The link to the submitted JOSS manuscript! For the sake of the poster, this is probably the cleanest link? https://github.com/ME-ICA/tedana/blob/joss/paper/paper.md Should I directly say JOSS is short-staffed on reviewers and we'd love to have someone volunteer? Joking/Not Joking

As for additional authors, the people on the JOSS manuscript, but not the poster are @effigies @Islast and @prantikk. If any of you want to be authors on this poster, please let me know ASAP. The poster upload deadline is around 24h from when I'm posting this comment.

jsheunis commented 3 years ago

I have no additions/changes to add to the poster, which looks great. I particularly like the summary of the governance structure :) Well done!

effigies commented 3 years ago

IIRC my only contributions have been related to dependency resolution, which feels pretty unrelated to the content of this poster. If you would like me to be on the poster, I'm willing (affiliation: Stanford), but I don't feel a need for credit here.

Poster looks good, and I also like the focus on governance.

handwerkerd commented 3 years ago

@effigies In that case, I won't add you to the poster, but you're, obviously on the JOSS manuscript.

dowdlelt commented 3 years ago

Sorry that I am a bit late to this party but I am a fan of the poster. It highlights a lot of neat things - and I think the call outs to specific parts of the code are quite nice. Hyperlinks are also great. No complaints here, thanks for putting in all the hard work @handwerkerd and folks.

handwerkerd commented 3 years ago

I also just updated https://github.com/ME-ICA/ohbm-2021-multiecho with other OHBM multi-echo posters and the interactive reports are now live at: https://me-ica.github.io/ohbm-2021-multiecho/ Feedback is welcome

Of note, @emdupre identified posters by looking for 'multi-echo' or 'multiecho' in titles. We haven't yet done a search for posters with 'multi-echo' in the tags or abstract bodies. if someone feels like compiling them before I get a chance, post the info here: https://github.com/ME-ICA/ohbm-2021-multiecho/issues/2

dowdlelt commented 3 years ago

Having the reports is great - the 5-echo one looks to be working fine, but with the 3-echo report I'm getting no component maps (little broken image icon shows, no beautiful maps).

handwerkerd commented 3 years ago

Good catch @dowdlelt! It works better when I include the figures in the repo. It should be fixed now.

notZaki commented 3 years ago

Just a heads up, those reports don't match the CI which is likely due to differences in hardware/software (unless I'm missing something):

handwerkerd commented 3 years ago

I think the CI also runs on a fraction of the time series to save time. That said, I just realized I ran without the metric modularization code. Now that it's officially merged, I'll run again so that my reports will match what is now Main.

handwerkerd commented 3 years ago

What should be the final version of the poster is still at https://drive.google.com/drive/folders/14N7v_AMPcMyzwKHn6J8CUJV2PMpQR4S5?usp=sharing 'tedana_poser_OHBM2021_WithLinks.pdf' has added links to website & is the version I'll upload. I've also uploaded a version of the video which is a bit too long, but should easily be shorter with a few more attempts. Now I just have to wait until a major thunderstorm passes before re-recording the video & uploading (and hoping I don't lose power in the next hour)

handwerkerd commented 3 years ago

Poster and presentation are uploaded. Thank you all for your feedback and help!

handwerkerd commented 3 years ago

I went through the abstract texts and I'm fairly sure I got all the posters that mention multi-echo fMRI in the title or text. I've updated https://github.com/ME-ICA/ohbm-2021-multiecho/ with all those posters and the links. I haven't figured out a good way to see if any talks mention multi-echo in their titles or abstracts. I'll plan to tweet out that link Sunday night or Monday morning.

Also, for the people who might be slightly more connected to the OSR than I am (@emdupre @62442katieb @jsheunis @tsalo ) any clue when they're announcing a schedule for emergent talks? If I'm leading a session, it would be nice to have more than 24h to gather the speakers and to make sure I can be available!

handwerkerd commented 3 years ago

@ME-ICA/tedana-devs I've tentatively scheduled the multi-echo emergent session for Thursday, June 24, at 9:00AM EDT. I'm hoping that is a good time for many of you in multiple time zone. The emergent session schedule still looks fairly open so we may be able to shift.

Also, there are 3 presentation times for the tedana poster: Tuesday, 6/22, 8:00-9:00PM EDT Thursday, 6/24, 4:00-5:00AM EDT Friday, 6/25, 12:00-1:00PM EDT I should be able to be present for the Tuesday & Friday times, but others are very welcome to join. I do not plan to attend the Thursday time, but maybe @smoia or @eurunuela can be present for some or all of that time.