galaxyproject / training-material

A collection of Galaxy-related training material
https://training.galaxyproject.org
MIT License
294 stars 846 forks source link

New tutorial: "Phylogenetics: back to basics" #4611

Closed Adamtaranto closed 1 week ago

Adamtaranto commented 4 months ago

Hello GTN team,

We’re submitting a new tutorial “Phylogenetics: back to basics” for review and hope that you’ll find it suitable for the GTN. It’s based on a live workshop run by Australian BioCommons and Professor Michael Charleston from the University of Tasmania.

This workshop was very popular and we’re keen to share it via the GTN so that more people can take advantage of it.

This tutorial takes a slightly different approach to other GTN tutorials in that it’s not a “how-to guide”. Instead it is aimed at giving a better understanding of the principles of phylogenetics and how the methods work. It uses tools that are (mostly) available in Galaxy to demonstrate these principles. We think it makes a great companion to other evolution-themed GTN tutorials.

A couple of things we need your help with:

We’ve submitted the data to the GTN Zenodo community and are waiting on approval before we can add a link. We’d like to centre align the videos but this requires adding some new settings for the whole website. Can you help us with this?

Given that it’s close to the Christmas break we understand that reviews will take longer than usual. We’re taking a break too and look forward to hearing your feedback in the new year!

You can contact us through GitHub or by email.

Many thanks,

Michael Charleston (michael.charleston@utas.edu.au) Adam Taranto (adam.taranto@unimelb.edu.au) Melissa Burke (melissa@biocommons.org.au)

hexylena commented 4 months ago

We’ve submitted the data to the GTN Zenodo community and are waiting on approval before we can add a link.

you should be able to add the link now, the approval is only for it to show up in the GTN page in zenodo, the record should still be accessible even if it isn't yet in the Gtn zenodo community.

We’d like to centre align the videos but this requires adding some new settings for the whole website. Can you help us with this?

this is possible yes

hexylena commented 4 months ago

Hi @Adamtaranto, welcome to the GTN! (A longer response now that I have my laptop available.)

it’s not a “how-to guide”. Instead it is aimed at giving a better understanding of the principles of phylogenetics and how the methods work

In general this looks great, it doesn't have to be as hands on as other tutorials, we're happy to accept excellent learning resources like these. They're achieving similar didactic purposes through different methods, that's fantastic. This looks like an excellent resource and we'll be very glad to have it on the GTN (maybe it's an era of more aussie biocommons contributions? which would be very exciting for us!)

The videos are the main point of discussion from our side. We heavily discourage the use of videos, you'll note they aren't present in essentially any other tutorials with few exceptions. We do this because the most accessible video is still less accessible than a well marked up HTML page.

I've taken a look at the videos you've linked in the tutorial, it's great to see you've got manual captions on them, that is a baseline requirement for us. That said, they're not maybe what one would refer to as high production slides? there's little to no animation, it's mostly talking through a slide deck (which is great and useful as-is!!!) So, given the GTN's position on slide deck accessibility and that aspect of the existing videos, I'd like to propose a more accessible alternative where you can retain the videos, and we can offer something that will work for all learners, rather than just the sighted ones:

If you would be willing to turn the slide decks from those videos, into GTN slide decks, then from our side we'll contribute a widget/box that you can use within the tutorial, that will give learners the option to choose their preferred in which to consume the tutorial; a video, the gtn remark.js slide deck, or the "plain text" slides which don't require javascript, and help you link to the correct places in each of those resources. (I wish I could offer here an automated method for doing that conversion since you have a lot of these videos, but we've only just written that into our next grant and don't if that'll be funded or when. I have a proof of concept python script that really only works with google slides currently.)

Would this be an option for y'all? Because we'd really love to have this tutorial in for everyone to access, and this one requirement would help this resource meet our accessibility commitments.

Adamtaranto commented 4 months ago

Tagging in @burkemlou @mcharleston for comments.

My thoughts on videos and accessibility: This tutorial is designed to work without the videos, we've included them as something extra for self-guided learners to be able to access a taste of the in-person workshop experience.

Do you have a tutorial on making GTN formatted slides?

burkemlou commented 4 months ago

Hi @hexylena,

Thanks for your feedback, it's really useful as always.

Echoing Adam's comments, these videos were created in the spirit of those done for the Smorgasbord events and to do exactly what you're suggesting - provide flexibility in the way that people can interact with the tutorial.

The ideas you suggested for improving accessibility sound good to me. We're always happy to work with you on this!

That said, we're all on leave for at least couple of weeks over Christmas/New Year. I'll get in touch once I'm back in the office (after the 11th January) and we can figure out the best way forward. In the meantime, if you have tutorials or guidelines you can point us towards that would be really helpful.

Thanks again and wishing you a restful end to 2023 and a happy 2024.

Melissa

tagging @Adamtaranto and @mcharleston

burkemlou commented 4 months ago

I forgot to add, thanks for approving the Zenodo record! We'll add that in next year too.

hexylena commented 4 months ago

something extra for self-guided learners to be able to access a taste of the in-person workshop experience

yes, and we love this! that's so great to offer that! from our side, I just want to make sure everyone can access that resource because I think there's a lot of good information in those talks, that might not be available otherwise, and in any case it'll make excellent review materials for a motivated and dedicated learner.

Do you have a tutorial on making GTN formatted slides?

We have some materials, maybe it's time we improved them. There's a slide deck on how to format the slides since they use slightly different markup, but there's no specific tutorial.

The header metadata for a slide deck is essentially the same as a tutorial. I'll work on getting that documented in a better way, we have such a nice tutorial for contributing tutorials, but not for slides yet.

Since y'all're still on break, hopefully I can get some materials together in the meantime!

burkemlou commented 4 months ago

Hi @hexylena, happy new year!

Any chance we could have a quick chat about this next week?

I'm free Wednesday or Thursday morning before 11am Europe time.

hexylena commented 4 months ago

Yes, that's definitely possible. Let's try for wednesday? Could you send a calendar invite? :) 9-10 CET should be doable.

burkemlou commented 4 months ago

Great, thanks. I've just sent an invite and a Zoom link to your Gmail account.

Talk to you tomorrow.

On Mon, 15 Jan 2024 at 19:39, Helena @.***> wrote:

Yes, that's definitely possible. Let's try for wednesday? Could you send a calendar invite? :) 9-10 CET should be doable.

— Reply to this email directly, view it on GitHub https://github.com/galaxyproject/training-material/pull/4611#issuecomment-1891703829, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQNUHBFLUBEVB23XGMBFI4TYOT2M3AVCNFSM6AAAAABA7DHBPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJRG4YDGOBSHE . You are receiving this because you were mentioned.Message ID: @.***>

hexylena commented 4 months ago

From the chat with @burkemlou it's all looking quite positive!

I learned the slides were in beamer format, this is fantastic news, you can then use pandoc to convert them probably pretty easily. I suspect the command will be something like pandoc -f beamer -t reveal.js < slides.tex > slides.html (which will then need some reformatting but I will test that and make a further reply here.)

The biocommons logo can be added to each slide deck, please set the logo field in the metadata header with a path (in-repository) to the image

Since there are fiveish slide decks I would recommend the following: creating a folder for each, with a single slides.html inside.

As I find time I'll start working on a nice widget to embed these and give end-users a choice of their content format, just leave the youtube links as-is until then :)

hexylena commented 4 months ago

Edit, I stand corrected about the pandoc conversion, pandoc can only output beamer, it can't use it as input, it can't use any of the slide formats as input sadly.

Looking around, others had similar issues https://rmarkdown.rstudio.com/articles_beamer.html https://stackoverflow.com/questions/39453252/pandoc-to-move-knitr-beamer-slides-to-rmarkdown

So maybe something to try there is just treating it as a plain latex file

12:48:49 w-galaxy:[~/arbeit/galaxy/training-material @pr-4630‽]$ pandoc -f latex -t markdown < /rw/home/user/arbeit/presentations/2019-GCC/gxadmin.tex
# About

::: frame
![image](galaxy-eu.512.png){width="90%"}  \

::: center
Community \| Users \| Training \| Infrastructure
:::
:::

::: frame
About Me

-   Joined team 16 months ago

-   UseGalaxy.eu System Administrator

-   de.NBI Cloud Freiburg Administrator\
    (3.5k cores, 15TB ram)

-   Developed Galaxy Circos/JBrowse tools, RStudio GIE
:::

# Community

::: frame
Community

::: center
Community is the most important part of what we do
:::
:::

This is, maybe not sooo painful to convert further, then you'd just need to replace all ::: frame with --- to mark the slide transition, and delete the corresponding :::, and it'd be done?

mcharleston commented 3 months ago

Hi Helene, thanks so much for your kind and useful comments! I was also hoping pandoc would do it but that's ok, I will see how much of a faff it is to convert from my LaTeX to the markdown. :)

Adamtaranto commented 3 months ago

Hi @hexylena, do you know why the CI checks are failing on the latest commit?

Also, should I squash the history down into a single commit before it gets merged?

hexylena commented 3 months ago

@mcharleston hope pandoc gets you close at least :)

@Adamtaranto I will comment directly on the page for the linting errors but they're easy fixes :)

Also, should I squash the history down into a single commit before it gets merged?

no, please don't do that (or rather, there is no reason from our side to. it's nice to have the history.)

burkemlou commented 3 months ago

Thanks Helena. I'll start working through your suggestions and get back to you.

burkemlou commented 2 months ago

Hi @hexylena we're looking into the process of converting slides to markdown and it's looking like it will be a manual process. Being able to preview the slides (like you can for the tutorial itself) as we rebuild them would be very helpful in this process.

Is there a way to do this within GitPod?

Thanks!

hexylena commented 2 months ago

Hey @burkemlou, yes, absolutely that can be done in GitPod! It should be done automatically, and it sounds like you're doing that already for the tutorials, could you share what issues are you encountering with the slides?

burkemlou commented 2 months ago

Thanks @hexylena the issue (at the moment at least) is that we didn't know it was possible. Let us try it out and we'll get back to you if we need more help.

burkemlou commented 2 months ago

Hi @hexylena I've replaced formulas that were images with math markup which is working great. But I have the peculiar problem that the formulas show up in the preview of the details box. I'm sure I'm missing some tag somewhere but I can't figure it out. Any suggestions?

It's at ~ line 624 in the tutorial.md file

Screenshot 2024-02-26 at 1 07 51 pm
burkemlou commented 2 months ago

Hi @hexylena I've replaced formulas that were images with math markup which is working great. But I have the peculiar problem that the formulas show up in the preview of the details box. I'm sure I'm missing some tag somewhere but I can't figure it out. Any suggestions?

It's at ~ line 624 in the tutorial.md file Screenshot 2024-02-26 at 1 07 51 pm

I have fixed this so that the formulas are now contained within the details box. It was something to do with line breaks. I'd like to centre align them but can't get it to work. Left aligned will do!

burkemlou commented 2 months ago

Hi @hexylena,

Quick update from our side. I think we have now worked through and addressed your comments above. Please let us know if we've missed something.

We have figured out how to convert Michael's slides into the GTN format and have added the first one into the GTN to practice on. We're working on formatting of this slide correctly and converting the remaining slides over.

Michael would like to use this tutorial in teaching from mid-April. If we're able to convert slides over by the end of March do you think we could have it published before then?

Thanks again for your input and help troubleshooting.

Melissa

Tagging @PatCapon39 and @mcharleston for comments on the slide conversion process.

hexylena commented 2 months ago

Michael would like to use this tutorial in teaching from mid-April. If we're able to convert slides over by the end of March do you think we could have it published before then?

yes absolutely!

Thank you all for your hard work on this, I cannot wait to announce it in our channels.

hexylena commented 2 months ago

@burkemlou since y'all're committed to including the slides, I'd also be open to merging this sooner. Since the tutorial itself looks ready, we could ship that now as is, and as the slides come in we'll update it accordingly to also link to the alternative versions of that.

Would you like to also write a short news post announcing the new tutorial? á la something like https://training.galaxyproject.org/training-material/news/2023/01/26/pathogen-detection-tutorial.html

hexylena commented 2 months ago

There are some error messages but I wanted to re-write them here where they're easier to read :)

there is a 'url failing error' which is because the gtn doesn't support slides not named slides.html, I will think of how to best handle this.

PatCapon39 commented 1 month ago

@hexylena I'm having some trouble getting GTN to recognise math markdown using the $$ formatting described here https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-tutorial-content/tutorial.html#writing-mathematical-expressions

The particular file I'm trying to get math markdown to show up on is here https://github.com/Adamtaranto/training-material/blob/dev/topics/evolution/tutorials/abc_intro_phylo/03-msa_GTNformat.html

Any suggestions would be greatly appreciated!

burkemlou commented 1 month ago

Hi @hexylena,

Thanks for the fixes above.

And an update on slides in general. We've decided to push ahead and upload all the slides before going live. Michael's teaching schedule has changed which gives us some flexibility with our timeline for getting this done. We are still aiming for ~ end of April.

I'll write a story for the GTN and the BioCommons newsletter once it is out.

hexylena commented 1 month ago

having some trouble getting GTN to recognise math markdown using the $$ formatting described here.

@PatCapon39 the most likely cause is that recently i made mathjax load conditionally when it found $$ on the page, to save bandwidth for users. it's likely there is something wrong on my end with that. I'll look into it!

workflows, where do we add the licence and creator?

huh we don't have an faq for this, we really should. If you load the workflow in the galaxy workflow editor you'll find this menu where you can add relevant creators and organisations: Screenshot_20240404-092651

Would renaming to Slides_1.html etc fix this error?

they need to be in separate folders, with a slides.html per folder (at the same level as other tutorials). I can push a commit with this change today probably and provide the linking

We are still aiming for ~ end of April.

that sounds great! I will ensure it is merged by then

hexylena commented 1 month ago

Fyi, when testing this tutorial I found some issues with the visualisations, just in case y'all hadn't experienced the same https://github.com/galaxyproject/galaxy/issues/17770

hexylena commented 1 month ago

I've split the slide decks into 4 different folders, so they'll pass rendering checks.

And for the math rendering it looks like there is indeed a bug of some sort :( I will look into it further.

hexylena commented 1 month ago

in the MSA slide deck I see: internal image images/DPcell.pdf does not exist, is that file maybe missing?

PatCapon39 commented 1 month ago

in the MSA slide deck I see: internal image images/DPcell.pdf does not exist, is that file maybe missing?

Yes we saw that too, thanks! And thanks for your efforts to fix the math markdown too

PatCapon39 commented 1 month ago

@hexylena It seems like the math markdown is working now, but every time I exist the maths via double dollar signs, it starts a new line. How do I keep the maths on the same line as normal text? ChatGPT tells me to use a single dollar sign, but I don't think GTN recognises those.

hexylena commented 1 month ago

but every time I exist the maths via double dollar signs, it starts a new line

@PatCapon39 I believe this will be fixed with some commits from @shiltemann that she'll push shortly. She's got it rendering correctly with the $$ .. $$ in inline-mode as well as block mode

shiltemann commented 1 month ago

I pushed a fix for the math, any math expressions should be placed between double dollar signs $$ .. $$ (inline or not) and it should work again :+1:

hexylena commented 1 month ago

@shiltemann looks like the fix possibly complicated another tutorial, FNN. Which has some really fascinating markup that looks like a markdown link: (fnn/tutorial.md#L265) https://saxarona.github.io/mathjax-viewer/?input=%5Cboldsymbol%7B%5Cdelta%7D%5E%7B%5BL%5D%28j%29%7D+%3D+%5Cnabla_%7B%5Cboldsymbol%7B%5Chat%7By%7D%5E%7B%28j%29%7D%7D%7D%5Cmathcal%7BL%7D+%5Codot+%28g%5E%7B%5BL%5D%7D%29%5E%7B%27%7D+%28%5Cboldsymbol%7Bz%7D%5E%7B%5BL%5D%28j%29%7D%29+%3D+%5Cboldsymbol%7B%5Chat%7By%7D%5E%7B%28j%29%7D%7D+-+%5Cboldsymbol%7By%5E%7B%28j%29%7D%7D+

shiltemann commented 1 month ago

@shiltemann looks like the fix possibly complicated another tutorial, FNN. Which has some really fascinating markup that looks like a markdown link: (fnn/tutorial.md#L265) https://saxarona.github.io/mathjax-viewer/?input=%5Cboldsymbol%7B%5Cdelta%7D%5E%7B%5BL%5D%28j%29%7D+%3D+%5Cnabla_%7B%5Cboldsymbol%7B%5Chat%7By%7D%5E%7B%28j%29%7D%7D%7D%5Cmathcal%7BL%7D+%5Codot+%28g%5E%7B%5BL%5D%7D%29%5E%7B%27%7D+%28%5Cboldsymbol%7Bz%7D%5E%7B%5BL%5D%28j%29%7D%29+%3D+%5Cboldsymbol%7B%5Chat%7By%7D%5E%7B%28j%29%7D%7D+-+%5Cboldsymbol%7By%5E%7B%28j%29%7D%7D+

oh oops, I will look into it

shiltemann commented 1 month ago

ok, there are too many edge cases that go wrong with my fix, so let's go back to the mathjax standard; please:

I updated this in your trees slidedeck

If you notice any math expressions that still do not render well, please let me know!

PatCapon39 commented 1 month ago

Thanks @shiltemann! I've fixed the maths in the msa slide set. We've uploaded another slide set called MLE trees, but I can't quite get it to go in the right spot. Rather than being grouped with the other phylogenetics topics, it's listed under other. Can you or @hexylena please help me get it into the correct location? See attached picture for what I mean.

Screenshot 2024-04-11 at 12 09 06 PM
PatCapon39 commented 1 month ago

I am also trying to follow @hexylena's set up of the sym links for image folders for this new slide set called mle-trees, but I can't figure out how to create a sym link like you did. Can you please show me how? I can't find anything on GTN and google is confusing me!

PatCapon39 commented 1 month ago

Lastly, I have tried to use inline maths for slide set 5 (mle-trees) but it isn't behaving as the msa slides did and I don't understand why. I also have a series of "IQTree outputs" slides that appear the be coded the same, but are rendering the title differently and the last one is not obeying the .reduce50[] command, again I am unsure why this is happening. Any help would be great please!

shiltemann commented 1 month ago

@PatCapon39

PatCapon39 commented 1 month ago

Thanks @shiltemann! For whatever reason the subtopic setting wasn't displaying for me in gitpod yesterday, it's working today thankfully. The maths still doesn't look right for MLE-trees at slide 9, it doesn't seem to be recognising the \\(..\\) properly. Maybe something to do with the bullet points? For slided 13, I would like to line up the equals signs. Or at least, get them to all be left aligned, but it doesn't seem to working with the .left[] code that I've used previously.

The layout for iqtree slides has worked nicely, thanks! It does mean that those slides are missing the little blue back arrow at the top left though.

I am struggling to get the image on slide 26 to reduce in size properly. It doesn't seem to be recognising the image-25 code (or other sizes like 60 or 70 that I've tried). Is there something I'm doing wrong?

PatCapon39 commented 1 month ago

The image resizing on slide 26 seems to be fixed now. I think this is a GitPod issue where it is not fully refreshing. The image resizing worked later this morning when I had to re-open the workspace following a time out.

shiltemann commented 1 month ago

@PatCapon39 thanks for your feedback, your slides are a great test case to find and fix al edge cases of our slides framework :)

Thanks for your patience and of course these great materials!

hexylena commented 1 month ago

The promised multi-format selector is implemented, the buttons don't look so nice but they'll be updated in #4890 I am very happy to see this come to fruition, it is so nice as a user to have immediate easy access to the alternative formats that might work better for me. This is fantastic work, all.

a screenshot of a tutorial showing the selector, a green box has options for "Instructor Recording", slides, and plain text

hexylena commented 1 month ago

Slide logos are fixed as well :)

image

burkemlou commented 3 weeks ago

Hello @hexylena - following up on your request to update the workflow file from a while back.

I tried loading the tree-bulding.ga file into Galaxy but I get an error 'Uncaught exception in exposed API method:'. I can load other .ga files so it seems to be something about this one. Any suggestions? I haven't used this feature before so let me know if I'm doing something odd here.

Alternatively, I can see in the .ga file that there is this part:

"creator": [ { "class": "Person", "identifier": "0000-0001-9760-8992", "name": "Helena Rasche" } ], "format-version": "0.1", "license": "MIT",

Is this the part that needs updating? I can do that manually and add a CC-BY licence. Is there a particular format needed for the licence?

@Adamtaranto and @mcharleston I see that we still have main_workflow.ga and main_workflow-tests.yml files in the folder for this tutorial. These seem to be copies of the workflow from another tutorial. Do we need these files or are they left over from when you were first setting up the folders and structure for the tutorial?

Nearly done with the slides here. I'll get back to you later in the week.

hexylena commented 3 weeks ago

@burkemlou I've updated the tree-building.ga that would not load in galaxy, it loads now. I cannot explain the issue, but it seems like the tool was updated and now it works.

I see that we still have main_workflow.ga and main_workflow-tests.yml files in the folder for this tutorial. These seem to be copies of the workflow from another tutorial. Do we need these files or are they left over from when you were first setting up the folders and structure for the tutorial?

ahh that would explain for me why they didn't match at all the tutorial, it was why I ended up making the main-workflow.ga file (7fa178fb37c403159227871d0848937022142508) based on the instructions of the tutorial (hence the creator information in that file). But please feel free to change its license to whatever you prefer (and make any improvements or changes you like.)

In that case perhaps we can delete them?

burkemlou commented 2 weeks ago

@hexylena @mcharleston FYI I have a problem with visualising the outputs of IQTREE and FASTREE related to this activity:

`Estimating a Maximum Likelihood tree with IQTree

Find the {% tool IQTree %} program in the tool finder, and run it with the following parameters:

In "General options": {% icon param-file %} "Specify input alignment file in PHYLIP, FASTA, NEXUS, CLUSTAL or MSF format.": your aligned sequence data (output of MAFFT {% icon tool %}) "Specify sequence type as either of DNA, AA, BIN, MORPH, CODON or NT2AA for DNA, amino-acid, binary, morphological, codon or DNA-to-AA-translated sequences": DNA In "Modelling Parameters": In "Rate heterogeneity": "Write maximum likelihood site ratios to .mlrate file": Yes In "Bootstrap Parameters": In "Ultrafast bootstrap parameters": "Specify number of bootstrap replicates (>=1000).": 1000 Click "Run tool".

Click on the output of IQ-Tree and select the visualisation icon. Select 'Phyloviz' to view your tree.`

The visualisation is not working at all in Galaxy AU (Galaxy Australia Support request. 199713). It produces a tree in Galaxy EU but the options to modify the tree are not working as expected (also logged). Galaxy AU reports that this is not trivial to fix.

@mcharleston can we consider a different tool for the purposes of this tutorial?

hexylena commented 2 weeks ago

Yes I saw the same last month, and filed it as https://github.com/galaxyproject/galaxy/issues/17770 and pinged some folks (@neoformit et al).

I thought I linked it here to ensure everyone was in the loop but seems like I potentially forgot to since I can't find it now.

I don't know what to do about that, perhaps y'all can use an external visualisation (they can maybe copy paste their newick) because galaxy isn't working at all for tree viz.