Psy-Fer
commented
1 month ago Hey,
Okay, so let me go through these
It is a great suggestion about the model conversions. @Psy-Fer, we should maintain some server to standalone model mapping page in https://github.com/Psy-Fer/buttery-eel/blob/main/docs/. The tricky thing is these models keep changing from version to version, so perhaps we can document the use of the following command.
The "new" dorado method is not present in the dorado-server method, as the dorado-server method is still based on the old guppy basecaller paradigms. The model naming and command line methods will only change in buttery-eel as they change in dorado-server, as I'm not going to try and do translations between 2 tools. Also keep in mind, that dorado-server is what is running when you run MinKNOW live basecalling, and the models are still evoked this way internally with MinKNOW.
However, adding a table in the docs to show the model names and locations could be helpful, as well as providing some cmdline code or a script for dumping the model info for the user.
I am not sure if Dorado server supports alignment. @Psy-Fer Does it?
Dorado-server does support alignment, however buttery-eel does not, and will not, support it. This decision was made for 3 reasons, some of which @hasindu2008 already mentioned
- It is my firm belief that tools that run chronologically should be run that way, rather than interleaved, when possible unless there is some specific need for it. There is no time saving doing alignment during basecalling vs after and I would guess it's actually faster to do them separately.
- Maintaining alignment as a function of buttery-eel adds extra work on top of already dealing with constant breaking changes from ONT (and my own bugs). Alignment has nothing to do with S/Blow5 files, so is not part of the solution butter-eel is trying to solve
- I don't trust alignment in dorado or dorado-server and neither should you
further to this, if you want to run everything in "one command", a few lines in a bash script will get you there.
I hope this helps to clarify things. I know I have some strong opinions here, but they are not without some pretty strong evidence over the years working with these ONT tools/APIs.
Let me know if you have more questions
Cheers, James
jelber2
hasindu2008
SBurnard
@SBurnard Moving this https://github.com/hasindu2008/nci-scripts/issues/1 conversation to here as these questions are about buttery-eel.
It is a great suggestion about the model conversions. @Psy-Fer, we should maintain some server to standalone model mapping page in https://github.com/Psy-Fer/buttery-eel/blob/main/docs/. The tricky thing is these models keep changing from version to version, so perhaps we can document the use of the following command.
@SBurnard Buttery-eel relies on the dorado-server from ONT (which does the live basecalling in MinKNOW) to implement the basecalling. So these model configuration convention comes from ONT's dorado-server and due to some reason, they have a different convention in standalone Dorado. How I find the models is as follows on Gadi:
About the second question, slow5-dorado is a fork of the standalone Dorado, so all the extra features in Dorado such as alignment are there. But we have not made a release recently:
The good thing with the dorado-server is we can simply get the binary from ONT and use the client-server approach (implemented in buttery-eel) to access BLOW5 files.
I am not sure if Dorado server supports alignment. @Psy-Fer Does it? However, even if it supports alignment I personally believe that having basecalling and alignment to be modular has greater benefits:
I can go on .....
Let me cite the following extract from Heng Li, Minimap and miniasm: fast mapping and de novo assembly for noisy long sequences, Bioinformatics "We hope in this process, the community could standardize the input and output formats of various tools, so that a developer could focus on a component he or she understands best. Such a modular approach has been proved to be fruitful in the development of short-read tools—in fact, the best short-read pipelines all consist of components developed by different groups—and will be equally beneficial to the future development of long-read mappers and assemblers."
I understand that having a single command that runs all could be convenient, but not sure if it really worth considering the above factors. What do you think?