search

igemmcmaster / genome-transformer

Pretrained efficient transformers on genomes -- WIP
3 stars 2 forks source link

Hello! #12

Open lucidrains opened 3 years ago

lucidrains commented 3 years ago

This is great! Anyway I can help out? Are you all doing this for school or part of a lab?

lucidrains commented 3 years ago

Have you seen https://github.com/kheyer/Genomic-ULMFiT

Lev1ty commented 3 years ago

Hi Phil! Thank you for reaching out! We were learning from some of the repositories you've created just earlier yesterday. To answer your questions... Yes! We would appreciate any help; we're currently at the stage of implementing the Performer, Sub-linear memory performer, and nystromformer. We're prioritizing nystromformer because it seems easier to implement and apparently fairs better in terms of time to convergence and long range attention compared to performer. This project is a part of a student research team/club at our university called iGEM McMaster and our focus is on genetic engineering and synthetic biology. We found that there's some missing literature on efficient transformers for analyzing longer gene sequences (100k+ nucleotides) so we've decided to open up this project. The genomic-ULMFiT seems cool and the reading links on that repository are definitely relevant. If you're interested, we could reach out to you over email to further discuss collaboration.

-Adam