Update UTA and automate future updates - Githubissues

biocommons / hackathon-2023

Hackathon 2023 projects and planning.

0 stars 0 forks source link

Update UTA and automate future updates #6

Open korikuzma opened 1 year ago

korikuzma commented 1 year ago

Submitter Name

@reece

Submitter Affiliation

MyOme

Requested By

Everyone using UTA

Lead(s)

@reece

biocommons Repo

uta

Project Details

Hackathon Project Slide

UTA data has not been updated since Jan 29, 2021. Instructions for updating UTA is here. This project aims to automate UTA updates and releases to dl.biocommons.org and Docker images.

Hackathon project plan:

Incorporate #11 into this project
Review release process and identify challenges
Update UTA manually once
Develop vision for ideal loading process and automation strategy/tools
Identify subprojects to tackle during hackathon and get to it!

Skill Level

Advanced

Required Skills

Python, Docker

korikuzma commented 1 year ago

@reece @andreasprlic Would you be able to provide any additional information on this?

andreasprlic commented 1 year ago

The main problem is that the update procedure is tied to a local setup on a specific system
Specific steps in a virtual env on that system need to get run.
Sometimes these steps break, since there are file changes at NCBI. steps need to get updated. (we prob can't easily fix this aspect)
The best way going forward is to change the update procedure to wrap these steps as tasks in a workflow (pick your favorite workflow engine)
After an update we would be doing manual QC on the new UTA build. Can we add automated QC as a new set of steps at the end of the workflow?

andreasprlic commented 1 year ago

Suggestions for workflow engines: Conductor (slight preference), Airflow, Nextflow

andreasprlic commented 1 year ago

QC steps:

make sure all entrez gene make it into the database.
make sure the latest version of each transcript is in the database.
no null gene symbols
no null cigar strings
make sure alignments are complete for transcripts (compare the length with the cigar strings)
make sure cds start/stop and NP accessions are associated with all coding transcripts
check all transcripts and write warns for deprecated transcripts to a file
check all transcripts and write warns for transcripts with ref-disagrees to a file

andreasprlic commented 1 year ago

Feature request for improved update procedure:

get the cigar string for exon alignment from NCBI, rather than computing ourselves (big speed up)

andreasprlic commented 1 year ago

@reece can we grant all people who will work on this project access to the minion host (or whatever it is called nowadays)?

does this host have a recent copy of ncbi-mirrors?

reece commented 1 year ago

@andreasprlic Yes.

The machine is stuart (yes, a minion). Yes, current ncbi-mirrors.

Since this account is paid for by MyOme, I'd like to not share root/sudo. That means we'll need a bit of planning.

It would be helpful to have login,real name, and ssh pub key beforehand so that I can do this in a batch.

I'll also make a separate pg RDS database for hackathon dev.

ktennessenInvitae commented 1 year ago

@andreasprlic Yes.

The machine is stuart (yes, a minion). Yes, current ncbi-mirrors.

Since this account is paid for by MyOme, I'd like to not share root/sudo. That means we'll need a bit of planning.

It would be helpful to have login,real name, and ssh pub key beforehand so that I can do this in a batch.

I'll also make a separate pg RDS database for hackathon dev.

@reece What is the best way to send you ssh pub key?

reece commented 1 year ago

Hi @ktennessenInvitae: I just DM'd you on slack. @andreasprlic : Ditto