search

neherlab / ncov-simple

2 stars 1 forks source link

Use prealignment from ncov-ingest, save computation time #23

Open corneliusroemer opened 3 years ago

corneliusroemer commented 3 years ago

Doesn't seem to run daily though as of now, this would be the endpoint s3://nextstrain-ncov-private/aligned.fasta.xz

Blocked until daily updates.

Edit: seems to run most weekdays, so unblocked

emmahodcroft commented 3 years ago

I think this runs every weekday? Not sure why I don't see one for today - but searching back in the slack, there's one for 4th, (not 3rd - ingest broke), 2nd, 1st, then October 29, 28, 26, 25 (I don't know why not 27th?). It seems like it's trying to run every weekday, but maybe ingest is breaking now and then?

corneliusroemer commented 3 years ago

Thanks for checking Emma, my Slack search was apparently not as thorough as yours, maybe I don't know the right search query to do it well.

So I'll unblock.

emmahodcroft commented 3 years ago

Well - might still be worth looking into a little - figuring out why it might be failing about once a week? Or implement a check on your side so you know if it worked or not (no use re-running yesterday's stuff).

I just searched for Updated s3://nextstrain-ncov-private/aligned.fasta.xz available.

here's all the dates I find on the first page:

October
M  T  W  R  F  S  S
      6  7  8 
11 12 13 14    16
18 19    21 22
25 26    28 29

Nov
M  T  W  R  F  S  S
1  2     4

unsure why we seem to have a weakness on Wednesdays and Fridays?

corneliusroemer commented 3 years ago

You're a star! Wednesdays no new data, maybe? If it fails Friday, would be nice to just always run Sat/Sun.

emmahodcroft commented 3 years ago

Looks like at least this Wed & last Wed, ingest broke (different reasons). Wed Oct 20th, we got new metadata and sequences files (so implies new data) but no alignment. No real clue why on slack. Similarly, today we got new metadata & sequences, but no alignment 🤷