Use prealignment from ncov-ingest, save computation time

[corneliusroemer]

Doesn't seem to run daily though as of now, this would be the endpoint s3://nextstrain-ncov-private/aligned.fasta.xz

Blocked until daily updates.

Edit: seems to run most weekdays, so unblocked

[emmahodcroft]

I think this runs every weekday? Not sure why I don't see one for today - but searching back in the slack, there's one for 4th, (not 3rd - ingest broke), 2nd, 1st, then October 29, 28, 26, 25 (I don't know why not 27th?). It seems like it's trying to run every weekday, but maybe ingest is breaking now and then?

[corneliusroemer]

Thanks for checking Emma, my Slack search was apparently not as thorough as yours, maybe I don't know the right search query to do it well.

So I'll unblock.

[emmahodcroft]

Well - might still be worth looking into a little - figuring out why it might be failing about once a week? Or implement a check on your side so you know if it worked or not (no use re-running yesterday's stuff).

I just searched for Updated s3://nextstrain-ncov-private/aligned.fasta.xz available.

here's all the dates I find on the first page:

October
M  T  W  R  F  S  S
      6  7  8 
11 12 13 14    16
18 19    21 22
25 26    28 29

Nov
M  T  W  R  F  S  S
1  2     4 

unsure why we seem to have a weakness on Wednesdays and Fridays?

[corneliusroemer]

You're a star! Wednesdays no new data, maybe? If it fails Friday, would be nice to just always run Sat/Sun.

[emmahodcroft]

Looks like at least this Wed & last Wed, ingest broke (different reasons). Wed Oct 20th, we got new metadata and sequences files (so implies new data) but no alignment. No real clue why on slack. Similarly, today we got new metadata & sequences, but no alignment 🤷