search

phenopolis / phenopolis_genomics_browser

Python API and React frontend for the Phenopolis Genomics Browser
https://dev-live.phenopolis.org
MIT License
31 stars 2 forks source link

Replace PySAM with cyvcf2 #229

Closed alanwilter closed 3 years ago

alanwilter commented 3 years ago

To address #49

But attention!

It's a complete new docker image based on amazon linux. Though it may be more reassuring in terms of compatibility with AWS EC2, tests in GitHub workflow will be longer.

@IsmailM I've attempt to guess a proper solution for .platform/hooks/prebuild/01_install_packages.sh. You need to be ready to intervene once the merge is done and for some unbeknownst reason dev-live does not work.

CRITICAL cyvcf2 does not need (actually, does not work with) boto3. It depends entirely on amazon env vars:

AWS_SECRET_ACCESS_KEY=.....
AWS_ACCESS_KEY_ID=....

So @IsmailM you'll need to update that at AWS EC2 servers, i.e., replace VCF_S3_KEY and VCF_S3_SECRET accordingly wherever you see them.

And users/developers, you'll also need to update your private.env.

alanwilter commented 3 years ago

I'm testing with a browser http://localhost:8888/variant/14-76129628-T-C and it's not working:

Unhandled Rejection (TypeError): undefined is not an object (evaluating 'error.response.status')
(anonymous function)
src/redux/actions/variant.js:20
  17 |   }
  18 | })
  19 | .catch((error) => {
> 20 |   if (error.response.status === 401) {
     | ^  21 |     dispatch({
  22 |       type: SET_STATUS,
  23 |       payload: { code: 401, message: error.response.data.error, relink: '/variant/' + param },
View compiled
This screen is visible only in development. It will not appear if the app crashes in production.
Open your browser’s developer console to further inspect this error.  Click the 'X' or hit ESC to dismiss this message.
alanwilter commented 3 years ago

I'm wondering if this error is due to the variant reorganisation where subset arg was removed and, perhaps, the front end needs an accordingly update.

IsmailM commented 3 years ago

Looks amazing. I will probably merge into dev-live on Friday afternoon - I have a few deadlines this week and will not be able to put aside half a day for this until then (and I'm sure I will need half a day as I'm sure something will go wrong - Murphy's law - which seems to especially apply with any HTSLIB stuff I do lol)

Saying that, if I get a moment before then, I'll probably manually install cyvcf as per your pip3 install command and see if I can run some exemplar code in the python3 interactive shell.

And if that works, then we can simply push to dev-live and restart the servers (for a clean build) after an initial successful build...

alanwilter commented 3 years ago

Indeed @IsmailM, thread carefully, take your time and test as you mentioned. cyvcf2 may take a while to compile. If any issues, let me know.

IsmailM commented 3 years ago

Hey @alanwilter,

I merged the PR - however, I am not seeing anything on the exemplar variant page under the frequency tab on the dev site.

On the other hand, there is a single row shown on the live website.

I removed the web server, causing a new instance to be created from scratch AND CyVCF seems to be installed perfectly:

Xnip2020-11-11_22-50-13

Also, I cannot see anything obvious on the web logs

Do you have any ideas why this is an issue?

Do you see a difference on your local variant page compared to the one in production?

alanwilter commented 3 years ago

Have a look at phenopolis_react slack channel, we've been discussing earlier about issues like you've seen. I can tell you it's not due to CyVCF, but we've done, mostly Pablo, changes in the variant.py structure, however there's also, I believe, things for Yuan to catch up.