search

lidaof / eg-react

WashU Epigenome Browser
https://epigenomegateway.wustl.edu
Other
68 stars 30 forks source link

Data fetch failed error on gene annotation using own backend API #101

Closed jos4uke closed 5 years ago

jos4uke commented 5 years ago

Hi @lidaof

I installed a local browser and deploy MongoDB to have my own backend API following the steps on https://eg.readthedocs.io/en/latest/installation.html

I then did empty the AWS_API to enable my own API:

# eg-react/frontend/src/dataSources/GeneSource.js
//export const AWS_API = "https://api.epigenomegateway.org"; 
export const AWS_API = ""

The frontend restarted automatically after I saved the modifications. But I issued an error onto the gene annotation tracks with a message Data fetch failed. Reload page or change view to retry. (Dismiss). I got the same message for mm10 refGene and gencodeM19Basic annotation tracks.

I reloaded the page, also changed the view to some other genome coordinates, zoom in and out but still the same message.

I checked the mongo databases, they are ok for all genomes. I checked the backend APi is alive on port 3001.

What do you think, did I miss something deploying the backend API?

Jos

jos4uke commented 5 years ago

I also tried using the backend API url but did not work:

# eg-react/frontend/src/dataSources/GeneSource.js
//export const AWS_API = "https://api.epigenomegateway.org"; 
export const AWS_API = "http://localhost:3001"
lidaof commented 5 years ago

Hi @jos4uke , could you please open the web console see what error msg pop up? when you open http://localhost:3001/documentation, can you test the API as well?

jos4uke commented 5 years ago

the API documentation url works for me, and I also tested the API as well

washu_epigenome_backend_api_doc_test

jos4uke commented 5 years ago

I checked the webconsole and saw the following Cross-Origin Request Blocked warning:

washu_epigenome_browser_webconsole

lidaof commented 5 years ago

i see, can you try this?

add a proxy field to your package.json:

"proxy": "http://localhost:3001",

then in GeneSource, just use / as the API root, like remove $AWS_API:

return axios.get(`/${genome}/genes/${this.trackModel.name}/queryRegion`, {
                params: params
            });

you may need re-run npm start

jos4uke commented 5 years ago

I made the modifications and got the following error:

washu_epigenome_webconsole_remove_AWS_API

lidaof commented 5 years ago

can http://localhost:3000/documentation be opened?

jos4uke commented 5 years ago

yes i can

washu_epigenome_backend_api_doc

lidaof commented 5 years ago

sure, i mean the 3000 port on documentation

jos4uke commented 5 years ago

oh sorry, no it does not, I got a 404 error

lidaof commented 5 years ago

can i see you package.json file?

lidaof commented 5 years ago

can you also please check this link https://facebook.github.io/create-react-app/docs/proxying-api-requests-in-development? thanks.

jos4uke commented 5 years ago

Hi @lidaof

here is the content of eg-react/frontend/package.json

{
  "name": "epgg",
  "version": "47.2.8",
  "repository": {
    "type": "git",
    "url": "https://github.com/lidaof/eg-react.git"
  },
  "scripts": {
    "build": "react-app-rewired build --scripts-version react-scripts-ts && npm run build-dist",
    "build-dist": "rm -f umd/* && cd build && cp static/js/*.js ../umd/epgg.js && cp static/css/*.css ../umd/epgg.css && cp *worker.js ../umd/ && cd ../umd && sed -i ''  's//browser/./g' *.js",
    "build-storybook": "build-storybook -s public",
    "eject": "react-scripts-ts eject",
    "start": "react-app-rewired start --scripts-version react-scripts-ts",
    "storybook": "start-storybook -p 9009 -s public",
    "test": "react-app-rewired test --env=jsdom --scripts-version react-scripts-ts",
    "deploy": "aws s3 sync ~/eg-react/frontend/build/ s3://epigenomegateway.org/browser/ --delete",
    "build-pkg": "rm -rf dist && cross-env NODE_ENV=production babel src/ --out-dir dist --extensions \".ts,.tsx,.js,.jsx\" --copy-files --ignore __tests__,spec.js,test.js,__snapshots__,umd,build,public,node_modules"
  },
  "dependencies": {
    "acorn": "^6.0.4",
    "ajv": "^6.6.1",
    "eslint": "^5.10.0",
    "eslint-config-react-tools": "^1.1.6",
    "jest": "^23.6.0",
    "react-scripts": "^2.1.1",
    "webpack": "^3.12.0"
  },
  "peerDependencies": {
    "prop-types": "^15.6.0",
    "react": "^16.3.0",
    "react-dom": "^16.3.0"
  },
  "devDependencies": {
    "@babel/cli": "^7.1.5",
    "@babel/core": "^7.1.5",
    "@babel/plugin-proposal-class-properties": "^7.1.0",
    "@babel/plugin-proposal-object-rest-spread": "^7.0.0",
    "@babel/preset-env": "^7.1.5",
    "@babel/preset-react": "^7.0.0",
    "@babel/preset-typescript": "^7.1.0",
    "@firebase/app": "^0.3.3",
    "@storybook/addon-actions": "^3.3.15",
    "@storybook/addon-links": "^3.3.15",
    "@storybook/addons": "^3.3.15",
    "@storybook/react": "^3.3.15",
    "@types/d3-axis": "^1.0.10",
    "@types/d3-scale": "^2.0.1",
    "@types/d3-selection": "^1.3.1",
    "@types/firebase": "^3.2.1",
    "@types/jest": "^23.0.0",
    "@types/jquery": "^3.3.2",
    "@types/json5": "^0.0.29",
    "@types/lodash": "^4.14.109",
    "@types/memoize-one": "^3.1.1",
    "@types/node": "^10.3.2",
    "@types/query-string": "^6.1.0",
    "@types/react": "^16.3.17",
    "@types/react-dom": "^16.0.6",
    "@types/react-notify-toast": "^0.5.1",
    "@types/react-redux": "^6.0.2",
    "@types/shortid": "0.0.29",
    "@types/uuid": "^3.4.3",
    "axios": "^0.16.2",
    "babel-cli": "^6.26.0",
    "babel-plugin-transform-decorators-legacy": "^1.3.5",
    "circos": "^2.1.0",
    "cross-env": "^5.2.0",
    "d3": "^4.12.0",
    "enzyme": "^3.3.0",
    "enzyme-adapter-react-16": "^1.1.1",
    "firebase": "^5.9.2",
    "jest-image-snapshot": "^1.0.1",
    "jquery": "^3.2.1",
    "jquery.kinetic": "^2.2.4",
    "json5": "^1.0.1",
    "jsonschema": "^1.2.0",
    "lodash": "^4.17.5",
    "memoize-one": "^3.1.1",
    "nightmare": "^2.10.0",
    "p": "^0.2.0",
    "parse-color": "^1.0.0",
    "promise-worker": "^1.1.1",
    "prop-types": "^15.6.0",
    "query-string": "^6.1.0",
    "react": "^16.3.0",
    "react-app-rewired": "^1.3.5",
    "react-autosuggest": "^9.3.2",
    "react-beautiful-dnd": "^3.0.0",
    "react-bootstrap-tabs": "^1.0.2",
    "react-collapsible": "^2.3.1",
    "react-color": "^2.14.0",
    "react-copy-to-clipboard": "^5.0.1",
    "react-dom": "^16.3.0",
    "react-hot-keys": "^1.2.2",
    "react-modal": "^3.6.1",
    "react-notify-toast": "^0.5.0",
    "react-popper": "^0.8.2",
    "react-radio-buttons": "^1.2.2",
    "react-redux": "^5.0.7",
    "react-redux-firebase": "^2.1.8",
    "react-router-dom": "^4.3.1",
    "react-scripts-ts": "^2.16.0",
    "react-speech-recognition": "^1.0.7",
    "react-table": "^6.7.4",
    "react-test-renderer": "^16.3.0",
    "redux-undo": "^1.0.0-beta9-9-7",
    "script-loader": "^0.7.2",
    "shortid": "^2.2.13",
    "tcp-port-used": "^0.1.2",
    "typescript": "^2.9.1",
    "underscore": "^1.8.3",
    "uuid": "^3.3.2",
    "worker-loader": "^1.1.1",
    "zlib": "^1.0.5"
  },
  "homepage": "http://epigenomegateway.wustl.edu/browser/",
  "license": "SEE LICENSE IN LICENSE.txt",
  "keywords": [
    "WashU Epigenome Browser",
    "genomics",
    "data visualization"
  ],
  "main": "dist/lib.js",
  "module": "dist/lib.js",
  "files": [
    "umd",
    "dist"
  ],
  "proxy": "http://localhost:3001"
}
jos4uke commented 5 years ago

Thanks @lidaof I checked for the link you sent me and I realized I don't put the proxy property in the right place, I set it in the frontend app instead of the backend, my bad.

Here is the content of eg-react/backend/package.json

{
  "name": "backend",
  "version": "1.0.0",
  "description": "eg-react backend",
  "main": "main.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "setup": "node setup/setupMongo.js",
    "start": "node main.js dev",
    "start-prod": "node main.js prod",
    "start-api": "node main.js api"
  },
  "author": "",
  "license": "MIT",
  "dependencies": {
    "assert": "^1.4.1",
    "boom": "^7.2.0",
    "good": "^8.0.0",
    "good-console": "^7.1.0",
    "good-squeeze": "^5.0.2",
    "hapi": "^17.2.0",
    "hapi-swagger": "^9.1.1",
    "inert": "^5.1.0",
    "joi": "^13.1.2",
    "lodash": "^4.17.5",
    "mongodb": "^3.0.1",
    "vision": "^5.3.1",
    "yesno": "0.0.1"
  },
  "proxy": "http://localhost:3001"
}

And now it works

washu_epigenome_own_backend_api_works

lidaof commented 5 years ago

Glad it’s working now :)

On Friday, April 5, 2019, Joseph Tran notifications@github.com wrote:

Thanks @lidaof https://github.com/lidaof I checked for the link you sent me and I realized I don't put the proxy property in the right place, I set it in the frontend app instead of the backend, my bad.

Here is the content of eg-react/backend/package.json

{ "name": "backend", "version": "1.0.0", "description": "eg-react backend", "main": "main.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1", "setup": "node setup/setupMongo.js", "start": "node main.js dev", "start-prod": "node main.js prod", "start-api": "node main.js api" }, "author": "", "license": "MIT", "dependencies": { "assert": "^1.4.1", "boom": "^7.2.0", "good": "^8.0.0", "good-console": "^7.1.0", "good-squeeze": "^5.0.2", "hapi": "^17.2.0", "hapi-swagger": "^9.1.1", "inert": "^5.1.0", "joi": "^13.1.2", "lodash": "^4.17.5", "mongodb": "^3.0.1", "vision": "^5.3.1", "yesno": "0.0.1" }, "proxy": "http://localhost:3001" }

And now it works

[image: washu_epigenome_own_backend_api_works] https://user-images.githubusercontent.com/1325049/55611617-52ecde00-5786-11e9-9840-a2e7e3c58e3d.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lidaof/eg-react/issues/101#issuecomment-480178348, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGRlqand3KTYoyHuB7w2KPl0d8lAG8Mks5vdv0WgaJpZM4cdPf3 .

-- Daofeng

jos4uke commented 5 years ago

I have another use case which leads to the same error. I tried to use a remote host server to deploy eg-react instead of local. So, I rebind mongodb server with the remote address in /etc/mongod.conf

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1,remoteHost.domain

I updated the backend app with the remoteHost.domain address in those files:

I created in both frontend/backend apps a new file .env.development like recommended here https://github.com/facebook/create-react-app/issues/2271 

# eg-react/frontend/.env.development
# eg-react/backend/.env.development
HOST=remoteHost.domain

I then rebooted the frontend and backend apps

# backend
npm start

> backend@1.0.0 start /root/eg-react/backend
> node main.js dev

Starting server in DEV mode...
Established MongoDB connection at mongodb://remoteHost.domain:27017
Server running at http://remoteHost.domain:3001

# frontend
npm start
ts-loader: Using typescript@2.9.2 and /root/eg-react/frontend/tsconfig.json
ts-loader: Using typescript@2.9.2 and /root/eg-react/frontend/tsconfig.json
ts-loader: Using typescript@2.9.2 and /root/eg-react/frontend/tsconfig.json
ts-loader: Using typescript@2.9.2 and /root/eg-react/frontend/tsconfig.json
Compiled successfully!

You can now view epgg in the browser.

  http://remoteHost.domain:3000/

Note that the development build is not optimized.
To create a production build, use yarn build.

But I issued the same kind of error "Data fetch failed" as before. Maybe you know how to fix it.

washu_epigenome_remoteHost_data_fetch_failed

lidaof commented 5 years ago

is your remote API accessible through the remote ip address:port?

jos4uke commented 5 years ago

yes it is accessible

washu_epigenome_remoteHost_own_backend_api

lidaof commented 5 years ago

so the backend and frontend are in different servers or same server?

jos4uke commented 5 years ago

they are on the same physical server machine

jos4uke commented 5 years ago

digging into the error messages in the web console i saw the request is sent to the wrong port 3000 instead of 3001 so I believe the proxy i set into backend/package.json is not working

  "proxy": "http://remoteHost.domain:3001"

washu_epigenome_remoteHost_webconsole_api_errors

Is there another way to proxy the request?

jos4uke commented 5 years ago

I tested the api with the right port 3001 in the terminal with curl and it works:

curl -I "http://a09787.ips2.u-psud.fr:3001/mm10/genes/gencodeM19Basic/queryRegion?chr=chr6&start=52423564&end=52427674"
HTTP/1.1 200 OK
content-type: application/json; charset=utf-8
cache-control: no-cache
content-length: 2
Date: Fri, 05 Apr 2019 18:44:32 GMT
Connection: keep-alive

190405/184432.501, (1554489872501:a09787:2112:ju4eozwp:10001) [response,api] http://a09787.ips2.u-psud.fr:3001: head /mm10/genes/gencodeM19Basic/queryRegion {"chr":"chr6","start":52423564,"end":52427674} 200 (9ms)
lidaof commented 5 years ago

in the main.js from backend, can you try modify host: 'localhost', to host: 'a09787.ips2.u-psud.fr',?

jos4uke commented 5 years ago

I already did the change in backend/main.js

    if (environment === "dev") {
        const DEV_CONFIG = {
            dbUrl: 'mongodb://a09787.ips2.u-psud.fr:27017',
            host: 'a09787.ips2.u-psud.fr',
            port: 3001
        };
lidaof commented 5 years ago

i see....i don't know why right now in my mind,, i google'd around the proxy only works for development....

https://stackoverflow.com/questions/52877492/proxy-not-working-for-create-react-app-in-production

jos4uke commented 5 years ago

ok i understand now it may not be possible without using a proxy server like nginx I will try to install a proxy server

Thanks a lot @lidaof

lidaof commented 5 years ago

sure thing...sorry didn't help fixed your problem.

lidaof commented 5 years ago

our setup is we have a standalone EC2 instance at AWS, we are also switching to AWS lambda API endpoint.

jos4uke commented 5 years ago

not an easy use case don't worry : ) the proxy server should be a workaround for me

due to our policy I cannot use AWS unfortunately, we manage to do it with our own VMs so it should be fine I will talk about that with my IT guy thanks for all Best

lidaof commented 5 years ago

you are welcome, good luck!

jos4uke commented 5 years ago

Just to tell you using nginx as a proxy did the trick : )

lidaof commented 5 years ago

that's great! I am closing this one, feel free to open new issue.