Fix Swagger documentation and review gulpfile

tschaffter commented 4 years ago

The initial issue is reported in #535.

So far we only had the swagger documentation served when running the portal in development environment. The documentation is served on <portal_url>/api-docs.

In production environment, accessing <portal_url>/api-docs redirects to Swagger pet store example. The issue is that in production environment, comments (and so the swagger documentation) is stripped from the file as part of the build process to minimize the size of the app. Individual files where Swagger was configured to look at also no longer exist.

The solution tested in this PR is to build the swagger document swagger.json and then have the server reading this document when the server starts. The solution developed should support test, development and production environments.

tschaffter commented 4 years ago

A default gulp task could be the following:

gulp.series(
    gulp.parallel('lint', 'test'), 'build', gulp.parallel('watch', 'start')
)

tschaffter commented 4 years ago

[x] Review the task build

tschaffter commented 4 years ago

Notes on Environment variables (config)

Now

Before deploying the portal in development or production environment, we source a configuration file that includes environment variables. Each configuration file such as envvars-dev and envvars-prod include all the env vars, with a few different values depending on the environment.

Enhancement

The idea is to take inspiration from the approach used for the portal seeds and apply the same approach to configuration. Another motivation is to integrate the reading of the configuration as part of the deployment of the portal without relying any more on the user to run source <file> first.

We would have a default.env file that contains all the env vars required for the portal to run. Depending on the environment run (development, test, production), a second config file is provided that includes only the few env vars that must override the values defined in default.env. These second files would be development.env, test.env and production.env.

These config files need to be used at two different places:

In gulpfile.babel.js
In docker-compose.yml, where several env files can be provided, for example:

env_file:
  - ./common.env
  - ./apps/web.env
  - /opt/secrets.env

Edit: Using env_file is actually not possible as it should be possible to deploy the app using docker-compose up without depending on configuration files such as when we deploy with CodeDeploy, where all the config parameters are provided directly as env vars. Instead, I have written a gulp task to run docker-compose locally. The task does 1) create env vars from the local config file and 2) runs docker-compose up. The approach of loading env vars in gulp task has an extra benefit that is to not polute the env vars of the host.

tschaffter commented 4 years ago

Notes on SSL certificates

Now

The SSL certificates (certs/server.cert and server.key) are set as environment variables when sourcing envvars-dev, for example. As we are now reading env vars in gulpfile.babel.js, the content of the SSL certificates is now longer properly done (cat command is not executed).

Solution

Gulp tasks need to be added to load the SSL certificates and assign their content to environment variables. The server codebase must be changed so that it always read the SSL certification from the environment variables.

[x] Done

tschaffter commented 4 years ago

Here is the current error encountered by Travis:

[04:27:57] Error: ENOENT: no such file or directory, open 'config/certs/server.cert'

  at Object.openSync (fs.js:443:3)

  at Object.readFileSync (fs.js:343:35)

  at /home/travis/build/Sage-Bionetworks/PHCCollaborationPortal/gulpfile.babel.js:208:41

  at taskWrapper (/home/travis/build/Sage-Bionetworks/PHCCollaborationPortal/node_modules/undertaker/lib/set-task.js:13:15)

  at bound (domain.js:402:14)

  at runBound (domain.js:415:12)

  at asyncRunner (/home/travis/build/Sage-Bionetworks/PHCCollaborationPortal/node_modules/async-done/index.js:55:18)

  at process._tickCallback (internal/process/next_tick.js:61:11)

[ ] Solved

tschaffter commented 4 years ago

Commenting out Swagger in server/routes.js while working on gulp tasks to run the server.

tschaffter commented 4 years ago

Notes on loading of config and SSL certificate/key

Config files in config/*.env are loaded only if they exist
- These files likely exist in the local environment
- These files does not exist on Travis or when deployed using CodeDeploy (passed as env vars)
SSL certificate and private key are loaded only if they exist
- These files likely exist in the local environment
- These files does not exist on Travis or when deployed using CodeDeploy (passed as env vars)

Moreover, here are the current commands to build and start the portal in production environment immediately after having cloned the repos (work in progress).

npm run init:config
npm run init:ssl:server
npm run build
gulp serve:dist

Ultimately I would like to have a command that performs all these tasks, e.g. npm run quickstart.

tschaffter commented 4 years ago

Working on a gulp task to start the server in development environment. There is an issue:

[20:33:39] Starting 'start:client'...
ℹ ｢wds｣: Project is running at http://localhost:443/

It should be https://localhost:443/ (not http)

[x] This issue should be resolved with using new WebpackDevServer instead of the command webpack-dev-server.

tschaffter commented 4 years ago

[x] Cleanup webpack.server.js and its configuration in webpack.make.js.

tschaffter commented 4 years ago

[ ] In webpack.make.js, restore TerserPlugin with sourceMap: true.

tschaffter commented 4 years ago

Notes on MongoDB

I have updated the docker image phccp-mongo a week ago. Mongo now shows a deprecation warning for the following arguments:

phccp-mongo    | 2019-09-30T04:19:26.641+0000 W  CONTROL  [main] Option: sslMode is deprecated. Please use tlsMode instead.
phccp-mongo    | 2019-09-30T04:19:26.643+0000 W  CONTROL  [main] Option: sslPEMKeyFile is deprecated. Please use tlsCertificateKeyFile instead.

[ ] Address deprecation warnings (wait first for the docker-compose to work with gulp)

tschaffter commented 4 years ago

Notes on Docker and populating DB

This error is reported by the docker container phccp. activities involves making call to phccp-prov, which does not seems to be ready at the time of the population.

finished populating webapps
phccp          | finished populating starred messages
phccp-mongo    | 2019-09-30T04:19:30.426+0000 I  INDEX    [conn3] index build: done building index _id_ on ns phccp.messages
phccp-mongo    | 2019-09-30T04:19:30.428+0000 I  STORAGE  [conn5] createCollection: phccp.threads with generated UUID: 7dbb4ffc-24bb-4c88-992a-43d1bc187adb and options: {}
phccp-mongo    | 2019-09-30T04:19:30.442+0000 I  INDEX    [conn5] index build: done building index _id_ on ns phccp.threads
phccp          | finished populating messages
phccp          | finished populating articles
phccp          | finished populating action permissions
phccp          | finished populating data catalogs
phccp          | finished populating entity notifications
phccp          | finished populating message notifications
phccp          | finished populating reports
phccp          | finished populating states
phccp          | finished populating tools
phccp          | finished populating entity permissions
phccp          | finished populating threads
phccp          | finished populating users
phccp          | error populating activities { RequestError: Error: connect ECONNREFUSED 172.21.0.4:8080
phccp          |     at new RequestError (/usr/src/app/node_modules/request-promise-core/lib/errors.js:14:15)
phccp          |     at Request.plumbing.callback (/usr/src/app/node_modules/request-promise-core/lib/plumbing.js:87:29)
phccp          |     at Request.RP$callback [as _callback] (/usr/src/app/node_modules/request-promise-core/lib/plumbing.js:46:31)
phccp          |     at self.callback (/usr/src/app/node_modules/request/request.js:185:22)
phccp          |     at Request.emit (events.js:189:13)
phccp          |     at Request.onRequestError (/usr/src/app/node_modules/request/request.js:881:8)
phccp          |     at ClientRequest.emit (events.js:189:13)
phccp          |     at Socket.socketErrorListener (_http_client.js:392:9)
phccp          |     at Socket.emit (events.js:189:13)
phccp          |     at emitErrorNT (internal/streams/destroy.js:82:8)
phccp          |     at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
phccp          |     at process._tickCallback (internal/process/next_tick.js:63:19)
phccp          |   name: 'RequestError',
phccp          |   message: 'Error: connect ECONNREFUSED 172.21.0.4:8080',
phccp          |   cause:
phccp          |    { Error: connect ECONNREFUSED 172.21.0.4:8080
phccp          |        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1097:14)
phccp          |      errno: 'ECONNREFUSED',
phccp          |      code: 'ECONNREFUSED',
phccp          |      syscall: 'connect',
phccp          |      address: '172.21.0.4',
phccp          |      port: 8080 },
phccp          |   error:
phccp          |    { Error: connect ECONNREFUSED 172.21.0.4:8080
phccp          |        at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1097:14)
phccp          |      errno: 'ECONNREFUSED',
phccp          |      code: 'ECONNREFUSED',
phccp          |      syscall: 'connect',
phccp          |      address: '172.21.0.4',
phccp          |      port: 8080 },
phccp          |   options:
phccp          |    { method: 'POST',
phccp          |      uri: 'http://phccp-prov:8080/rest/v1/activities/batch',
phccp          |      body: [ [Object], [Object] ],
phccp          |      headers: { 'User-Agent': 'Request-Promise' },
phccp          |      json: true,
phccp          |      callback: [Function: RP$callback],
phccp          |      transform: undefined,
phccp          |      simple: true,
phccp          |      resolveWithFullResponse: false,
phccp          |      transform2xxOnly: false },
phccp          |   response: undefined }
phccp          | Express server listening on 443, in production mode

phccp-prov is only ready later:

phccp-prov     |  * Serving Flask app "synprov.config" (lazy loading)
phccp-prov     |  * Environment: production
phccp-prov     |    WARNING: This is a development server. Do not use it in a production deployment.
phccp-prov     |    Use a production WSGI server instead.
phccp-prov     |  * Debug mode: off
phccp-prov     | INFO:werkzeug: * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)

[ ] Fix this issue

tschaffter commented 4 years ago

Notes on Docker and missing app.html

Running sudo gulp docker:server:dist shows the following error:

phccp          | Error: ENOENT: no such file or directory, stat '/usr/src/app/client/app.html'

[ ] Fix this issue

Sage-Bionetworks / sagebio-collaboration-portal