search

wso2 / api-manager

All issues, tasks, improvements and new features of WSO2 API Manager
Apache License 2.0
34 stars 8 forks source link

Distribution of API Developer Portal #3343

Closed piyumaldk closed 4 days ago

piyumaldk commented 2 weeks ago

Description

The task is to determine the most suitable approach for distributing the API Developer Portal, focusing on efficiency and minimising platform dependencies. This involves exploring, implementing, and testing various methods to finalise the best distribution strategy.

Affected Component

None

Version

No response

Related Issues

No response

Suggested Labels

No response

piyumaldk commented 2 weeks ago

Update [2024-11-06]

TODO: Try adding express related modules manually to packaging and test

piyumaldk commented 2 weeks ago

Update [2024-11-07]

   "dependencies": {
    "devportal-webapp": "file:./artifacts/devportal-webapp-1.0.59.tgz",
    "chokidar": "^4.0.1",
    "fs-extra": "^11.2.0",
    "path": "^0.12.7"
  }
> devportal-developer-webapp@1.0.0 devportal
> sh startup.sh single

Starting Backend service...
Starting Devportal application...

> devportal-developer-webapp@1.0.0 build-css
> node watcher.js

pkg/prelude/bootstrap.js:657
    const error = new Error(
                  ^

Error: File '/**/api-developer-portal-core/src/utils/Users/piyumal/Desktop/forked-apim/api-developer-portal/src/pages/home/page.hbs' was not included into executable at compilation stage. Please recompile adding it as asset or script.
    at error_ENOENT (pkg/prelude/bootstrap.js:657:19)
    at readFileFromSnapshot (pkg/prelude/bootstrap.js:1046:29)
    at Object.readFileSync (pkg/prelude/bootstrap.js:1094:18)
    at renderTemplate (/snapshot/api-developer-portal-core/src/utils/util.js:83:33)
    at loadOrgContentFromFile (/snapshot/api-developer-portal-core/src/controllers/orgContentController.js:36:12)
    at loadOrganizationContent (/snapshot/api-developer-portal-core/src/controllers/orgContentController.js:16:22)
    at Layer.handle [as handle_request] (/snapshot/api-developer-portal-core/node_modules/express/lib/router/layer.js:95:5)
    at next (/snapshot/api-developer-portal-core/node_modules/express/lib/router/route.js:149:13)
    at Route.dispatch (/snapshot/api-developer-portal-core/node_modules/express/lib/router/route.js:119:3)
    at Layer.handle [as handle_request] (/snapshot/api-developer-portal-core/node_modules/express/lib/router/layer.js:95:5) {
  errno: -2,
  code: 'ENOENT',
  path: '/snapshot/api-developer-portal-core/src/utils/Users/piyumal/Desktop/forked-apim/api-developer-portal/src/pages/home/page.hbs',
  pkg: true
}

TODO: Debug the new issue

[1] https://www.npmjs.com/package/npm-pack-all [2] https://www.geeksforgeeks.org/node-js-process-execpath-property/

piyumaldk commented 2 weeks ago

Update [2024-11-08]

webpack.config.js in root

const path = require('path');
const CopyWebpackPlugin = require('copy-webpack-plugin');

module.exports = {
  mode: 'production',
  stats: {
    errorDetails: true
  },
  entry: {
    // app: './src/app.js',              
    // multiTenant: './src/multi-tenant.js', 
    singleTenant: './src/single-tenant.js' 
  },

  output: {
    path: path.resolve(__dirname, 'dist'), 
    filename: '[name].bundle.js', 
  },

  target: 'node',

  module: {
    rules: [
      {
        test: /\.js$/,
        use: {
          loader: 'babel-loader',
          options: {
            presets: ['@babel/preset-env'],
          },
        },
      },
    ],
  },

  plugins: [
    new CopyWebpackPlugin({
      patterns: [
        { from: './src/pages/tryout', to: './src/pages/tryout' },
      ],
    }),
  ],

  resolve: {
    extensions: ['.js', '.json'],
    alias: {
      '@src': path.resolve(__dirname, 'src'),
    },
  },
};

package.json in root

"scripts": {
    "design-mode": "npm run build-css --watch & nodemon src/dev-server.js",
    "devportal": "npm run build-css --watch & nodemon src/app.js",
    "multi-tenant": "npm run build-css --watch & nodemon src/multi-tenant.js",
    "build-css": "node watcher.js",
    "build": "webpack --config webpack.config.js"
  },
"devDependencies": {
    "@babel/core": "^7.26.0",
    "@babel/preset-env": "^7.26.0",
    "babel-loader": "^9.2.1",
    "copy-webpack-plugin": "^12.0.2",
    "webpack": "^5.96.1",
    "webpack-cli": "^5.1.4"
  }

TODO: There is a warning in the server like below.

Warning: connect.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.

This needs to be further investigated

piyumaldk commented 1 week ago

Update [2024-11-11]

TODO: Once the approval is there, should work on the Jenkins build

[1] https://stackoverflow.com/questions/10760620/using-memorystore-in-production [2] https://github.com/orgs/wso2/projects/115/views/1

piyumaldk commented 1 week ago

Update [2024-11-12]

on: pull_request: types: [closed] branches: [main] # Adjust if you want a different branch

jobs: build: if: github.event.pull_request.merged == true runs-on: ubuntu-latest

steps:
  - name: Checkout code
    uses: actions/checkout@v4 # Update to the latest version

  - name: Set up Node.js
    uses: actions/setup-node@v4 # Update to the latest version
    with:
      node-version: '20' # Use Node.js 20

  - name: Install dependencies
    run: npm install

  - name: Build project
    run: npm run build

  - name: Upload dist folder as an artifact
    uses: actions/upload-artifact@v4 # Update to the latest version
    with:
      name: dist-output
      path: dist/

release: if: github.event.pull_request.merged == true runs-on: ubuntu-latest needs: build

steps:
  - name: Download dist folder artifact
    uses: actions/download-artifact@v4 # Update to the latest version
    with:
      name: dist-output

  - name: Create release directory
    run: mkdir release && cp -R dist/* release/

  - name: Create GitHub release
    uses: ncipollo/release-action@v1
    with:
      token: ${{ secrets.GITHUB_TOKEN }}
      tag: v1.${{ github.event.pull_request.number }} # Custom tag format using PR number
      name: Release for PR #${{ github.event.pull_request.number }}
      body: "Automated release generated on PR merge."
      draft: false
      prerelease: false
      artifacts: "release/*"

- Final WebPack config is like below

const path = require('path'); const CopyWebpackPlugin = require('copy-webpack-plugin'); const TerserPlugin = require('terser-webpack-plugin');

module.exports = { mode: 'production', stats: { errorDetails: true, }, entry: { // app: './src/app.js', // multiTenant: './src/multi-tenant.js', singleTenant: './src/single-tenant.js', },

output: { path: path.resolve(__dirname, 'dist'), filename: '[name].bundle.js', },

target: 'node',

module: { rules: [ { test: /.js$/, use: { loader: 'babel-loader', options: { presets: ['@babel/preset-env'], }, }, }, ], },

plugins: [ new CopyWebpackPlugin({ patterns: [ { from: './src/pages/tryout', to: './src/pages/tryout' }, ], }), ],

resolve: { extensions: ['.js', '.json'], alias: { '@src': path.resolve(__dirname, 'src'), }, },

optimization: { minimize: true, minimizer: [ new TerserPlugin({ terserOptions: { format: { comments: false, }, }, extractComments: false, }), ], }, };

- Participated the PR review call for devportal
   - In there, it was suggested that to ignore webpack as it needs node to run and try a method in exe.

- Again start working on pkg (As mentioned in the issue, I have worked on this approach for some time so I did not have to start from the beginning)

- Found out that when using execute path [1], it's like below which will also include the exe file.
   - exe path - /Users/piyumal/Desktop/forked-apim/api-developer-portal/artifacts/devportal-webapp
   - Because of that, change the _dirname with execute path like below

on: pull_request: types: [closed] branches: [master] # Only trigger for PRs targeting the master branch

permissions: contents: write # Ensures the workflow can create releases

jobs: build: if: github.event.pull_request.merged == true runs-on: ubuntu-latest

steps:
  - name: Checkout code from master branch
    uses: actions/checkout@v4
    with:
      ref: master  # Explicitly checkout master branch

  - name: Set up Node.js
    uses: actions/setup-node@v4
    with:
      node-version: '20'

  - name: Install dependencies
    run: npm install

  - name: Build project
    run: npm run build

  - name: Upload dist folder as an artifact
    uses: actions/upload-artifact@v4
    with:
      name: dist-output
      path: dist/

release: if: github.event.pull_request.merged == true runs-on: ubuntu-latest needs: build

steps:
  - name: Checkout master branch
    uses: actions/checkout@v4
    with:
      ref: master  # Ensure we're on master for tagging

  - name: Download dist folder artifact
    uses: actions/download-artifact@v4
    with:
      name: dist-output
      path: ./dist-output  # Download artifact to a local folder

  - name: Create tag for release
    run: |
      git config --global user.name "github-actions[bot]"
      git config --global user.email "github-actions[bot]@users.noreply.github.com"
      git tag v1.${{ github.event.pull_request.number }}
      git push origin v1.${{ github.event.pull_request.number }}

  - name: Debug downloaded artifact contents
    run: ls -R ./dist-output  # Lists the contents for debugging purposes

  - name: Create GitHub release
    uses: ncipollo/release-action@v1
    with:
      tag: v1.${{ github.event.pull_request.number }}  # Use the manually created tag
      name: Release for PR #${{ github.event.pull_request.number }}
      body: "Automated release generated on PR merge."
      draft: false
      prerelease: false
      artifacts: "./dist-output/*"  # Include all files in the dist-output folder as release assets


   - Below is the output as expected
![image](https://github.com/user-attachments/assets/4926ab7f-9bdc-40e9-b00e-6e6adbfb84a9)

- Tested the macos file in devportal. Script failed due to permission issue.

TODO: Check on the permission issue

[1] https://www.geeksforgeeks.org/node-js-process-execpath-property/
piyumaldk commented 1 week ago

Update [2024-11-13]

name: Build and Release on Tag Push

on:
  push:
    tags:
      - 'v*'  

permissions:
  contents: write  

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm install

      - name: Build project
        run: npm run build

      - name: Upload dist folder as an artifact
        uses: actions/upload-artifact@v4
        with:
          name: dist-output
          path: dist/

  release:
    runs-on: ubuntu-latest
    needs: build

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Download dist folder artifact
        uses: actions/download-artifact@v4
        with:
          name: dist-output
          path: ./dist-output  

      - name: Zip each file in dist-output
        run: |
          mkdir -p zipped-output
          for file in ./dist-output/*; do
            filename=$(basename "$file")
            zip -r "zipped-output/$filename.zip" "$file"
          done

      - name: Debug zipped artifact contents
        run: ls -R ./zipped-output  

      - name: Create or Update GitHub Release
        uses: ncipollo/release-action@v1
        with:
          tag: ${{ github.ref_name }}  
          name: Release ${{ github.ref_name }}
          body: "Automated release generated on tag push."
          draft: false
          prerelease: false
          artifacts: "./zipped-output/*"  
          allowUpdates: true

TODO: Debug and find out the reason of failing the portal

piyumaldk commented 1 week ago

Update [2024-11-14]